(🎁) warm the cache / pre-cache the stdlib

[KotlinIsland]

mypy-playground can be slow, mostly due to checking the stdlib:

playground

Succeeded!! (2047 ms)
Success: no issues found in 1 source file
LOG: Could not load plugins snapshot: @plugins_snapshot.json

LOG: Mypy Version: 0.971
LOG: Config File: Default
LOG: Configured Executable: None
LOG: Current Executable: /opt/python3.10/bin/python3
LOG: Cache Dir: /dev/null
LOG: Compiled: True
LOG: Exclude: []
LOG: Found source: BuildSource(path='main.py', module='main', has_text=False, base_dir=None)
LOG: Could not load cache for main: main.meta.json
LOG: Metadata not found for main
LOG: Parsing main.py (main)
LOG: Could not load cache for builtins: builtins.meta.json
LOG: Metadata not found for builtins
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/builtins.pyi (builtins)
LOG: Could not load cache for sys: sys.meta.json
LOG: Metadata not found for sys
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/sys.pyi (sys)
LOG: Could not load cache for types: types.meta.json
LOG: Metadata not found for types
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/types.pyi (types)
LOG: Could not load cache for _ast: _ast.meta.json
LOG: Metadata not found for _ast
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_ast.pyi (_ast)
LOG: Could not load cache for _collections_abc: _collections_abc.meta.json
LOG: Metadata not found for _collections_abc
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_collections_abc.pyi (_collections_abc)
LOG: Could not load cache for _typeshed: _typeshed/init.meta.json
LOG: Metadata not found for _typeshed
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_typeshed/init.pyi (_typeshed)
LOG: Could not load cache for collections.abc: collections/abc.meta.json
LOG: Metadata not found for collections.abc
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/collections/abc.pyi (collections.abc)
LOG: Could not load cache for io: io.meta.json
LOG: Metadata not found for io
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/io.pyi (io)
LOG: Could not load cache for typing: typing.meta.json
LOG: Metadata not found for typing
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/typing.pyi (typing)
LOG: Could not load cache for typing_extensions: typing_extensions.meta.json
LOG: Metadata not found for typing_extensions
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/typing_extensions.pyi (typing_extensions)
LOG: Could not load cache for importlib.abc: importlib/abc.meta.json
LOG: Metadata not found for importlib.abc
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/abc.pyi (importlib.abc)
LOG: Could not load cache for importlib.machinery: importlib/machinery.meta.json
LOG: Metadata not found for importlib.machinery
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/machinery.pyi (importlib.machinery)
LOG: Could not load cache for array: array.meta.json
LOG: Metadata not found for array
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/array.pyi (array)
LOG: Could not load cache for ctypes: ctypes/init.meta.json
LOG: Metadata not found for ctypes
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/ctypes/init.pyi (ctypes)
LOG: Could not load cache for mmap: mmap.meta.json
LOG: Metadata not found for mmap
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/mmap.pyi (mmap)
LOG: Could not load cache for pickle: pickle.meta.json
LOG: Metadata not found for pickle
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/pickle.pyi (pickle)
LOG: Could not load cache for os: os/init.meta.json
LOG: Metadata not found for os
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/os/init.pyi (os)
LOG: Could not load cache for collections: collections/init.meta.json
LOG: Metadata not found for collections
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/collections/init.pyi (collections)
LOG: Could not load cache for codecs: codecs.meta.json
LOG: Metadata not found for codecs
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/codecs.pyi (codecs)
LOG: Could not load cache for abc: abc.meta.json
LOG: Metadata not found for abc
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/abc.pyi (abc)
LOG: Could not load cache for importlib: importlib/init.meta.json
LOG: Metadata not found for importlib
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/init.pyi (importlib)
LOG: Could not load cache for importlib.metadata: importlib/metadata/init.meta.json
LOG: Metadata not found for importlib.metadata
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/metadata/init.pyi (importlib.metadata)
LOG: Could not load cache for os.path: os/path.meta.json
LOG: Metadata not found for os.path
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/os/path.pyi (os.path)
LOG: Could not load cache for contextlib: contextlib.meta.json
LOG: Metadata not found for contextlib
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/contextlib.pyi (contextlib)
LOG: Could not load cache for subprocess: subprocess.meta.json
LOG: Metadata not found for subprocess
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/subprocess.pyi (subprocess)
LOG: Could not load cache for _codecs: _codecs.meta.json
LOG: Metadata not found for _codecs
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/_codecs.pyi (_codecs)
LOG: Could not load cache for pathlib: pathlib.meta.json
LOG: Metadata not found for pathlib
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/pathlib.pyi (pathlib)
LOG: Could not load cache for email.message: email/message.meta.json
LOG: Metadata not found for email.message
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/message.pyi (email.message)
LOG: Could not load cache for importlib.metadata._meta: importlib/metadata/_meta.meta.json
LOG: Metadata not found for importlib.metadata._meta
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/importlib/metadata/_meta.pyi (importlib.metadata._meta)
LOG: Could not load cache for posixpath: posixpath.meta.json
LOG: Metadata not found for posixpath
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/posixpath.pyi (posixpath)
LOG: Could not load cache for email: email/init.meta.json
LOG: Metadata not found for email
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/init.pyi (email)
LOG: Could not load cache for email.charset: email/charset.meta.json
LOG: Metadata not found for email.charset
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/charset.pyi (email.charset)
LOG: Could not load cache for email.contentmanager: email/contentmanager.meta.json
LOG: Metadata not found for email.contentmanager
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/contentmanager.pyi (email.contentmanager)
LOG: Could not load cache for email.errors: email/errors.meta.json
LOG: Metadata not found for email.errors
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/errors.pyi (email.errors)
LOG: Could not load cache for email.policy: email/policy.meta.json
LOG: Metadata not found for email.policy
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/policy.pyi (email.policy)
LOG: Could not load cache for genericpath: genericpath.meta.json
LOG: Metadata not found for genericpath
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/genericpath.pyi (genericpath)
LOG: Could not load cache for email.header: email/header.meta.json
LOG: Metadata not found for email.header
LOG: Parsing /layers/google.python.pip/pip/lib/python3.10/site-packages/mypy/typeshed/stdlib/email/header.pyi (email.header)
LOG: Loaded graph with 38 nodes (0.358 sec)
LOG: Found 2 SCCs; largest has 37 nodes
LOG: Processing SCC of size 37 (typing_extensions typing types sys subprocess posixpath pickle pathlib os.path os mmap io importlib.metadata._meta importlib.metadata importlib.machinery importlib.abc importlib genericpath email.policy email.message email.header email.errors email.contentmanager email.charset email ctypes contextlib collections.abc collections codecs array abc _typeshed _collections_abc _codecs _ast builtins) as inherently stale
LOG: Processing SCC singleton (main) as inherently stale
LOG: No fresh SCCs left in queue
LOG: Build finished in 1.761 seconds with 38 modules, and 1 errors

Is it possible to include the cache files in the cloud_function? I'm not quite sure how they work.

[ymyzk]

Thanks for filing the ticket. I had a similar idea before. One idea I had is that running mypy against a Python script like the following at build time to populate the cache.

import abc
import <other stdlibs>

Although this will obviously increase the size of sandbox environments (Docker images, Cloud Functions, etc.) which can affect start up time and hosting cost. We need to create a PoC and see how it works.