Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-870802: Dictionary changed size during iteration when running in a multithreaded environment #1648

Closed
fjetter opened this issue Jul 18, 2023 · 1 comment · Fixed by #1665
Closed

SNOW-870802: Dictionary changed size during iteration when running in a multithreaded environment #1648

fjetter opened this issue Jul 18, 2023 · 1 comment · Fixed by #1665
Labels
bug triaged

Comments

@fjetter
Copy link

fjetter commented Jul 18, 2023 •
edited
Loading

Python version

3.10

Operating system and processor architecture

OSX

Installed packages

Transaction
  
    Prefix: /Users/runner/miniconda3/envs/test-environment
  
    Updating specs:
  
     - python=3.10
     - dask
     - distributed
     - pandas
     - pyarrow[version='<11']
     - snowflake-connector-python[version='>=2.6.0']
     - snowflake-sqlalchemy
     - pytest
     - python=3.10
  
  
    Package                            Version  Build                 Channel                 Size
  ──────────────────────────────────────────────────────────────────────────────────────────────────
    Install:
  ──────────────────────────────────────────────────────────────────────────────────────────────────
  
    + asn1crypto                         1.5.1  pyhd8ed1ab_0          conda-forge/noarch      81kB
    + aws-c-auth                         0.7.0  h1eac0a6_2            conda-forge/osx-64      89kB
    + aws-c-cal                          0.6.0  h865601d_0            conda-forge/osx-64      40kB
    + aws-c-common                      0.8.23  h0dc2134_0            conda-forge/osx-64     180kB
    + aws-c-compression                 0.2.17  hd42d803_1            conda-forge/osx-64      18kB
    + aws-c-event-stream                 0.3.1  h7dedc3b_1            conda-forge/osx-64      47kB
    + aws-c-http                        0.7.11  h3f765f7_0            conda-forge/osx-64     164kB
    + aws-c-io                         0.13.28  h8caa2fd_0            conda-forge/osx-64     135kB
    + aws-c-mqtt                        0.8.14  h81f52c0_2            conda-forge/osx-64     131kB
    + aws-c-s3                          0.3.13  h54f3bce_2            conda-forge/osx-64      76kB
    + aws-c-sdkutils                    0.1.11  hd42d803_1            conda-forge/osx-64      47kB
    + aws-checksums                     0.1.16  hd42d803_1            conda-forge/osx-64      49kB
    + aws-crt-cpp                       0.20.3  h9f4e08a_4            conda-forge/osx-64     272kB
    + aws-sdk-cpp                      1.10.57  h6c51782_17           conda-forge/osx-64       3MB
    + bokeh                              3.2.0  pyhd8ed1ab_0          conda-forge/noarch       6MB
    + brotlipy                           0.7.0  py310h90acd4f_1005    conda-forge/osx-64     376kB
    + bzip2                              1.0.8  h0d85af4_4            conda-forge/osx-64     159kB
    + c-ares                            1.19.1  h0dc2134_0            conda-forge/osx-64     103kB
    + ca-certificates                 2023.5.7  h8857fd0_0            conda-forge/osx-64     149kB
    + certifi                         2023.5.7  pyhd8ed1ab_0          conda-forge/noarch     152kB
    + cffi                              1.15.1  py310ha78151a_3       conda-forge/osx-64     221kB
    + charset-normalizer                 3.2.0  pyhd8ed1ab_0          conda-forge/noarch      46kB
    + click                              8.1.5  unix_pyh707e725_0     conda-forge/noarch      85kB
    + cloudpickle                        2.2.1  pyhd8ed1ab_0          conda-forge/noarch      28kB
    + colorama                           0.4.6  pyhd8ed1ab_0          conda-forge/noarch      25kB
    + contourpy                          1.1.0  py310h88cfcbd_0       conda-forge/osx-64     215kB
    + cryptography                      38.0.4  py310hdd0c95c_0       conda-forge/osx-64       1MB
    + cytoolz                           0.12.0  py310h90acd4f_1       conda-forge/osx-64     360kB
    + dask                            2023.7.0  pyhd8ed1ab_0          conda-forge/noarch       7kB
    + dask-core                       2023.7.0  pyhd8ed1ab_0          conda-forge/noarch     852kB
    + distributed                     2023.7.0  pyhd8ed1ab_0          conda-forge/noarch     778kB
    + exceptiongroup                     1.1.2  pyhd8ed1ab_0          conda-forge/noarch      19kB
    + filelock                          3.12.2  pyhd8ed1ab_0          conda-forge/noarch      15kB
    + freetype                          2.12.1  h3f81eb7_1            conda-forge/osx-64     600kB
    + fsspec                          2023.6.0  pyh1a96a4e_0          conda-forge/noarch     118kB
    + gflags                             2.2.2  hb1e8313_1004         conda-forge/osx-64      95kB
    + glog                               0.6.0  h8ac2a54_0            conda-forge/osx-64     101kB
    + gmp                                6.2.1  h2e338ed_0            conda-forge/osx-64     792kB
    + greenlet                           2.0.2  py310h9e9d8ca_1       conda-forge/osx-64     181kB
    + idna                                 3.4  pyhd8ed1ab_0          conda-forge/noarch      57kB
    + importlib-metadata                 6.8.0  pyha770c72_0          conda-forge/noarch      26kB
    + importlib_metadata                 6.8.0  hd8ed1ab_0            conda-forge/noarch       9kB
    + iniconfig                          2.0.0  pyhd8ed1ab_0          conda-forge/noarch      11kB
    + jinja2                             3.1.2  pyhd8ed1ab_1          conda-forge/noarch     101kB
    + krb5                              1.21.1  hb884880_0            conda-forge/osx-64       1MB
    + lcms2                               2.15  h2dcdeff_1            conda-forge/osx-64     228kB
    + lerc                               4.0.0  hb486fe8_0            conda-forge/osx-64     290kB
    + libabseil                     20230125.3  cxx17_h000cb23_0      conda-forge/osx-64       1MB
    + libarrow                          10.0.1  h0b3ac1b_34_cpu       conda-forge/osx-64      19MB
    + libblas                            3.9.0  17_osx64_openblas     conda-forge/osx-64      15kB
    + libbrotlicommon                    1.0.9  hb7f2c08_9            conda-forge/osx-64      70kB
    + libbrotlidec                       1.0.9  hb7f2c08_9            conda-forge/osx-64      32kB
    + libbrotlienc                       1.0.9  hb7f2c08_9            conda-forge/osx-64     279kB
    + libcblas                           3.9.0  17_osx64_openblas     conda-forge/osx-64      15kB
    + libcrc32c                          1.1.2  he49afe7_0            conda-forge/osx-64      20kB
    + libcurl                            8.1.2  h5f667d7_1            conda-forge/osx-64     353kB
    + libcxx                            16.0.6  hd57cbcb_0            conda-forge/osx-64       1MB
    + libdeflate                          1.18  hac1461d_0            conda-forge/osx-64      67kB
    + libedit                     3.1.20191231  h0678c8f_2            conda-forge/osx-64     105kB
    + libev                               4.33  haf1e3a3_1            conda-forge/osx-64     101kB
    + libevent                          2.1.12  ha90c15b_1            conda-forge/osx-64     373kB
    + libffi                             3.4.2  h0d85af4_5            conda-forge/osx-64      51kB
    + libgfortran                        5.0.0  11_3_0_h97931a8_31    conda-forge/osx-64     160kB
    + libgfortran5                      12.2.0  he409387_31           conda-forge/osx-64       2MB
    + libgoogle-cloud                   2.12.0  h37a168a_1            conda-forge/osx-64      33MB
    + libgrpc                           1.56.2  he6801ca_0            conda-forge/osx-64       4MB
    + libjpeg-turbo                    2.1.5.1  hb7f2c08_0            conda-forge/osx-64     458kB
    + liblapack                          3.9.0  17_osx64_openblas     conda-forge/osx-64      15kB
    + libnghttp2                        1.52.0  he2ab024_0            conda-forge/osx-64     613kB
    + libopenblas                       0.3.23  openmp_h429af6e_0     conda-forge/osx-64       6MB
    + libpng                            1.6.39  ha978bb4_0            conda-forge/osx-64     272kB
    + libprotobuf                       4.23.3  h5feb325_0            conda-forge/osx-64       2MB
    + libsqlite                         3.42.0  h58db7d2_0            conda-forge/osx-64     879kB
    + libssh2                           1.11.0  hd019ec5_0            conda-forge/osx-64     260kB
    + libthrift                         0.18.1  h88b220a_2            conda-forge/osx-64     324kB
    + libtiff                            4.5.1  hf955e92_0            conda-forge/osx-64     395kB
    + libutf8proc                        2.8.0  hb7f2c08_0            conda-forge/osx-64      99kB
    + libwebp-base                       1.3.1  h0dc2134_0            conda-forge/osx-64     346kB
    + libxcb                              1.15  hb7f2c08_0            conda-forge/osx-64     314kB
    + libzlib                           1.2.13  h8a1eda9_5            conda-forge/osx-64      59kB
    + llvm-openmp                       16.0.6  hff08bdf_0            conda-forge/osx-64     296kB
    + locket                             1.0.0  pyhd8ed1ab_0          conda-forge/noarch       8kB
    + lz4                                4.3.2  py310h8d4e1d9_0       conda-forge/osx-64      34kB
    + lz4-c                              1.9.4  hf0c8a7f_0            conda-forge/osx-64     156kB
    + markupsafe                         2.1.3  py310h6729b98_0       conda-forge/osx-64      23kB
    + msgpack-python                     1.0.5  py310ha23aa8a_0       conda-forge/osx-64      77kB
    + ncurses                              6.4  hf0c8a7f_0            conda-forge/osx-64     828kB
    + numpy                             1.25.1  py310h7451ae0_0       conda-forge/osx-64       6MB
    + openjpeg                           2.5.0  h13ac156_2            conda-forge/osx-64     330kB
    + openssl                            3.1.1  h8a1eda9_1            conda-forge/osx-64       2MB
    + orc                                1.9.0  hef23039_1            conda-forge/osx-64     441kB
    + oscrypto                           1.2.1  pyhd3deb0d_0          conda-forge/noarch     123kB
    + packaging                           23.1  pyhd8ed1ab_0          conda-forge/noarch      46kB
    + pandas                             2.0.3  py310h5e4fcda_1       conda-forge/osx-64      12MB
    + partd                              1.4.0  pyhd8ed1ab_0          conda-forge/noarch      20kB
    + pillow                            10.0.0  py310hd63a8c7_0       conda-forge/osx-64      46MB
    + pip                                 23.2  pyhd8ed1ab_0          conda-forge/noarch       1MB
    + pluggy                             1.2.0  pyhd8ed1ab_0          conda-forge/noarch      22kB
    + psutil                             5.9.5  py310h90acd4f_0       conda-forge/osx-64     369kB
    + pthread-stubs                        0.4  hc929b4f_1001         conda-forge/osx-64       6kB
    + pyarrow                           10.0.1  py310h435aefc_34_cpu  conda-forge/osx-64       3MB
    + pycparser                           2.21  pyhd8ed1ab_0          conda-forge/noarch     103kB
    + pycryptodomex                     3.18.0  py310h6729b98_0       conda-forge/osx-64       1MB
    + pyjwt                              2.7.0  pyhd8ed1ab_0          conda-forge/noarch      24kB
    + pyopenssl                         22.1.0  pyhd8ed1ab_0          conda-forge/noarch     125kB
    + pysocks                            1.7.1  pyha2e5f31_6          conda-forge/noarch      19kB
    + pytest                             7.4.0  pyhd8ed1ab_0          conda-forge/noarch     244kB
    + python                           3.10.12  had23ca6_0_cpython    conda-forge/osx-64      13MB
    + python-dateutil                    2.8.2  pyhd8ed1ab_0          conda-forge/noarch     246kB
    + python-tzdata                     2023.3  pyhd8ed1ab_0          conda-forge/noarch     143kB
    + python_abi                          3.10  3_cp310               conda-forge/osx-64       6kB
    + pytz                              2023.3  pyhd8ed1ab_0          conda-forge/noarch     187kB
    + pyyaml                               6.0  py310h90acd4f_5       conda-forge/osx-64     167kB
    + re2                           2023.03.02  h096449b_0            conda-forge/osx-64     185kB
    + readline                             8.2  h9e318b2_1            conda-forge/osx-64     256kB
    + requests                          2.31.0  pyhd8ed1ab_0          conda-forge/noarch      57kB
    + setuptools                        68.0.0  pyhd8ed1ab_0          conda-forge/noarch     464kB
    + six                               1.16.0  pyh6c4a22f_0          conda-forge/noarch      14kB
    + snappy                            1.1.10  h225ccf5_0            conda-forge/osx-64      35kB
    + snowflake-connector-python         3.0.4  py310hdf934cd_1       conda-forge/osx-64     702kB
    + snowflake-sqlalchemy               1.4.7  pyhd8ed1ab_0          conda-forge/noarch      34kB
    + sortedcontainers                   2.4.0  pyhd8ed1ab_0          conda-forge/noarch      26kB
    + sqlalchemy                        1.4.49  py310h6729b98_0       conda-forge/osx-64       2MB
    + tblib                              1.7.0  pyhd8ed1ab_0          conda-forge/noarch      15kB
    + tk                                8.6.12  h5dbffcc_0            conda-forge/osx-64       4MB
    + tomli                              2.0.1  pyhd8ed1ab_0          conda-forge/noarch      16kB
    + toolz                             0.12.0  pyhd8ed1ab_0          conda-forge/noarch      49kB
    + tornado                            6.3.2  py310h6729b98_0       conda-forge/osx-64     641kB
    + typing_extensions                  4.7.1  pyha770c72_0          conda-forge/noarch      36kB
    + tzdata                             2023c  h71feb2d_0            conda-forge/noarch     118kB
    + urllib3                          1.26.15  pyhd8ed1ab_0          conda-forge/noarch     113kB
    + wheel                             0.40.0  pyhd8ed1ab_1          conda-forge/noarch      57kB
    + xorg-libxau                       1.0.11  h0dc2134_0            conda-forge/osx-64      13kB
    + xorg-libxdmcp                      1.1.3  h35c211d_0            conda-forge/osx-64      17kB
    + xyzservices                     2023.7.0  pyhd8ed1ab_0          conda-forge/noarch      36kB
    + xz                                 5.2.6  h775f41a_0            conda-forge/osx-64     238kB
    + yaml                               0.2.5  h0d85af4_2            conda-forge/osx-64      84kB
    + zict                               3.0.0  pyhd8ed1ab_0          conda-forge/noarch      36kB
    + zipp                              3.16.2  pyhd8ed1ab_0          conda-forge/noarch      19kB

What did you do?

We encountered an exception in our dask-snowflake CI, see https://github.com/coiled/dask-snowflake/actions/runs/5581071003/jobs/10198820063

        def _log_telemetry_imported_packages(self) -> None:
            if self._log_imported_packages_in_telemetry:
                # filter out duplicates caused by submodules
                # and internal modules with names starting with an underscore
>               imported_modules = {
                    k.split(".", maxsplit=1)[0]
                    for k in sys.modules.keys()
                    if not k.startswith("_")
                }
E               RuntimeError: dictionary changed size during iteration

This thing can happen if you are running in a multithreaded environment. If another thread is importing a new package such that sys.modules is mutated while a new SnowflakeConnection is established this will raise such an exception. I haven't tried to reproduce it.

Converting the keys view to a list or set in the dictionary comprehension should guard against this

For example,

            imported_modules = {
                k.split(".", maxsplit=1)[0]
                for k in list(sys.modules)
                if not k.startswith("_")
            }

What did you expect to see?

No exception

Can you set logging to DEBUG and collect the logs?

No response
@github-actions github-actions bot changed the title Dictionary changed size during iteration when running in a multithreaded environment SNOW-870802: Dictionary changed size during iteration when running in a multithreaded environment Jul 18, 2023
@fjetter fjetter mentioned this issue Jul 18, 2023
Open
@jrbourbeau jrbourbeau mentioned this issue Jul 18, 2023
Merged
@sfc-gh-aalam sfc-gh-aalam mentioned this issue Jul 24, 2023
Merged
7 tasks
@zyqhpz
Copy link

zyqhpz commented Oct 21, 2024 •
edited
Loading

RuntimeError: dictionary changed size during iteration

somehow it still occurs to me, is there any better way we can handle this?
like using .copy()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

SNOW-870802: make log_telemetry_imported_packages threadsafe snowflakedb/snowflake-connector-python
3 participants
@fjetter @zyqhpz @sfc-gh-aalam