SNOW-1655751: ERROR with GenericFunction 'flatten' is already registered and is going to be overridden

[nadavuoptimove]

Hi.

1. What version of Python are you using?

Python 3.9.6 (default, Feb 3 2024, 15:58:27)

2. What operating system and processor architecture are you using?

macOS-14.5-arm64-arm-64bit (it happens also when running it on docker --platform=linux/amd64 public.ecr.aws/docker/library/python:3.11.7-slim)

3. What are the component versions in the environment (pip freeze)?

asn1crypto==1.5.1
cachetools==5.5.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==2.1.1
cryptography==36.0.2
exceptiongroup==1.2.2
filelock==3.16.0
google-api-core==2.19.2
google-auth==2.34.0
google-cloud-core==2.4.1
google-cloud-pubsub==2.23.0
google-cloud-secret-manager==2.20.2
google-cloud-storage==2.18.2
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.65.0
grpc-google-iam-v1==0.13.1
grpcio==1.66.1
grpcio-status==1.66.1
idna==3.8
iniconfig==2.0.0
numpy==1.26.4
packaging==24.1
pandas==1.5.3
platformdirs==4.3.1
pluggy==1.5.0
proto-plus==1.24.0
protobuf==5.28.0
psycopg2==2.9.9
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycparser==2.22
PyJWT==2.9.0
pyodbc==4.0.39
pyOpenSSL==22.0.0
pysnowflake==0.1.3
pytest==7.4.0
pytest-mock==3.10.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.0
python-json-logger==2.0.7
pytz==2024.1
requests==2.28.1
rsa==4.9
six==1.16.0
snowflake-connector-python==3.12.1
snowflake-sqlalchemy==1.6.1
sortedcontainers==2.4.0
SQLAlchemy==2.0.34
tomli==2.0.1
tomlkit==0.13.2
tornado==6.4.1
typing_extensions==4.12.2
urllib3==1.26.20

4. What did you do?

we have number of different python micro services that run on K8, lately we are seeing an issue when running this jobs:

SAWarning: The GenericFunction 'flatten' is already registered and is going to be overridden.
  functions.register_function("flatten", flatten)

This jobs are using the most updated version of snowflake-sqlalchemy and sqlalchemy.

We think the issue is coming from the following line that was pushed in the following commit :

d78f0c0

Before this commit we didn't have this error.

5. Steps to reproduce:
   Install the pypi packages we provided and start running python code, the error will show at the beginning.

Waiting to hear from you.

Thanks,
Nadav

[sfc-gh-dszmolka]

hi and thank you for raising this issue with us - does the above sentence

Install the pypi packages we provided and start running python code, the error will show at the beginning.

mean that you wanted to attach something (repro script, etc) to this issue? If so, it would be immensely helpful to have it. Thank you in advance !

[nadavuoptimove]

Hi,

No, because this error occurs in many different python jobs, so the python code you are running is not important.

Thanks

[top-jvn]

Hi,

We experience the same problem. Example:

from typing import Any

from snowflake.connector import connect
from sqlalchemy import create_engine, text


def connection_factory():
    connection_params = {
        "account": ...,
        "user": ...,
        "role": ...,
        "warehouse": ...,
        "database": ...,
        "schema": ...,
        "password": ...,
    }
    return connect(**connection_params)


def run():
    engine: Any = create_engine("snowflake://", creator=connection_factory)

    with engine.connect() as connection:
        try:
            [results] = connection.execute(
                text("select 'hello world' as msg")
            ).fetchone()
            print(results)
        finally:
            connection.close()

    engine.dispose()


if __name__ == "__main__":
    run()

[sfc-gh-dszmolka]

thank you for the example ! we're going to take a look.

[sfc-gh-dszmolka]

work in progress - draft PR open at #532 . Please bear with us while its fully worked out and merged (and a fixed connector is released) I'll keep this thread posted.

[catzc]

Hi! I see #532 is released. Do we know when a new package will be released?

[sfc-gh-dszmolka]

Indeed the fix is merged ! At this moment I do not have any indication of the new release, but I would assume it doesn't happen before the end of this month. Thank you for bearing with us !