Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade pandas to 1.4 to fix pandas binary incompatibility with numpy #295

Open
jlewi opened this issue Aug 2, 2022 · 0 comments
Open

Comments

@jlewi
Copy link

jlewi commented Aug 2, 2022

When I install tensorflow-federated I end up with incompatible builds of numpy and pandas. Here's the main error from the stacktrace (full stack trace below).

    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Based on this Stack Overflow post it looks like there is a problem in the way pandas is getting built and pandas is being built against a version of numpy which is different from the installed version of numpy.

I was able to resolve the issue just by installing a newer version of pandas.

pip install --upgrade pandas~=1.4.3

I think this may solve the problem because newer versions of pandas have wheel's available and don't require compiling from source.

Given pandas 1.1.4 is almost 2 years old. Would it be reasonable to upgrade the version of pandas in tensorflow/privacy?

It looks like pandas is only used in a couple places.
https://github.com/tensorflow/privacy/search?q=pandas

The usage of pandas seems pretty straightforward and seems like low risk to an upgrade.

Here's the full stack trace for import tensorflow_federated

>>> import tensorflow_federated as tff
2022-08-01 21:46:22.522372: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
2022-08-01 21:46:22.522419: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/__init__.py", line 20, in <module>
    from tensorflow_federated.python import aggregators
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/__init__.py", line 24, in <module>
    from tensorflow_federated.python.aggregators.concat import concat_factory
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/concat.py", line 24, in <module>
    import tensorflow as tf
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/__init__.py", line 473, in <module>
    keras._load()
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/util/lazy_loader.py", line 41, in _load
    module = importlib.import_module(self.__name__)
  File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/opt/conda/lib/python3.10/site-packages/keras/__init__.py", line 24, in <module>
    from keras import models
  File "/opt/conda/lib/python3.10/site-packages/keras/models/__init__.py", line 18, in <module>
    from keras.engine.functional import Functional
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/functional.py", line 31, in <module>
    from keras.engine import training as training_lib
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/training.py", line 30, in <module>
    from keras.engine import compile_utils
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/compile_utils.py", line 20, in <module>
    from keras import metrics as metrics_mod
  File "/opt/conda/lib/python3.10/site-packages/keras/metrics/__init__.py", line 33, in <module>
    from keras.metrics.metrics import MeanRelativeError
  File "/opt/conda/lib/python3.10/site-packages/keras/metrics/metrics.py", line 22, in <module>
    from keras import activations
  File "/opt/conda/lib/python3.10/site-packages/keras/activations.py", line 20, in <module>
    import keras.layers.activation as activation_layers
  File "/opt/conda/lib/python3.10/site-packages/keras/layers/__init__.py", line 27, in <module>
    from keras.engine.base_preprocessing_layer import PreprocessingLayer
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/base_preprocessing_layer.py", line 19, in <module>
    from keras.engine import data_adapter
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/data_adapter.py", line 38, in <module>
    import pandas as pd  # pylint: disable=g-import-not-at-top
  File "/opt/conda/lib/python3.10/site-packages/pandas/__init__.py", line 30, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "/opt/conda/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant