ValueError: numpy.ndarray size changed, may indicate binary incompatibility

[jlewi]

After installing TensorFlow federated in a python3.10 image I got the following error on import.

>>> import tensorflow_federated as tff
2022-08-01 21:46:22.522372: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
2022-08-01 21:46:22.522419: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/__init__.py", line 20, in <module>
    from tensorflow_federated.python import aggregators
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/__init__.py", line 24, in <module>
    from tensorflow_federated.python.aggregators.concat import concat_factory
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_federated/python/aggregators/concat.py", line 24, in <module>
    import tensorflow as tf
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/__init__.py", line 473, in <module>
    keras._load()
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/util/lazy_loader.py", line 41, in _load
    module = importlib.import_module(self.__name__)
  File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/opt/conda/lib/python3.10/site-packages/keras/__init__.py", line 24, in <module>
    from keras import models
  File "/opt/conda/lib/python3.10/site-packages/keras/models/__init__.py", line 18, in <module>
    from keras.engine.functional import Functional
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/functional.py", line 31, in <module>
    from keras.engine import training as training_lib
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/training.py", line 30, in <module>
    from keras.engine import compile_utils
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/compile_utils.py", line 20, in <module>
    from keras import metrics as metrics_mod
  File "/opt/conda/lib/python3.10/site-packages/keras/metrics/__init__.py", line 33, in <module>
    from keras.metrics.metrics import MeanRelativeError
  File "/opt/conda/lib/python3.10/site-packages/keras/metrics/metrics.py", line 22, in <module>
    from keras import activations
  File "/opt/conda/lib/python3.10/site-packages/keras/activations.py", line 20, in <module>
    import keras.layers.activation as activation_layers
  File "/opt/conda/lib/python3.10/site-packages/keras/layers/__init__.py", line 27, in <module>
    from keras.engine.base_preprocessing_layer import PreprocessingLayer
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/base_preprocessing_layer.py", line 19, in <module>
    from keras.engine import data_adapter
  File "/opt/conda/lib/python3.10/site-packages/keras/engine/data_adapter.py", line 38, in <module>
    import pandas as pd  # pylint: disable=g-import-not-at-top
  File "/opt/conda/lib/python3.10/site-packages/pandas/__init__.py", line 30, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "/opt/conda/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

This was in a docker container where I had been manually messing with things. So I'm hoping maybe in a clean build this error doesn't occur.

[jlewi]

https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp

Suggests an issue with numpy version

[jlewi]

It looks like installing TensorFlow federated installs

Collecting numpy~=1.21.4
  Downloading numpy-1.21.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)

[jlewi]

pip freeze shows

numpy==1.21.6
pandas==1.1.5

[jlewi]

Try installing and upgrading numpy and I get the following

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow-privacy 0.8.2 requires numpy~=1.21.4, but you have numpy 1.23.1 which is incompatible.
tensorflow-federated 0.32.0 requires numpy~=1.21.4, but you have numpy 1.23.1 which is incompatible.
scipy 1.7.3 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.23.1 which is incompatible.
dp-accounting 0.1.2 requires numpy~=1.21.4, but you have numpy 1.23.1 which is incompatible.
Successfully installed numpy-1.23.1

However import tensorflow_federated seems to work now.

In [1]: import tensorflow_federated
2022-08-01 22:03:29.638709: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
2022-08-01 22:03:29.638760: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.1
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

In [2]: 

[jlewi]

Looks like the versions get set here
https://github.com/tensorflow/federated/blob/e1d5cfb4b1b389ef96b1ab1f352e52fa1ee944ec/tensorflow_federated/tools/python_package/setup.py#L65

My guess is its pinning numpy which is incompatible with the default version of pandas but not pinning a version of pandas compatible with that numpy

[jlewi]

For a definition of ~= syntax see
https://stackoverflow.com/questions/39590187/in-requirements-txt-what-does-tilde-equals-mean#:~:text=~%3D%20means%20a%20compatible%20version,10%20and%20higher%20(0.6.

[jlewi]

pandas 1.1.5 is quite old
https://pandas.pydata.org/docs/whatsnew/v1.1.5.html

[jlewi]

It looks like TensorFlow privacy is pinning 1.1.4
of pandas.
https://github.com/tensorflow/privacy/blob/a8a520684164bf1bf4336e17a2e0931a4bdd8427/setup.py#L36

[jlewi]

Same problem error if I try to pin pandas and numpy as below

pip3 install --upgrade numpy~=1.21.4 pandas~=1.1.4
Requirement already satisfied: numpy~=1.21.4 in /opt/conda/lib/python3.10/site-packages (1.21.6)
Requirement already satisfied: pandas~=1.1.4 in /opt/conda/lib/python3.10/site-packages (1.1.4)
Collecting pandas~=1.1.4
  Downloading pandas-1.1.5.tar.gz (5.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 43.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... \

I suspect as the stackoverflow answer says this may be due to using a different version for building and install.

https://stackoverflow.com/a/66743692/3670531

It looks like its building pandas but using a prebuilt version of numpy so its possible the version of numpy pandas is building against is different from the one installed.

[jlewi]

Lets try forcing pandas to install from a binary

pip3 install --force --only-binary pandas  --upgrade numpy~=1.21.4 pandas~=1.1.4
Collecting numpy~=1.21.4
  Using cached numpy-1.21.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
ERROR: Could not find a version that satisfies the requirement pandas~=1.1.4 (from versions: 1.3.3, 1.3.4, 1.3.5, 1.4.0rc0, 1.4.0, 1.4.1, 1.4.2, 1.4.3)
ERROR: No matching distribution found for pandas~=1.1.4

Looks like there is no longer a prebuilt version of pandas available which is why its compiling from source.

[jlewi]

What if we try forcing numpy to be built from source.

pip3 install --force --no-binary numpy  --upgrade numpy~=1.21.4 pandas~=1.1.4

Does this cause it to be built from source in a compatible way.

[jlewi]

Note this doesn't appear to be directly related to TensorFlow. import pandas has the same problem.

pip3 install --force --no-binary numpy  --upgrade numpy~=1.21.4 pandas~=1.1.4

Didn't fix the problem.

[jlewi]

Looks like we can just upgrade pandas after installing it to work around the issue.

Filed: tensorflow/privacy#295
to see if we could upgrade pandas in the project.

[jlewi]

The workaround for now is just to upgrade pandas after installing TensorFlow federated.