Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pickling the AmiciObjective requires an AMICI installation with HDF5 support #703

Closed
harshitaojha opened this issue Jul 13, 2021 · 13 comments
Labels

Comments

@harshitaojha
Copy link

I have libhdf5-serial-dev installed for hdf5 support of amici but still getting the following error.

Amici version = 0.11.7
(can't use the updated amici version as I get error in that)

Traceback (most recent call last):
File "/home/harshita/PycharmProjects/Project_ICL_pyPESTO/pypesto_opti.py", line 55, in
result_powell = optimize.minimize(problem=problem,
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/optimize/optimize.py", line 122, in minimize
ret = engine.execute(tasks, progress_bar=progress_bar)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/engine/multi_process.py", line 58, in execute
pickled_tasks = [pickle.dumps(task) for task in tasks]
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/engine/multi_process.py", line 58, in
pickled_tasks = [pickle.dumps(task) for task in tasks]
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/objective/amici.py", line 234, in getstate
amici.writeSolverSettingsToHDF5(
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/amici/init.py", line 262, in writeSolverSettingsToHDF5
amici.writeSolverSettingsToHDF5(_get_ptr(solver), file, location)
AttributeError: ("module 'amici.amici' has no attribute 'writeSolverSettingsToHDF5'", 'Pickling the AmiciObjective requires an AMICI installation with HDF5 support.')

@dweindl
Copy link
Member

dweindl commented Jul 13, 2021

I have libhdf5-serial-dev installed for hdf5 support of amici but still getting the following error.

Did you install libhdf5-serial-dev before installing AMICI or after? It needs to be present before. If you installed AMICI before, please try reinstalling it. If it still does not work, please post the installation log (pip install -v amici).

Amici version = 0.11.7
(can't use the updated amici version as I get error in that)

Feel free to post an issue over at https://github.com/AMICI-dev/AMICI/issues

@dweindl dweindl added the AMICI label Jul 13, 2021
@harshitaojha
Copy link
Author

It didnt work.

Using pip install -v amici, I get this:

Using pip 21.1.3 from /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-usie88w8
Created temporary directory: /tmp/pip-req-tracker-566v6z4n
Initialized build tracking at /tmp/pip-req-tracker-566v6z4n
Created build tracker: /tmp/pip-req-tracker-566v6z4n
Entered build tracker: /tmp/pip-req-tracker-566v6z4n
Created temporary directory: /tmp/pip-install-wgs28u5e
Requirement already satisfied: amici in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (0.11.7)
Requirement already satisfied: h5py in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from amici) (3.3.0)
Requirement already satisfied: pandas in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from amici) (1.3.0)
Requirement already satisfied: wurlitzer in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from amici) (2.1.0)
Requirement already satisfied: python-libsbml in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from amici) (5.19.0)
Requirement already satisfied: pkgconfig in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from amici) (1.5.4)
Requirement already satisfied: sympy>=1.6.0 in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from amici) (1.8)
Requirement already satisfied: mpmath>=0.19 in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from sympy>=1.6.0->amici) (1.2.1)
Requirement already satisfied: numpy>=1.19.3 in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from h5py->amici) (1.21.0)
Requirement already satisfied: pytz>=2017.3 in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from pandas->amici) (2021.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from pandas->amici) (2.8.1)
Requirement already satisfied: six>=1.5 in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas->amici) (1.16.0)
Created temporary directory: /tmp/pip-unpack-pfiae32m
Removed build tracker: '/tmp/pip-req-tracker-566v6z4n'

@dweindl
Copy link
Member

dweindl commented Jul 13, 2021

Requirement already satisfied: amici in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (0.11.7)

Ah that didn't actually re-install amici. pip install -U -v amici, missed -U. Please re-try.

@harshitaojha
Copy link
Author

Using pip 21.1.3 from /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-799ele2j
Created temporary directory: /tmp/pip-req-tracker-8ao97xj1
Initialized build tracking at /tmp/pip-req-tracker-8ao97xj1
Created build tracker: /tmp/pip-req-tracker-8ao97xj1
Entered build tracker: /tmp/pip-req-tracker-8ao97xj1
Created temporary directory: /tmp/pip-install-_v9nppto
Requirement already satisfied: amici in /home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages (0.11.7)
1 location(s) to search for versions of amici:

On running importer.create_problem() in the .py file, I get this:
Traceback (most recent call last):
File "/home/harshita/PycharmProjects/Project_ICL_pyPESTO/pypesto_opti.py", line 25, in
problem = importer.create_problem()
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 447, in create_problem
objective = self.create_objective(**kwargs)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 236, in create_objective
model = self.create_model(force_compile=force_compile)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 106, in create_model
if self._must_compile(force_compile):
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 144, in _must_compile
importlib.import_module(self.model_name)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 855, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/harshita/PycharmProjects/Project_ICL_pyPESTO/amici_models/Boehm_JProteomeRes2014/Boehm_JProteomeRes2014/init.py", line 7, in
raise RuntimeError('Cannot use model Boehm_JProteomeRes2014, generated with AMICI '
RuntimeError: Cannot use model Boehm_JProteomeRes2014, generated with AMICI version 0.11.7, together with AMICI version 0.11.18 which is present in your PYTHONPATH. Install the AMICI package matching the model version or regenerate the model with the AMICI currently in your path.

@dweindl
Copy link
Member

dweindl commented Jul 13, 2021

Okay, the important info is unfortunately missing here.

In which location are your HDF5 libraries installed? (if using something Debian-based: dpkg -L libhdf5-serial-dev)

RuntimeError: Cannot use model Boehm_JProteomeRes2014, generated with AMICI version 0.11.7, together with AMICI version 0.11.18 which is present in your PYTHONPATH. Install the AMICI package matching the model version or regenerate the model with the AMICI currently in your path.

That is easy to fix: Just delete /home/harshita/PycharmProjects/Project_ICL_pyPESTO/amici_models/Boehm_JProteomeRes2014 and re-run your code.

@harshitaojha
Copy link
Author

In which location are your HDF5 libraries installed?
$ dpkg -s libhdf5-dev
Package: libhdf5-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 11889
Maintainer: Ubuntu Developers [email protected]
Architecture: amd64
Source: hdf5
Version: 1.10.4+repack-11ubuntu1
Replaces: libhdf5-serial-dev (<< 1.8.12-9~)
Provides: libhdf5-serial-dev
Depends: libhdf5-103 (= 1.10.4+repack-11ubuntu1), zlib1g-dev, libjpeg-dev, libaec-dev, hdf5-helpers, libhdf5-cpp-103 (= 1.10.4+repack-11ubuntu1)
Suggests: libhdf5-doc
Breaks: libhdf5-serial-dev (<< 1.8.12-9~)
Description: Hierarchical Data Format 5 (HDF5) - development files - serial version
HDF5 is a file format and library for storing scientific data.
HDF5 was designed and implemented to address the deficiencies of
HDF4.x. It has a more powerful and flexible data model, supports
files larger than 2 GB, and supports parallel I/O.
.
This package contains development files for serial platforms.
Homepage: http://hdfgroup.org/HDF5/
Original-Maintainer: Debian GIS Project [email protected]

That is easy to fix: Just delete /home/harshita/PycharmProjects/Project_ICL_pyPESTO/amici_models/Boehm_JProteomeRes2014 and re-run your code.

Doing this, gives me the following error!

Using existing amici model in folder /home/harshita/PycharmProjects/Project_ICL_pyPESTO/amici_models/Boehm_JProteomeRes2014.
2021-07-14 16:04:17,898 - pypesto.petab.importer - INFO - Using existing amici model in folder /home/harshita/PycharmProjects/Project_ICL_pyPESTO/amici_models/Boehm_JProteomeRes2014.
Traceback (most recent call last):
File "/home/harshita/PycharmProjects/Project_ICL_pyPESTO/pypesto_opti.py", line 25, in
problem = importer.create_problem()
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 447, in create_problem
objective = self.create_objective(**kwargs)
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 242, in create_objective
edatas = self.create_edatas(
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/pypesto/petab/importer.py", line 197, in create_edatas
return amici.petab_objective.create_edatas(
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/amici/petab_objective.py", line 487, in create_edatas
edata = create_edata_for_condition(
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/amici/petab_objective.py", line 561, in create_edata_for_condition
y, sigma_y = _get_measurements_and_sigmas(
File "/home/harshita/anaconda3/envs/Project_ICL_lotkaVolterra/lib/python3.9/site-packages/amici/petab_objective.py", line 659, in _get_measurements_and_sigmas
observable_ix = observable_ids.index(measurement[OBSERVABLE_ID])
ValueError: tuple.index(x): x not in tuple

@dweindl
Copy link
Member

dweindl commented Jul 14, 2021

Doing this, gives me the following error!

See #701 (comment)

@harshitaojha
Copy link
Author

harshitaojha commented Jul 14, 2021

Doing this, gives me the following error!

See #701 (comment)

In this issue, the problem was in PEtab file which is getting generated. But here I am not getting any error on using linter(#701 (comment))

The files in this problem are different. Currently, I am using Benchmark Models Boehm JProteome problem. And the files are the same.

@harshitaojha
Copy link
Author

The only difference is I have generated a folder named "amici models" again, as suggested by you above because it was giving an error.

@harshitaojha
Copy link
Author

Screenshot from 2021-07-15 00-28-16

This is what it prints when I looked for measurement object, OBSERVABLE_ID etc.

PS: I am only trying to replicate Boehm_JProteomeRes2014, please ignore the Lotka Volterra names of dir.

@dweindl
Copy link
Member

dweindl commented Jul 15, 2021

@harshitaojha: Can you please verify that you are using the most recent pypesto/amici/petab packages and model files. If things still don't work, please post a minimal reproducible example here.

@harshitaojha
Copy link
Author

I verified the package versions, and they all were recent updates.
Now, to check if there's problem with pypesto or Boehm model, I extended the Lotka Volterra model(https://github.com/yaml2sbml-dev/yaml2sbml/blob/main/doc/examples/Lotka_Volterra/Lotka_Volterra_python/Lotka_Volterra.ipynb) and tested different optimizers mentioned in COMBINE Notebook(https://github.com/jvanhoefer/COMBINE-2020-talk/blob/master/jupyter%20notebook/pypesto_talk_COMBINE_2020.ipynb).

It worked and I got these plots, attaching them below for your ref.
image

Finally, I change the odes, parameters and the data for the project I am working on, I get the following error.
Traceback (most recent call last):
File "/home/harshita/PycharmProjects/LV/main.py", line 26, in
problem = importer.create_problem()
File "/home/harshita/anaconda3/envs/LV/lib/python3.9/site-packages/pypesto/petab/importer.py", line 447, in create_problem
objective = self.create_objective(**kwargs)
File "/home/harshita/anaconda3/envs/LV/lib/python3.9/site-packages/pypesto/petab/importer.py", line 236, in create_objective
model = self.create_model(force_compile=force_compile)
File "/home/harshita/anaconda3/envs/LV/lib/python3.9/site-packages/pypesto/petab/importer.py", line 109, in create_model
self.compile_model(**kwargs)
File "/home/harshita/anaconda3/envs/LV/lib/python3.9/site-packages/pypesto/petab/importer.py", line 166, in compile_model
amici.petab_import.import_model(
File "/home/harshita/anaconda3/envs/LV/lib/python3.9/site-packages/amici/logging.py", line 194, in wrapper_timer
rval = func(*args, **kwargs)
File "/home/harshita/anaconda3/envs/LV/lib/python3.9/site-packages/amici/petab_import.py", line 503, in import_model_sbml
free_syms = sorted(sp.sympify(formula).free_symbols,
AttributeError: 'function' object has no attribute 'free_symbols'

I have emailed you and @yannikschaelte the necessary files to reproduce the model I was working on. Sorry, sharing it here is not under my purview.

@dweindl
Copy link
Member

dweindl commented Jul 21, 2021

File "[...]/site-packages/amici/petab_import.py", line 503, in import_model_sbml
free_syms = sorted(sp.sympify(formula).free_symbols,
AttributeError: 'function' object has no attribute 'free_symbols'

I have emailed you and @yannikschaelte the necessary files to reproduce the model I was working on. Sorry, sharing it here is not under my purview.

Reporting the result here for future reference: The problem was a parameter named N in the observable formula which is interpreted as sympy.core.evalf.N. This needs to be fixed in AMICI and probably libpetab-python. Furthermore, math grammar needs to be defined properly for PEtab (PEtab-dev/PEtab#522).

(This is independent of the opening post on pickling and HDF5 support).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants