Reading Allpix² files - uproot.interpretation.objects.CannotBeAwkward: arbitrary pointer

[prakhub]

Hi

I am currently trying to read some ROOT files generated in Allpix² and am experimenting with uproot in order to keep the analysis code more concise and "pythonic". The file I am trying to read (link, rename data.txt to data.root) consist of a TTree with a TBranch ("DepositedCharge") storing std::vector<allpix::DepositedCharge*> objects (see below)

>>> import uproot
>>> branch = uproot.open('data.root:DepositedCharge/mydetector')
>>> branch.show()
name                 | typename                 | interpretation                
---------------------+--------------------------+-------------------------------
mydetector           | std::vector<allpix::Depo | AsObjects(AsVector(True, AsPoi
>>> branch.typename
'std::vector<allpix::DepositedCharge*>'
>>> branch.interpretation
AsObjects(AsVector(True, AsPointer(Model_allpix_3a3a_DepositedCharge)))

When trying to read the branch into an array I get the following error:

>>> branch = uproot.open('data.root:DepositedCharge/mydetector')
>>> branch_arrays = branch.arrays(library="ak")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 1059, in arrays
    return self.parent.arrays(
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 1125, in arrays
    _ranges_or_baskets_to_arrays(
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 3502, in _ranges_or_baskets_to_arrays
    uproot.source.futures.delayed_raise(*obj)
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/source/futures.py", line 36, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 3446, in basket_to_array
    basket_arrays[basket.basket_num] = interpretation.basket_array(
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/interpretation/objects.py", line 140, in basket_array
    form = self.awkward_form(branch.file, index_format="i64")
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/interpretation/objects.py", line 120, in awkward_form
    return self._model.awkward_form(
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/containers.py", line 779, in awkward_form
    uproot._util.awkward_form(
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/_util.py", line 555, in awkward_form
    return model.awkward_form(
  File "/home/hop/anaconda3/envs/hephy/lib/python3.10/site-packages/uproot/containers.py", line 551, in awkward_form
    raise uproot.interpretation.objects.CannotBeAwkward("arbitrary pointer")
uproot.interpretation.objects.CannotBeAwkward: arbitrary pointer

If I use numpy instead of awkward (branch.arrays(library='np')) then the file can be read without error:

>>> branch = uproot.open('data.root:DepositedCharge/mydetector')
>>> branch_arrays = branch.arrays(library="np")
>>> branch_arrays
array([<STLVector [<allpix::DepositedCharge (version 4) at 0x7f7ac0ff0d90>, ...] at 0x7f7ac0ff0e50>,
       <STLVector [<allpix::DepositedCharge (version 4) at 0x7f7ab9bae620>, ...] at 0x7f7ab9bdaf20>,
        ...
       <STLVector [<allpix::DepositedCharge (version 4) at 0x7f7ab9bea0e0>, ...] at 0x7f7ab9be9ff0>,
       <STLVector [<allpix::DepositedCharge (version 4) at 0x7f7ab9bfebc0>, ...] at 0x7f7ab9bfead0>],
       dtype=object)
>>>

However, reading the array is extremly slow (~5s for ~0.5MB data).

What do I need to fix in order to use awkward (which hopefully speeds up the execution time)?

For the <allpix::DepositedCharge> objects, header files are available at DepositedCharge.hpp.
I wrote code aimed at the same goal using PyROOT where I load the object definitions from a shared library (using gSystem.Load("/path/to/allpix-squared/lib/libAllpixObjects.so").
Is there some way to pass the same information to uproot?

[jpivarski]

If the object you want to read has pointers within it, then it does not fit in Awkward Array's type system. C++ objects are arbitrary graphs, but objects in Awkward Arrays must be trees.

You may be able to read this with library="np", since that turns the data into Python objects (slowly). Python objects are also arbitrary graphs.

You can do it with PyROOT as well by giving ROOT the shared library describing the C++ classes that it should fill (libAllpixObjects.so). This is because PyROOT objects are just proxies pointing to C++ data. That makes the PyROOT solution a little different from Uproot's library="np": since Uproot's objects are pure Python, they should be pickleable and I expect that the PyROOT objects are not, but this difference might not be relevant to you. Neither Uproot's library="np" nor PyROOT are very fast because Python is in the loop in any iteration over these objects.

Uproot can't make use of any C++ libraries (like libAllpixObjects.so) because it doesn't run any C++ code. Awkward Array wouldn't be able to use it, either, because its data model is completely different from C's notion of a struct: Awkward Arrays are columnar (a different memory layout), and compiled C++ code specifies a particular memory layout.

In the end, there isn't much that can be done with this data type because it has cross-linking pointers. Awkward Array generalized arrays beyond just numbers, but it hasn't generalized it to the extent of supporting cross-links. (There were some attempts to do that in Awkward 0.x, but they were fragile—didn't work very well—and not in high demand.) This sounds like a perfect job for Julia and UnROOT, but then your project wouldn't be "Pythonic" so much as "Julionic." (What's the right word?)

Note: I don't know if UnROOT can read general data types like this one; I'm just saying that it fits the Julia type system, as Julia has an ordinary language's type system and not a specialized DSL for array programming in an interpreted language.

[prakhub]

Hi @jpivarski
Thank you very much for this in-depth Answer!

I will have a look at UnROOT, it seems that there is already some (experimental) support for using custom structs.
But I guess at the end of the day, it is probably easiest to just iterate over the file using ROOT itself and output the data in a new Tree or some other data format entirely.