pick up behavior without slice after setting parameters manually

[lukasheinrich]

I'm working on a awkward-array integration with vector and awkward for pylhe

This is useful exercise for building up peharps a more complete event data model for ATLAS as well (@nikoladze @matthewfeickert @kratsg @gordonwatts @alexander-held @henryiii )

The raw arry (ignoring references/links that come layer is this (filtering for briefness)

>>> events = ak.from_iter({
  'particles': [
      {k if k!= 'e' else 'E':getattr(p,k) for k in pylhe.LHEParticle.fieldnames if k in ['px','py','pz','e']} for p in e.particles 
  ],
  'eventinfo':  {k:getattr(e.eventinfo,k) for k in pylhe.LHEEventInfo.fieldnames if k in ['nparticles','weight']}
  }
  
  for e in pylhe.readLHEWithAttributes(skhep_testdata.data_path('pylhe-testfile-pr29.lhe'))
)
>>> ak.type(events)
791 * {"particles": var * {"px": float64, "py": float64, "pz": float64, "E": float64}, "eventinfo": {"nparticles": float64, "weight": float64}}

and layout

>>> events.layout.contents[0].content.setparameter('__record__','Particle')
... events.layout.contents[1].setparameter('__record__','EventInfo')
... events.layout.setparameter('__record__','Event')
... ak.type(events)
791 * Event["particles": var * Particle["px": float64, "py": float64], "eventinfo": EventInfo["nparticles": float64]]

I can now give names to the records via
which captures the nesting:

• The Datatset is an array of events
• The Event consists of EventInfo and an variable array of Particles
• Particles consist of momentum components

registering the following dummy behaviors

class Particle(ak.Record):
    pass

class ParticleArray(ak.Array):
    pass

class Event(ak.Record):
    pass

class EventArray(ak.Array):
    pass

class EventInfo(ak.Record):
    pass

class EventInfoArray(ak.Array):
    pass

ak.behavior['Particle'] = Particle
ak.behavior['*','Particle'] = ParticleArray

ak.behavior['Event'] = Event
ak.behavior['*','Event'] = EventArray

ak.behavior['EventInfo'] = EventInfo
ak.behavior['*','EventInfo'] = EventInfoArray

I can get any slice into the array to instantiate

>>> events[:].particles
<ParticleArray [[{px: -0.315, py: -0.63, ... E: 9.35}]] type='791 * var * Partic...'>

>>> events[0]
<Event ... nparticles: 9, weight: 0.0751}} type='Event["particles": var * Partic...'>

and the trivial slice gives me an EventArray

>>> events[:]
<EventArray [{particles: [, ... weight: 1.29e-05}}] type='791 * Event["particles...'>

** but ** printing out events on its own doesn't pick up the toplevel behavior

>>> events
<Array [{particles: [, ... weight: 1.29e-05}}] type='791 * Event["particles": va...'>

is there a way to "re-instantiate" the toplevel object so that it also becomes an EventArray?

[lukasheinrich]

the solution seems to be to either do

ak.Array(events) which should be zero-copy or to events[:] as above

[lukasheinrich]

thanks - I'll mark this as an answer!

[jpivarski]

Oh, I read this, but I didn't read it carefully enough: don't use setparameters directly, as there's an ancient issue to remove it: #177. Use the high-level function ak.with_parameter, which would give you a new object anyway.

I had misread this, thinking that you changed the ak.behavior after creating the array, which is another way you can get into this situation.

[lukasheinrich]

how would I achieve the scenario above where I need to set multiple parameters at different nesting levels?

[jpivarski]

For something as complex as this, the easiest way would be to build the whole structure from ak.layout.Content subclass instances, which each have a parameter argument to their constructors (instead of building the thing and then having to dig down to find it again).

Also, you should be aware that ak.from_iter is the slow path, especially if the data start as arrays (I suppose they don't in your case because you have to parse LHE files as text).

I think you want to have 6 growable arrays (a class that allocates a NumPy array and a pointer to "next entry," fills that array until it is full, then allocates and copies data to a new NumPy array that is 1.5× larger):

• entry offsets: integers starting with 0 (one longer than the number of events), collecting a cumulative sum of the number of particles per event
• px, py, pz, E, all as flat (no event boundaries) floating-point arrays,
• weight as a floating-point array (no need for nparticles, as users can get that using ak.num).

Then you can put these together as flat (no event boundaries) record arrays with the ak.layout.RecordArray constructor and break that into lists by wrapping it in an ak.layout.ListOffsetArray64. Adding the eventinfo is another ak.layout.RecordArray around that. By building it up in pieces, you'll have an opportunity to set the parameters at each stage.

Having said all of that, I just noticed that you can also use ak.ArrayBuilder, passing a name argument to the begin_record or record method. That would certainly be simpler, though ak.ArrayBuilder is the performance cost hidden inside ak.from_iter (it has to iterate over Python objects—that's the problem, which might not matter for you because you're coming from text files, anyway).

[lukasheinrich]

yeah the speed is probably limited by text IO/XML parsing but this seems to work nicely

builder = ak.ArrayBuilder()
for e in pylhe.readLHEWithAttributes(skhep_testdata.data_path('pylhe-testfile-pr29.lhe')):
    with builder.record(name = 'Event'):
        builder.field('eventinfo')
        with builder.record(name =  'EventInfo'):
            for fname in e.eventinfo.fieldnames:
                builder.field(fname).real(getattr(e.eventinfo,fname))
        builder.field('particles')
        with builder.list():
            for p in e.particles:
                with builder.record(name = 'Particle'):
                    builder.field('px').real(1.2)
arr = builder.snapshot()
arr