DPL Analysis: Table definition rewrite #13664

aalkin · 2024-11-07T12:16:37Z

Draft until AliceO2Group/O2Physics#8322 is merged.

github-actions · 2024-11-07T12:16:50Z

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1

DataFormats/simulation/include/SimulationDataFormat/ConstMCTruthContainer.h

Detectors/AOD/include/AODProducerWorkflow/AODProducerHelpers.h

Framework/Core/include/Framework/ASoAHelpers.h

Framework/Core/include/Framework/AnalysisDataModel.h

Framework/Core/include/Framework/AnalysisHelpers.h

Framework/Core/include/Framework/AnalysisManagers.h

ktf · 2024-11-08T12:11:03Z

I have done a few comments. As discussed privately yesterday, we should try to minimise the differences and introduce "safe" / "unrelated" changes as soon as possible, to minimise the diff. In particular I think:

We can introduce all the new classes / structs / concepts as long as they do not get used by the current code.
We can introduce safe standalone concept if they are already applicable now, and simply use them immediately (e.g. soa::is_table).
We can extend the macro to have an extra field (even if unused).

ktf · 2024-11-21T08:48:19Z

Is this ready now?

aalkin · 2024-11-21T09:00:42Z

There is a bit of an issue with O2Physics (hence the latest reverts), but otherwise it is functional.

Framework/Core/include/Framework/ASoA.h

alibuild · 2024-11-21T11:34:19Z

Error while checking build/O2/fullCI for d557502 at 2024-11-21 12:34:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:491:9: error: no type named 'ArrowType' in 'struct o2::framework::detail::ConversionTraits<std::array<float, 3> >'
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:492:9: error: no type named 'ArrowType' in 'struct o2::framework::detail::ConversionTraits<std::array<float, 3> >'
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:505:32: error: no type named 'ArrowType' in 'struct o2::framework::detail::ConversionTraits<std::array<float, 3> >'
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:596:33: error: call of overloaded 'policy()' is ambiguous
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:257:9: error: no type named 'ArrowType' in 'struct o2::framework::detail::ConversionTraits<std::array<float, 3> >'
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:258:9: error: no type named 'ArrowType' in 'struct o2::framework::detail::ConversionTraits<std::array<float, 3> >'
/sw/SOURCES/O2/13664-slc8_x86-64/0/Framework/Core/include/Framework/TableBuilder.h:260:39: error: no type named 'ArrowType' in 'struct o2::framework::detail::ConversionTraits<std::array<float, 3> >'
ninja: build stopped: subcommand failed.

Full log here.

alibuild · 2024-11-21T15:42:53Z

Error while checking build/O2/fullCI for 4ee33f4 at 2024-11-21 16:42:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2Physics-latest/log
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1700:47: error: template argument 1 is invalid
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1676:38: error: could not convert 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::FwdTrackId, o2::aod::ambiguous::BCIdSlice>()' from 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::FwdTrackId, o2::aod::ambiguous::BCIdSlice>' to 'o2::soa::DefaultIndexPolicy'
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1700:47: error: template argument 1 is invalid
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1676:38: error: could not convert 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::MFTTrackId, o2::aod::ambiguous::BCIdSlice>()' from 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::MFTTrackId, o2::aod::ambiguous::BCIdSlice>' to 'o2::soa::DefaultIndexPolicy'
ninja: build stopped: subcommand failed.


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/e8a55538dffd4b4e6615a3317dddc7dab1093b85/slc8_x86-64/o2checkcode/1.0-local583/etc/modulefiles
++ cat
--

Full log here.

alibuild · 2024-11-22T11:23:02Z

Error while checking build/O2/fullCI for 0d2761f at 2024-11-22 12:23:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2Physics-latest/log
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1702:47: error: template argument 1 is invalid
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1678:38: error: could not convert 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::FwdTrackId, o2::aod::ambiguous::BCIdSlice>()' from 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::FwdTrackId, o2::aod::ambiguous::BCIdSlice>' to 'o2::soa::DefaultIndexPolicy'
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1702:47: error: template argument 1 is invalid
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local1/include/Framework/ASoA.h:1678:38: error: could not convert 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::MFTTrackId, o2::aod::ambiguous::BCIdSlice>()' from 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::MFTTrackId, o2::aod::ambiguous::BCIdSlice>' to 'o2::soa::DefaultIndexPolicy'
ninja: build stopped: subcommand failed.


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/0a6e89f30ca1d463dabad114d8aa89b3c3c2319f/slc8_x86-64/o2checkcode/1.0-local592/etc/modulefiles
++ cat
--

Full log here.

alibuild · 2024-11-22T14:38:45Z

Error while checking build/O2/fullCI for 94344f6 at 2024-11-27 10:28:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2Physics-latest/log
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local2/include/Framework/ASoA.h:1702:47: error: template argument 1 is invalid
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local2/include/Framework/ASoA.h:1678:38: error: could not convert 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::FwdTrackId, o2::aod::ambiguous::BCIdSlice>()' from 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::FwdTrackId, o2::aod::ambiguous::BCIdSlice>' to 'o2::soa::DefaultIndexPolicy'
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local2/include/Framework/ASoA.h:1702:47: error: template argument 1 is invalid
/sw/slc8_x86-64/O2/13664-slc8_x86-64-local2/include/Framework/ASoA.h:1678:38: error: could not convert 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::MFTTrackId, o2::aod::ambiguous::BCIdSlice>()' from 'o2::framework::pack<o2::soa::Index<0, -1>, o2::aod::ambiguous::MFTTrackId, o2::aod::ambiguous::BCIdSlice>' to 'o2::soa::DefaultIndexPolicy'
ninja: build stopped: subcommand failed.

Full log here.

alibuild · 2024-12-03T11:09:24Z

Error while checking build/O2/fullCI for 9c625d2 at 2024-12-03 12:09:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2Physics-latest/log
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:23: error: 'is_soa_join_v' is not a member of 'o2::soa'
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:38: error: expected primary-expression before '>' token
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:39: error: expected primary-expression before ')' token
ninja: build stopped: subcommand failed.

Full log here.

alibuild · 2024-12-03T12:29:45Z

Error while checking build/O2/fullCI for d75c13a at 2024-12-03 13:29:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2Physics-latest/log
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:23: error: 'is_soa_join_v' is not a member of 'o2::soa'
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:38: error: expected primary-expression before '>' token
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:39: error: expected primary-expression before ')' token
ninja: build stopped: subcommand failed.

Full log here.

alibuild · 2024-12-03T13:41:53Z

Error while checking build/O2/fullCI for 9c86a2c at 2024-12-03 14:41:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2Physics-latest/log
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:23: error: 'is_soa_join_v' is not a member of 'o2::soa'
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:38: error: expected primary-expression before '>' token
/sw/SOURCES/O2Physics/13664-slc8_x86-64/0/PWGMM/Mult/Tasks/dndeta-hi.cxx:137:39: error: expected primary-expression before ')' token
ninja: build stopped: subcommand failed.

Full log here.

ktf · 2024-12-03T16:26:17Z

Framework/Core/include/Framework/ASoA.h

 struct Binding {
  void const* ptr = nullptr;
  size_t hash = 0;
+  std::span<TableRef const> refs;


Not for this PR, however why do you need a member for it? Can't you just have a function?

When binding is created I can only store some aspects of the type, since binding itself is not templated. For now hash is enough, but I have some ideas to improve both setting and checking the index binding with this.

ktf · 2024-12-03T16:27:35Z

Framework/Core/include/Framework/ASoA.h

  decltype(auto) getDynamicBinding()
  {
-    static_assert(std::is_same_v<decltype(&(static_cast<B*>(this)->mColumnIterator)), std::decay_t<decltype(B::mColumnIterator)>*>, "foo");
+    static_assert(std::same_as<decltype(&(static_cast<B*>(this)->mColumnIterator)), std::decay_t<decltype(B::mColumnIterator)>*>, "foo");


This shouuld be moved in the requirements in the next iteration.

ktf · 2024-12-03T16:27:53Z

Framework/Core/include/Framework/ASoA.h

    return &(static_cast<B*>(this)->mColumnIterator);
-    //return static_cast<std::decay_t<decltype(B::mColumnIterator)>*>(nullptr);
+    // return static_cast<std::decay_t<decltype(B::mColumnIterator)>*>(nullptr);


Drop the comment in the next iteration.

ktf · 2024-12-03T16:35:01Z

Thank you, amazing work. I tested on hyperloop and it seems to work as before. I will wait for the tests and then we can merge.

That said, how come we have 30% more code after the rewrite? I thought the whole goal was to have less code to review, not more. Is there any other cleanup we could do that was hold back? Did you check the compile times?

aalkin · 2024-12-03T21:33:06Z

Most of the extra code comes from separating pieces with a lot of if constexpr branching into restricted functions. In addition, I had to get creative with the table declaration macros in order to not break the data model, they are layered to avoid redeclaring the hashes. Then there is code to handle the static strings in hashes and a new test. The rest is more or less a replacement and I believe I have missed some commented out code still. But after this is merged, I can start removing compatibility workarounds, which requires an additional pass on O2Physics.

I have not measured the compile time for O2Physics, for O2 itself it seems to be only slightly less, since not many things depend on ASoA.

ktf · 2024-12-04T07:32:33Z

Sounds good. Let's merge this and start removing what can be removed.

aalkin requested a review from ktf November 7, 2024 12:16

alibuild mentioned this pull request Nov 7, 2024

Please consider the following formatting changes to #13664 aalkin/AliceO2#62

Merged

aalkin force-pushed the table-definition-rewrite branch from ceb880d to 6701988 Compare November 8, 2024 09:11