Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend file access to event streaming and data slicing #8

Open
vkuznet opened this issue Mar 24, 2017 · 0 comments
Open

Extend file access to event streaming and data slicing #8

vkuznet opened this issue Mar 24, 2017 · 0 comments

Comments

@vkuznet
Copy link
Owner

vkuznet commented Mar 24, 2017

So far the transfer are file based. We need an R&D for root IO event streaming and data slicing in a context of CWP:

In current paradigms, physicists consider all events at an equivalent level of detail and in the format offering the highest level of detail that needs to be considered in an analysis. However, not every event considered in analysis requires the same level of detail. One consideration to improve I/O throughput is to design event tiers with different sizes. All events can be considered at a lighter-weight tier while events of interest only can be accessed with a more information-rich tier. Machine learning and similar highly-iterative techniques also provide new challenges in data management and access. Data access that is column-based instead of row-based can be of enormous benefit to many analysis techniques. For instance, instead of reading events in sequence and searching for one particular type of object in the events, to have the data management system be able to return all objects of a certain type in a dataset. Data queries that consider histogram indexing is another feature that may or may not provide performance increases in analysis.

Go based root IO https://godoc.org/go-hep.org/x/hep/rootio

@vkuznet vkuznet added this to the July development milestone May 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant