Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allowing alternative InferenceData/Dataset storage #16

Open
sethaxen opened this issue Aug 29, 2022 · 1 comment
Open

Allowing alternative InferenceData/Dataset storage #16

sethaxen opened this issue Aug 29, 2022 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@sethaxen
Copy link
Member

Currently all Datasets wrap a Dimstack, which in turn wraps a set of NamedTuples. Likewise, InferenceData wraps a NamedTuple of Datasets. As noted in #15, this makes everything but the array values themselves immutable.

An alternative is to decouple the interface for Dataset and InferenceData from the storage. e.g. we could define NamedTupleStore, DictStore, NCDatasetStore, etc and implement the InferenceData/Dataset interfaces for each of these. This would allow users to strongly type everything with NamedTupleStore if they want to, or they could use a DictStore to have much more dynamic access. Having a NCDatasetStore would allow users to open a NetCDF file as an InferenceData and even incrementally write to such an InferenceData, having the NetCDF file automatically updated.

With a well-designed API for stores, this would actually look a lot like the InferenceData API proposed in arviz-devs/ArviZ.jl#154. Namely, the store API could be implemented for a type like MCMCChains.Chains, and then one could call InferenceData(chains) to (inefficiently) view it as an InferenceData. When efficiency is needed, one just converts to one of the native stores.

I started work on a small prototype of this. It's a fair amount of surgery to the existing code and increases the code complexity. Though it will be more work to do it then than now, I think it's better to hold off on this until later.

@sethaxen sethaxen self-assigned this Oct 29, 2022
@sethaxen sethaxen mentioned this issue Nov 12, 2022
@sethaxen sethaxen added the enhancement New feature or request label Feb 17, 2023
@sethaxen
Copy link
Member Author

This is closely related to rafaqz/DimensionalData.jl#473 and probably mostly depends on something like that being implemented in DimensionalData first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant