ESDL Compatibility

[suvayu]

1. What does a TEM solution add to the input dataset?
   When working from an ESDL file, TEM gets the graph structure, Asset/Flow attributes, and profiles from ESDL. After solving, what are the attributes that the solution would like to add back to the ESDL? E.g.:

• new Asset nodes?
• new connections between assets (Flow)?
• new attributes for assets?
• new attributes for flows?
• new profiles?

[clizbe]

2. Multiple-in-multiple-out conversions
   How are efficiencies assigned?

3. Consumer/Hub - separate or one asset
   Consumers can't have outflow?

4. General workflow using MapEditor
   Can we do "scenarios" in MapEditor? Do we need to?
   What is goes back to ESDL as output?
   Does ESDL do versioning? What if I want to rerun an old analysis?

5. How to handle the difference between ESDL files that work for Tulipa and the "standard" method of modelling

6. Will MapEditor be able to handle an OPERA scenario? (Imagine adding all Options)

[g-moralesespana]

• We should start identifying the must have vs. the nice to have features. We should not spend too much effort working in Tulipa-ESDL just to be able to charge a multiyear OPERA scenario, which can be part of another project.
• The most important part is that we can handle all the data needs correctly without depending on ESDL, that is how we will start using it anyway. Tulipa should be compatible with ESDL but it should not depend on ESDL.

[abelsiqueira]

We have three steps to use ESDL for small and large models, as briefly discussed (and tabled today):

• Convert from ESDL to Tulipa's data source. Whether it should be just DuckDB created via Python or if a static file option should be available is still up to discussion
• Get the solution from Tulipa's execution and add it to ESDL for visualisation purposes.
• Get the data from a model and generate fake/random "metadata" to create an ESDL.

This should allow these two workflows:

Small model:

• Create model on ESDL
• Convert data (using Python) to be accessed by Tulipa
• Run Tulipa and get solution
• Merge solution and ESDL (or create new ESDL?) to visualize

Large model

• Load existing data into DuckDB in a Tulipa-friendly format
• Run Tulipa and get solution
• Create ESDL with fake metadata
• Merge solution and ESDL to visualise

Notes:

• The creation of ESDL with fake metadata can happen earlier

[clizbe]

I think we need to take a good look at the pros/cons of ESDL, and maybe also break it down by different levels of implementation.

[clizbe]

I've messaged Joris Koornneef to find out why ESDL-compat is so important to him, what he wants to gain, what kind of compat, etc.

[clizbe]

Here's his response:

Advantages:
Building a broader community around data! That is the core of your tool. The formulas are of course important, but according to the principle: shit in shit out, your model is as good as your data. And transparency of your data is a must for future modeling. And if you are compatible, you can use data that other models also use and even link input-output much more easily so that you can handle more complex matters in the future.

You may have found disadvantages regarding formatting and categorization of ESDL that may not fit. But sleeves can be adjusted. (Direct translation of a Dutch expression.)

However, I am not a model expert so I say these things from a role that I often represent by explaining scenarios to stakeholders (governments, companies) and putting them in context. Then transparency and compatibility are a great asset!

[clizbe]

After another talk with Joris I've learned these are valuable for him/consultants:

• Spatial representation of the system
• Can work in a big group on one scenario because it's in the cloud
• Scenario transparency because you can click on stuff and show details of what's happening
• Map view of results
• Common data through EDR, as well as sending data between models (although this hasn't really been proven yet)

I'm actually a bit more convinced that the MapEditor can handle our big scenarios and would be useful. Whether we build a couple large scenarios and keep updating them, or build them from a script to begin and then update.
I think maybe it's more a problem that we haven't worked with the MapEditor enough to really fall in love with it.

[clizbe]

Also a map-based network builder is quite nice to have and sets us apart from some other models.

So we should probably discuss further, but at this point my vote is for input & output compatibility. For the first step just having an ugly ESDL and maybe working on a "standard ESDL to ugly Tulipa ESDL" converter later.

[clizbe]

Putting this here for future reference and closing the discussion for now:

Here’s a quick breakdown of the expected gains and the problems we faced:

Gain: Easily create systems in the MapEditor

Problem 1: The way the MapEditor is deployed, profiles can only be extracted from the ESDL file if they are linked from the EDR. Since the EDR is controlled by TNO, this causes multiple issues: it goes against our open-source principles, it creates a bottleneck of uploading data to the EDR, and it limits outside use and collaboration. This is a no-go for us. If map visualization becomes a major priority, we’ll likely use other tools to implement it.
Problem 2: As Wester mentioned, our use cases are large energy systems that would be time-consuming to implement in the MapEditor, when they can easily be implemented in tabular format. The MapEditor would be handy for other users once they’ve learned it, but that is complicated for Tulipa users because of Problems 3 and 4.
Problem 3: The MapEditor assumes a different system representation than Tulipa – converting is complicated to implement (though we have a basic implementation) and would be confusing to users who have to learn both ESDL/MapEditor and Tulipa and how they interact.
Problem 4: Significant additional data is necessary for running Tulipa, so having the user go specifically to the MapEditor to create the system, then export the ESDL file, then create other files for the rest the data, then run the model – adds unnecessary complication to the analysis pipeline.
Gain: Compatibility with other models

Problem 5: Every model uses ESDL differently, so even if both models are “ESDL compatible,” this does not mean they are interoperable. Especially because of Problem 3, Tulipa’s ESDL files would be difficult to make compatible with other models. Instead, Tulipa will have a clean and well-documented data format to facilitate interoperability.