Implementation of Dataplane that supports Non-Finite Provider Push Transfers

[gerbigf]

Overview

Implements #938

Explain the topic in 2 sentences

A Dataspace data consumer should be able to subscribe to a data offering that provides a "stream" of non-finite data transfers while the agreed contract terms are valid. Messages should be exchanged via HTTP (not kafka)

What's the benefit?

We reduce the complexity of setting up use cases that rely on frequent, provider triggered, transfers of data.

Furthermore, we correct the current anti-pattern that is used to PUSH data via HTTP to a partner: To receive data from a Company B, a Company A has to set up an asset that points to a REST API that shall receive the data. Company A has to act as the "data provider" in the EDC sense because he sets up the contract including that asset. Company B acts as the "data consumer" in the EDC sense because he needs to query the catalogue of A, negotiate a contract and then can use the EDR to HTTP POST or PUT information against A's (the data provider in EDC speak) data plane.

What are the Risks/Dependencies ?

Note: Its potentially breaking change and should allow use cases to be redesigned in a different, more efficient way.
We're working on a solution where both transfer types can exist in parallel.

Detailed explanation

Current implementation

Considering the given scenario 1.

Company A generates GBs of data on daily basis. Company B wants to consume that data as soon as it is generated.

Currently, Company B has to create a data offering that allows Company A to notify B when a new data was generated and where it can be found. This requires a protocol to be agreed between both companies, so the details of said notification and following steps are understood.

Once company B receives the notification, it triggers a search, negotiation and transfer of that particular data.

This is designed as such because the EDC closes the forward channel on any provider push transfer scenario, as soon as the transfer is either successful / failed.

Scenario 2.
See the anti-pattern in "what's the benefit".

Proposed improvements

This shall introduce a concept for a dataplane that is capable of keeping the forward channel open, and of handling and recovering from data transfer errors, while the agreed contract terms are valid.

Feature Team

Contributor

• @correiaafonso12
• @gerbigf

Committer

• @rafaelmag110

User Stories

• Issue 1, linked to specific repository
• Issue 2, linked to another specific repository

Acceptance Criteria

• A Data Consumer can subscribe to a data offer
• A Data Consumer can receive data updates from his subscription, when the data provider adds new data to the source

Test Cases

Test Case 1

A Data Consumer can subscribe to a data offer

Steps

Given a consumer and a provider exist.
Given a provider has a dataset which we can add new data to.
Given the provider has an offer for this dataset

1. The consumer finds, negotiates access to the providers dataset.
2. Using the obtained agreement, the consumer initiates a provider push data transfer for the respective dataset.

Expected Result

The data transfer stays active as long as the terms defined in the agreed contract are valid.

Test Case 2

Steps

Given a consumer is subscribed to a data offer from a provider.

A data provider add new data to its data source.

Expected Result

1. The consumer eventually receives the data in its data destination

Architectural Relevance

The following items are ensured (answer: yes) after this issue is implemented.

In the context of the standards 126 and 127, typically only one is applicable, depending on the specific use case. Please cross out one of the two standards that does not apply.

• This feature aligns with our current architectural guidelines
  • Data Sovereignty: All data sharing activities across company boundaries follow the Catena-X Regulatory Framework, in particular the Data Exchange Governance, and the Dataspace Protocol via a compliant Connector (like the tractusx-edc or similar, see Connector KIT)
    • CX-0010 Business Partner Number
    • CX-0013 Identity of Member Companies
    • CX-0018 Data Space Connectivity (EDC)
    • CX-0049 DID Document Schema
    • CX-0050 Framework Agreement Credential
    • CX-0149 Verified Company Identity
  • Interoperability: Digital Twins are used (compliant to the Digital Twin KIT and the Industry Core KIT)
    • CX-0001 EDC Discovery API
    • CX-0002 Digital Twins in Catena-X
    • CX-0018 Data Space Connectivity (EDC)
    • CX-0126 Industry Core: Part Type 2.0.0
    • CX-0127 Industry Core: Part Instance 2.0.0
  • Data Format:
    • The data model is based on a published Semantic Model
• The impact on the overall system architecture has been assessed. The Feature does not require changes to the architecture or any existing standard? Please have a look here on the overarching architecture
• Potential risks or conflicts with existing architecture has been assessed

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification)

Additional information

• I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)

[gerbigf]

@stephanbcbauer can you add the R25.06 and R25.09 label to that? We were missing the implementation of #938

[rafaelmag110]

We can probably have this feature as a non-breaking change, where both finite and non-finite transfers are possible. Finite transfers would be the default.

[gerbigf]

Thanks @rafaelmag110 - added that to the description.

[stephanbcbauer]

@stephanbcbauer can you add the R25.06 and R25.09 label to that? We were missing the implementation of #938

@gerbigf Not sure if we mention the same label :) I added Prep-R25.06, this means this features needs to be presented during Planning Days

[gerbigf]

That's perfect. I want to have visibility of this feature because it's important ;)

[stephanbcbauer]

@gerbigf Could we also define a roadmap item for this? What is the “customer need”? Which “pain is released” with this? If there is nothing ... also fine for me :)

[gerbigf]

Uhm, sure.

In simple terms: "We are trying to remove the current misuse of the EDC that we call "Consumer Push" where a Data Provider actually only provides an Asset to be able to receive data via HTTP POST". This implementation will make things right again.

A data provider offers the actual data asset and the consumer can specify an API endpoint where he wants that data to be delivered.

Does that make sense? It's specific EDC terminology but it will have a huge impact on compliance in the data space.

[lgblaumeiser]

Hi @gerbigf @rafaelmag110 I was just thinking, for http we only have a PULL transfer, not a PUSH transfer, because there are only very limited use cases for a one time push contract, right. The only thing that came to my mind is kind of an order for data that is then pushed as soon as it is available. But if we provide a infinite PUSH, it would be a straightforward thing to do a http PUSH that automatically is "infinite", as it allows to open up transfer processes as long as the contract has not expired, right.

Thinking this further, it could be the dataspace vehicle to a general publish subscribe as you could also provide a http PUSH setup which requires a backend to push to with a Kafka or corresponding backend operated on consumer side, so a plain publish subscribe setup would be really redundant, right? What do you think @giterrific ?

Do you consider that in your approach? Or is it basically based on a bucket PUSH transfer?