This repository contains a software development kit (SDK) to track data provenance across mutations, derive related annotations, and provide a generic architecture to assess those annotations to reach one or more conclusions.
- Installation
- Getting Started
- Basic SDK Usage
- Example Code
- Project Structure
- Opportunities for Improvement
This repository contains a module written in Golang.
It assumes a minimum Go version of 1.13.
To add it to your project: go get github.com/project-alvarium/go-sdk
Defines the terms used to communicate SDK concepts (which are echoed in implementation).
- Annotation. Information (metadata) derived from (and related to) specific data.
- Annotator. Evaluates data to create annotations.
- Assessment. Conclusion(s) derived from specific data and related annotations.
- Assessor. Assesses annotations and data to create an assessment.
- Data. Plural; two or more data points.
- Data Lineage. Description of the lifecycle of data to the current time. May Include details about the data's origin, who/what accessed or mutated the data, and where the data has transited. This is a key attribute of data provenance (that is, why-provenance).
- Data Point. Singular; a piece of information.
- Data Provenance. A historical record of the data and its origins.
- Identity. A specific data's unique identifier.
- Identity Provider. Derives identity from data.
- Publisher. Publishes annotations to external systems or persistence.
The SDK tracks the provenance of any type of data (for example, JSON, XML, or binary content) as long as the data can be represented as a Golang []byte and be reduced to an identity.
Identity can be derived algorithmically (for example, the result of a hash function applied to the data) or from introspection (for example, use of a known unique identifier within the data).
The SDK defines an identity abstraction to encapsulate data identity.
The SDK defines an identity provider abstraction to encapsulate the creation of identity from data.
The SDK tracks annotations associated with data via the data's identity.
As implemented, the SDK does not modify or maintain a copy of the data it tracks. There is nothing to prevent a custom annotator from keeping a copy of the data it tracks. The annotators currently provided with the SDK do not do so.
The SDK revolves around the concept of annotators and annotations. Pass data to the SDK and it delegates to the list of configured annotators to create zero or more annotations.
Related abstractions and SDK implementations are detailed in the Annotator documentation.
The SDK provides a minimal API -- New(), Create(), Mutate(), and Close().
func New(annotators []annotator.Contract) *instanceUsed to instantiate a new SDK instance with the specified list of annotators.
Takes a list of annotators and returns an SDK instance.
func (sdk *instance) Create(data []byte) []*status.ContractUsed to register creation of new data with the SDK. Passes data through the SDK instance's list of annotators.
SDK instance method. Takes the data to annotate and returns a status.
Returns nil (and does not annotate) if Close() was previously called for the instance.
func (sdk *instance) Mutate(oldData, newData []byte) []*status.ContractUsed to register mutation of existing data with the SDK. Passes data through the SDK instance's list of annotators.
SDK instance method. Takes the data to annotate (and its prior version) and returns a status.
Returns nil (and does not annotate) if Close() was previously called for the instance.
func (sdk *instance) Close()SDK instance method. Call to ensure proper shutdown of the SDK.
Simple example that uses the SDK's PKI annotator, PKI verifier, and example publisher.
hashProvider := sha256.New()
uniqueProvider := ulid.New()
idProvider := identityProvider.New(hashProvider)
persistence := memory.New()
filter := passthrough.New()
// create SDK instance for annotation.
sdkInstance := sdk.New(
[]annotator.Contract{
pkiAnnotator.New(
struct{ Node string }{Node: "origin"},
uniqueProvider,
idProvider,
persistence,
signpkcs1v15.New(
crypto.SHA256,
testInternal.ValidPrivateKey,
testInternal.ValidPublicKey,
hashProvider,
),
),
},
)
// register data creation.
data, _ := json.Marshal(test.FactoryRandomString())
_ = sdkInstance.Create(data)
// close SDK instance.
sdkInstance.Close()
// create SDK instance for assessment.
p := struct{ Node string }{Node: "evaluation"}
w := testwriter.New()
sdkInstance = sdk.New(
[]annotator.Contract{
assess.New(
p,
uniqueProvider,
idProvider,
persistence,
pkiAssessor.New(verifier.New()),
filter,
),
publish.New(p, uniqueProvider, idProvider, persistence, example.New(w), filter),
},
)
// assess and publish result.
_ = sdkInstance.Create(data)
// display it.
fmt.Printf("%s\n", w.Get())
// close SDK instance
sdkInstance.Close()Executing this code results in JSON output resembling the following:
[
{
"unique": "01E7JQF86JMQJHBF4QX1EFD6Y3",
"created": "2020-05-05T15:32:34.6429938Z",
"identityCurrentType": "hash",
"identityCurrent": {
"hash": "Vc1WRGhjPV5SoXMDfRBazQMrAcvUVzAJi0+9x+hnjbE="
},
"identityPreviousType": "hash",
"identityPrevious": null,
"metadataType": "pki",
"metadata": {
"provenance": {
"Node": "origin"
},
"identitySignature": "tFohwFE4I1AXi4JLkrir5GkJzts+JJzicHgpL71AMuDvEWiG6edmsk2VE46lX9mnBfV1ja56+kKhHFISQFESgyvB9H/dwoUngMxrGKKwII6SoHqXpKjGrZ6qfV8R+sjgogDQ0IKcNbB6ouGaK1+0pMAsht1vorlYGWVC9qnWpzIEHAHjyC0T6VUzoiY4TcCS2SIiIdTGFdkgxd+gGyFAmJ2pd4kSUJrnRZbWr22tsVNMkgDC/Yqca4OLYDQv0ulNHBZK10M3QP9pK7yIutFyHjo0YTzoiNZGjDxKmurwaO10+lduo1qIiEqG2U4pTdUfL4d7V0h81OZqYus/azJpPQ==",
"dataSignature": "30oRl1lXT2nASM9X9LYkKqQL0FWpjN8dm95H9czDya4OIsru7wwVb3PFiOgTBtAMT+V//XrXBSkr1Hhqes1K41Q03Qv3AJGYUBeImyOV60dGj0+DxKZEbNhvH1ElyzVhhDWNz2994WPbpQ5XbEzK76XGlCCBR/AIiRMKPjb1zr8DNdwlUyLz7yI5Pqt8eZxpBN4TbI/y1eCwRgRbMxIgr1bht0B38rw5mdhfy0I854HztbDmvxiQqahL4oooGWDHFrh7w0rtWHGBf3ZrtQPR16fxqu0QvRQ0N/tS7q+mqjGtkG3AlWzBJD32M11amGj9JXb/JP3vriY4t/rh1HnJ/Q==",
"publicKey": "LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUErQ2wrYVByWFlybUJMUGo3TWZsVgpTbkMwQ25iKzA4QzgrVXlReC9sbFdiZHpic09wcFVpeDJndk1lVUllcWJyRGdndTFhYzhvZ3dqUnVSZGlBTDBzCnlnQ3V0Q01ENGpCLzJkMHgyRkVndktiUmR1ZkQydGJPUTJQbmlxb0pyOUliWUhNSllXQUtES1dyQ1hjVitnNUoKay9oS1NDdG1Ecmd2R04zZ21iYXJndFdtRkdMYklHY0o5SHFmQWVMYzR0dUNaOTFpL0kxY3ZLVWFxZmRBSmtKZQpFYnJIdzBQdUdEeXdwcWZQbERDY2xKWCtjT09QVzRqQ0dXZDREYjQydmxTakNCZnE0WjBNQlgrK3U0UDh1eXg2ClNDMkxybEVkUk80M3lCL3J5QlIxN0lpb0NTeHVraGF2ZlpIYllQRXdyTFRZYXJ4OEZZNjJGVWdjZmZ6Y3NtaHgKWndJREFRQUIKLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0tCg==",
"signerType": "x509.pkcsv1",
"signerMetadata": {
"result": "success",
"signerHash": "sha256",
"reducerHash": "sha256"
}
}
},
{
"unique": "01E7JQF86JMQJHBF4QX3A9X93K",
"created": "2020-05-05T15:32:34.6429938Z",
"identityCurrentType": "hash",
"identityCurrent": {
"hash": "Vc1WRGhjPV5SoXMDfRBazQMrAcvUVzAJi0+9x+hnjbE="
},
"identityPreviousType": "hash",
"identityPrevious": null,
"metadataType": "assessment",
"metadata": {
"provenance": {
"Node": "evaluation"
},
"assessorType": "x509.pkcsv1",
"assessorMetadata": {
"result": "success",
"validSignature": true,
"unique": [
"01E7JQF86JMQJHBF4QX1EFD6Y3"
]
}
}
}
]Full executable source is at cmd/examples/readme/main.go.
README.md This file
README.assets/ Images and assets included in README.md
LICENSE Project's license
cmd/
examples/
multistage/
main.go Sample SDK usage (multiple stages)
readme/
main.go Sample SDK usage (as presented above)
internal/
pkg/
datetime/ Date and time stamp implementation
test/ Test-related implementation
metadata/ Annotation-specific assertions
pkg/
annotation/ Annotations
metadata/ Common annotation definitions
store/ Annotation store implementation
contract.go Annotation store abstraction
memory/ In-process in-memory store implementation
uniqueprovider/ Unique provider
contract.go Unique provider abstraction
ulid/ ULID-based implementation
annotator/ Annotators
README.md Annotator documentation
README.assets/ Images and assets included in README.md
contract.go Annotator abstraction
hashprovider/ Hash Provider (reduce data to unique hash)
contract.go Hash provider abstraction
md5/ MD5-based implementation
passthrough/ passthrough implementation
sha256/ SHA256-based implementation
identity/ Identity
contract.go Identity abstraction
hash/ Hash-based identity implementation
identityprovider/ Identity provider
contract.go Identity provider abstraction
hash/ Hash-based identity provider implementation
sdk/ Public SDK API
close.go SDK Close() implementation
create.go SDK Create() implementation
mutate.go SDK Mutate() implementation
sdk.go SDK factory function implementation
status/
contract.go Return value abstraction
The current SDK implementation is incomplete and limited. It was intended as a simple reference implementation. Opportunities for improvement exist -- in no particular order:
- It is not optimized or performant.
- While part of its guiding vision, the current implementation does not implement trust scoring. A scoring implementation would leverage the annotations created by the SDK.
- It does not include a non-transient annotation persistence implementation. Only an example in-memory store service is provided.
- While the annotation store contract implies immutability, there are no restrictions on implementation to enforce it.
- It does not version, sign, or encrypt individual annotations. Annotations created by the current implementation are not signed or secured against tampering.
- It does not conform to existing annotation standards. SDK annotations use bespoke JSON schema.
- It is currently limited to storing, retrieving, and processing only the annotations it originates. This precludes accessing and evaluating metadata originated and stored outside of Alvarium -- particularly limiting when Alvarium is not the primary annotation mechanism (as is expected in most use-cases).
- It does not currently delineate -- must less track and attest -- authorship. Authorship is the identity of the entity that recorded the annotation. Authorship could be captured using the current SDK implementation as ad-hoc context. This is less desirable/flexible than adding explicit authorship as an Alvarium annotation property.
- It does not currently delineate -- much less track and attest -- data ownership. Ownership is the identity of the entity currently responsible for the data and which acts as the source of truth for that data. Ownership could be captured using the current SDK implementation as ad-hoc context. This is less desirable/flexible than adding explicit ownership as an Alvarium annotation property.
- It does not evaluate conformance of data to a specification. It could be made to do so via a new purpose-specific annotator.
- It does not evaluate data consistency to specified dynamic tolerances. This would require definition of a data set -- a concept not currently recognized by the Alvarium architecture -- to bound evaluation across a subset of data.
- It does not evaluate data consistency to specified static tolerances. It could be made to do so through a new purpose-specific annotator.
- While Alvarium annotations are treated as immutable historical events and assessments can be successfully repeated (given an identical configuration of the SDK and no new related annotations have been added), Alvarium does not currently implement arbitrary assessment for a given point in time.
- It implements trust factor assessment as a function via purpose-built annotators. The function implementing assessment is not portable.
- It uses RSA PKCS#1 v1.5 for signatures and verification. Version 1 has know flaws. The SDK should be updated to use version 2 (i.e. OAEP and PSS).
- It does not include annotators that retrieve published annotations (from IPFS or IOTA Tangle).
- The IOTA Tangle publisher implementation uses a single transaction for storage which limits maximum annotation storage size.
- The IOTA Tangle publisher implementation uses a unique address for each transaction.