Skip to content

Commit

Permalink
chore: some style/consistency edits (#54)
Browse files Browse the repository at this point in the history
  • Loading branch information
ronny-mysten authored Jun 17, 2024
1 parent 4f3a319 commit 9294921
Show file tree
Hide file tree
Showing 5 changed files with 90 additions and 66 deletions.
10 changes: 5 additions & 5 deletions docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

To make communication as clear and efficient as possible, we make sure to use a single term for
every Walrus entity/concept and *do not* use any synonyms. The following table lists various
concepts, their canonical name, how they relate to / differ from other terms.
concepts, their canonical name, and how they relate to or differ from other terms.

Italicized terms in the description indicate other specific Walrus terms contained in the table.

Expand All @@ -15,12 +15,12 @@ Italicized terms in the description indicate other specific Walrus terms contain
| blob ID | cryptographic ID computed from a *blob*’s *slivers* |
| blob metadata | metadata of one *blob*; in particular, this contains a hash per *shard* to enable the authentication of *slivers* and recovery symbols |
| (end) user | any entity/person that wants to store or read *blobs* on/from Walrus; can act as a Walrus client itself or use the simple interface exposed by *publishers* and *caches* |
| publisher | service interacting with Sui and the *SNs* to store *blobs* on Walrus; offers a simple HTTP POST endpoint to *end users* |
| aggregator | service that reconstructs *blobs* by interacting with *SNs* and exposes a simple HTTP GET endpoint to *end users* |
| publisher | service interacting with Sui and the *SNs* to store *blobs* on Walrus; offers a simple `HTTP POST` endpoint to *end users* |
| aggregator | service that reconstructs *blobs* by interacting with *SNs* and exposes a simple `HTTP GET` endpoint to *end users* |
| cache | an *aggregator* with additional caching capabilities |
| (Walrus) client | entity interacting directly with the *SNs*; this can be an *aggregator*/*cache*, a *publisher*, or an *end user* |
| (blob) reconstruction | decoding of the primary *slivers* to obtain the blob; includes re-encoding the *blob* and checking the Merkle proofs |
| (shard/sliver) recovery | process of an SN recovering a *sliver* or full *shard* by obtaining recovery symbols from other *SNs* |
| (shard/sliver) recovery | process of an *SN* recovering a *sliver* or full *shard* by obtaining recovery symbols from other *SNs* |
| storage attestation | process where *SNs* exchange challenges and responses to demonstrate that they are storing their currently assigned *shards* |
| certificate of availability (CoA) | a *blob ID* with signatures of *SNs* holding at least $2f+1$ *shards* in a specific *epoch* |
| point of availability (PoA) | point in time when a *CoA* is submitted to Sui and the corresponding *blob* is guaranteed to be available until its expiration |
Expand All @@ -30,4 +30,4 @@ Italicized terms in the description indicate other specific Walrus terms contain
| member | an *SN* that is part of a *committee* at some *epoch* |
| storage epoch | the epoch for Walrus as distinct to the epoch for Sui |
| availability period | the period specified in *storage epochs* for which a *blob* is certified to be available on Walrus |
| expiry | the end epoch, at which a blob is no more available, and may be deleted. |
| expiry | the end epoch at which a blob is no longer available and can be deleted |
66 changes: 41 additions & 25 deletions docs/overview/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,48 +3,64 @@
The key actors in the Walrus architecture are the following:

- **Users** through **clients** want to store and read **blobs** identified by their **blob ID**.
They are ready to pay for service
when it comes to writes, and when it comes to non-best-effort reads. Users also want to prove

These actors are ready to pay for service
when it comes to writes and non-best-effort reads. Users also want to prove
the **availability** of a blob to third parties without the cost of sending or receiving the full
blob. Users may be malicious in various ways: they may wish to not pay for services, prove the
availability of an unavailable blobs, or modify / delete blobs without authorization, try to
exhaust resources of storage nodes, etc.
- **Storage nodes** hold one or many **shards** within a **storage epoch**. Each blob is erasure
encoded in many **slivers** and slivers from each stored blob become part of all shards. A shard
at any storage epoch is associated with a **storage node** that actually stores all slivers of
the shard, and is ready to serve them. The assignment of storage nodes to shards within
**storage epochs** is controlled by a Sui smart contract and we assume that more than 2/3 of the
shards are managed by correct storage nodes within each storage epoch. This means that we must
blob.

Users might be malicious in various ways: they might not want to pay for services, prove the
availability of unavailable blobs, modify/delete blobs without authorization, try to
exhaust resources of storage nodes, and so on.

- **Storage nodes** hold one or many **shards** within a **storage epoch**.

Each blob is erasure
encoded in many **slivers**. Slivers from each stored blob become part of all shards. A shard
at any storage epoch is associated with a storage node that actually stores all slivers of
the shard and is ready to serve them.

A Sui smart contract controls the assignment of storage nodes to shards within
**storage epochs** and Walrus assumes that more than 2/3 of the
shards are managed by correct storage nodes within each storage epoch. This means that Walrus must
tolerate up to 1/3 of the shards managed by Byzantine storage nodes (approximately 1/3 of the
storage nodes being Byzantine) within each storage epoch and across storage epochs.

- All clients and storage nodes operate a **blockchain** client (specifically on Sui), and mediate
payments, resources (space), mapping of shards to storage nodes, and metadata through blockchain
payments, resources (space), mapping of shards to storage nodes and metadata through blockchain
smart contracts. Users interact with the blockchain to acquire storage resources and upload
certificates for stored blobs. Storage nodes listen to the blockchain events to coordinate
their operations.

Walrus supports any additional number of optional infrastructure actors that can operate in a
permissionless way:

- **Caches** are **clients** that store one or more full blobs and make them available to users
over traditional web2 (HTTP, etc) technologies. They are optional in that end-users may also
- **Caches** are clients that store one or more full blobs and make them available to users
over traditional web2 technologies (such as HTTP).

They are optional in that end users can also
operate a local cache, and perform Walrus reads over web2 technologies locally. However, cache
infrastructures may also act as CDNs, split the cost of blob reconstruction over many requests,
be better connected, etc. A client can always verify that reads from such infrastructures
infrastructures can also act as CDNs, split the cost of blob reconstruction over many requests,
be better connected, and so on. A client can always verify that reads from such infrastructures
are correct.
- **Publishers** are **clients** that help end users store a blob using web2 technologies,
using less bandwidth and custom logic. They in effect receive the blob to be published over
traditional web2 protocols (e.g., HTTP) and run the Walrus store protocol on the end-user's

- **Publishers** are clients that help end users store a blob using web2 technologies,
using less bandwidth and custom logic.

In effect, they receive the blob to be published over
traditional web2 protocols (like HTTP) and run the Walrus store protocol on the end user's
behalf. This includes encoding the blob into slivers, distributing the slivers to shards,
collecting storage-node signatures and aggregating them into a certificate, as well as all
other on-chain actions. They are optional in that a user may directly interact with Sui and
other on-chain actions.

They are optional in that a user can directly interact with Sui and
the storage nodes to store blobs. An end user can always verify that a publisher
performed their duties correctly by checking that an event associated with the
[Point of Availability](./properties.md) for the blob exists on-chain
and then either performing a read to see if Walrus returns the blob or by encoding the blob
[Point of Availability](./properties.md) for the blob exists on chain
and then either performing a read to see if Walrus returns the blob or encoding the blob
and comparing the result to the blob ID in the certificate.

Caches, publishers, and end-users are not considered trusted components of the system, and they may
Caches, publishers and end users are not considered trusted components of the system, and they might
deviate from the protocol arbitrarily. However, some of the security properties of Walrus only hold
for honest end-users that use honest intermediaries (caches and publishers). We provide means for
end-users to audit the correct operation of both caches and publishers.
for honest end users that use honest intermediaries (caches and publishers). Walrus provides a
means for end users to audit the correct operation of both caches and publishers.
72 changes: 40 additions & 32 deletions docs/overview/objectives_use_cases.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,31 @@
# Objectives and use-cases
# Objectives and use cases

Walrus supports operations to store and retrieve blobs, and to prove and verify their availability.
It ensures content survives storage nodes suffering Byzantine faults and remains available and
retrievable. It provides APIs to access the stored content over a CLI, SDKs and over web2 HTTP
technologies, and supports content delivery infrastructures like caches and content distribution
networks (CDNs).

Under the hood, storage cost is a small fixed multiple of the size of blobs (around 5x) thanks to
advanced error correction coding, in contrast to the full replication of data traditional to
blockchains such as >100x for data stored in Sui objects. As a result much bigger resources, up to
several GiB, may be stored on Walrus at substantially lower cost compared to Sui or other
blockchains. Since encoded blobs are stored on all storage nodes Walrus also provides superior
robustness compared with designs with a small amount of replicas storing the full blob.
Under the hood, storage cost is a small fixed multiple of the size of blobs (around 5x).
Advanced error correction coding keeps the cost low, in contrast to the full replication of data
traditional to blockchains, such as the >100x multiple for data stored in Sui objects. As a result,
storage of much bigger resources (up to several GiB) is possible on Walrus at substantially lower
cost compared to Sui or other blockchains. Because encoded blobs are stored on all storage nodes,
Walrus also provides superior robustness compared to designs with a small amount of replicas
storing the full blob.

Walrus uses the Sui chain for coordination and payments. Available storage is represented as Sui
objects that can be acquired, owned, split, merged and transferred. Storage space can be tied to
a stored blob for a period of time. And the resulting Sui object may be used to prove
a stored blob for a period of time, with the resulting Sui object used to prove
availability on chain in smart contracts, or off chain using light clients.

In the [next chapter](./overview.md) we discuss in details the above operations relating to storage,
retrieval and availability.
The [next chapter](./overview.md) discusses the above operations relating to storage,
retrieval and availability in detail.

In the future, we plan to include in Walrus some minimal governance to allow storage nodes to
change between storage epochs. Walrus is also compatible with periodic payments for continued
storage. We also plan to implement storage attestation based on challenges to get confidence that
that blobs are stored or at least available. Walrus also allows light-nodes that store small parts
storage. We also plan to implement storage attestation based on challenges to get confidence
that blobs are stored or at least available. Walrus also allows light nodes that store small parts
of blobs to get rewards for proving availability and assisting recovery. We will cover these
topics in later documents. We also provide details of the encoding scheme in a separate document.

Expand All @@ -35,38 +36,45 @@ There are a few things that Walrus explicitly is not:
- Walrus does not reimplement a CDN that might be geo-replicated or have less than tens of
milliseconds of latency. Instead, it ensures that traditional CDNs are usable and compatible with
Walrus caches.

- Walrus does not re-implement a full smart contracts platform with consensus or execution. It
relies on Sui smart contracts when necessary, to manage Walrus resources and processes including
payments, storage epochs etc.
- Walrus supports storage of any blob including encrypted blobs, however Walrus itself is not the
payments, storage epochs, and so on.

- Walrus supports storage of any blob, including encrypted blobs, however Walrus itself is not the
distributed key management infrastructure that manages and distributed encryption or decryption
keys to support a full private storage eco-system. It can however provide the storage layer of for
keys to support a full private storage eco-system. It can, however, provide the storage layer for
such infrastructures.

## Use-cases
## Use cases

App builders may use Walrus in conjunction with Sui to build experiences that require large
amounts of data to be stored in a decentralized manner and possibly certified as available:
App builders can use Walrus in conjunction with Sui to build experiences that require storage
of large amounts of data in a decentralized manner and possibly certified as available:

- **Storage of media for NFT or dApps**: Walrus can directly store and serve media such as images,
sounds, sprites, videos, other game assets, and so on. This is publicly available media that is
accessed using HTTP requests at caches to create multimedia dApps.

- **Storage of media for NFT or dapps**: Walrus can directly store and serve media such as images,
sounds, sprites, videos, other game assets, etc. This is publicly available media that can be
accessed using HTTP requests at caches to create multimedia dapps.
- **AI related use cases**: Walrus can store clean data sets of training data, datasets with a
known and verified provenance, models weights, and proofs of correct training for AI models.
Or it may be used to store and ensure the availability of an AI model output.
- **Storage of long term archival of blockchain history**: Walrus can be used as a lower cost
decentralized store to store blockchain history. For Sui this can include sequences of
checkpoints with all associated transaction and effects content. As well as historic snapshots
known and verified provenance, models, weights and proofs of correct training for AI models.
It can also store and ensure the availability of an AI model output.

- **Storage of long term archival of blockchain history**: Walrus can act as a lower cost
decentralized store to store blockchain history. For Sui, this can include sequences of
checkpoints with all associated transaction and effects content, as well as historic snapshots
of the blockchain state, code or binaries.

- **Support availability for L2s**: Walrus allows parties to certify the availability of blobs, as
required by L2s that need data to be stored and be attested as available to all. This may also
include availability of extra audit data such as validity proofs, zero knowledge proofs of
correct execution or large fraud proofs.
- **Support a full decentralized web experience**: Walrus can host full decentralized web
experiences including all resources (such as js, css, html, media). These can provide content but
also host the UX of dapps to build dapps with fully decentralized front end and back ends on
chain. It brings the full "web" back in "web3".

- **Support a fully decentralized web experience**: Walrus can host fully decentralized web
experiences, including all resources (such as js, css, html, media). These can not only
provide content, but also host the UX of dApps to enable applications with fully decentralized
front end and back ends on chain. Walrus puts the full "web" into web3.

- **Support subscription models for media**: Creators can store encrypted media on Walrus and only
provide access via decryption keys to parties that have paid a subscription fee or have paid for
contents. (Note that Walrus provides the storage, encryption and decryption needs to happen off
Walrus).
content. Walrus provides the storage, encryption and decryption needs to happen off
the system.
4 changes: 2 additions & 2 deletions docs/overview/overview.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# System overview

This chapter provides an overview of the [security properties](./properties.md),
[architecture](./architecture.md), and [encoding mechanisms](./encoding.md) of the Walrus system.
[architecture](./architecture.md) and [encoding mechanisms](./encoding.md) of the Walrus system.

A [glossary](../glossary.md) of terms used is also available for reference.
Use the [Glossary](../glossary.md) as a reference for bolded terms used in this documentation.
4 changes: 2 additions & 2 deletions docs/tos.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Last updated: June 13, 2024

By using Mysten Labs devnet software, technologies, tools, and other services (collectively
By using Mysten Labs Devnet software, technologies, tools, and other services (collectively
“Devnet”), you agree to the general Terms of Service and these additional Devnet Terms of Service
(together, the “Terms”). If you do not agree, do not participate in DevNet. If you are using Devnet
(together, the “Terms”). If you do not agree, do not participate in Devnet. If you are using Devnet
on behalf of an organization, you represent and warrant that you are an authorized representative of
that organization and have the authority to bind that business or entity to the Terms.

Expand Down

0 comments on commit 9294921

Please sign in to comment.