Skip to content

Commit

Permalink
docs: update all docs following the Testnet redeployment (#202)
Browse files Browse the repository at this point in the history
Co-authored-by: Alexandros Tzimas <[email protected]>
Co-authored-by: George Danezis <[email protected]>
Co-authored-by: giac-mysten <[email protected]>
  • Loading branch information
4 people authored Jan 16, 2025
1 parent b433d7f commit 91ec49a
Show file tree
Hide file tree
Showing 24 changed files with 325 additions and 254 deletions.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ body:
value: |
Example:
1. Create an empty file: `touch empty`
2. Try to store the empty file: `walrus store empty`
2. Try to store the empty file: `walrus store empty --epochs 314`
validations:
required: true
- type: textarea
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -71,5 +71,5 @@ jobs:
RUST_LOG=site_builder=debug,walrus=debug,info
site-builder
update build/html ${{ vars.WALRUS_SITE_OBJECT }}
--epochs 200
--epochs 183
--force
2 changes: 1 addition & 1 deletion .github/workflows/update-binaries.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,5 @@ jobs:
RUST_LOG=site_builder=debug,walrus=debug,info
site-builder
update --list-directory site ${{ vars.WALRUS_SITE_BIN_OBJECT }}
--epochs 200
--epochs 183
--force
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
- [Devnet Update](./blog/02_devnet_update.md)
- [Announcing the Walrus Whitepaper](./blog/03_whitepaper.md)
- [Announcing Testnet](./blog/04_testnet_update.md)
- [Announcing Testnet v2](./blog/05_testnet_redeployment.md)

---

Expand Down
1 change: 1 addition & 0 deletions docs/assets/walrus-sites-portal-diagram.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
90 changes: 90 additions & 0 deletions docs/blog/05_testnet_redeployment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Announcing Testnet v2

Published on: 2025-01-16

We are today redeploying the Walrus Testnet to incorporate various improvements, including some
backwards-incompatible changes. Make sure to get the latest binary and configuration as described
in the [setup section](../usage/setup.md).

Note that all blob data on the previous Testnet instance has been wiped. All blobs need to be
re-uploaded to the new Testnet instance, including Walrus Sites. In addition, there is a new version
of the WAL token, so your previous WAL tokens will not work anymore. To use the Testnet v2, you
need to obtain new WAL tokens.

In the following sections, we describe the notable changes and the actions required for existing
Walrus Sites.

## Epoch duration

The epoch duration has been increased from one day to two days to emphasize that this duration is
different from Sui epochs (at Mainnet, epochs will likely be multiple weeks long). In addition, the
maximum number of epochs a blob can be stored for has been reduced from 200 to 183 (corresponding
to one year).

The `walrus store` command now also supports the `--epochs max` flag, which will store
the blob for the maximum number of epochs. Note that the `--epochs` flag is now mandatory.

## New features

Besides many improvements to the contracts and the storage-node service, the latest Walrus release
also brings several user-facing improvements.

- The `walrus store` command now supports storing multiple files at once. This is faster and more
cost-effective compared to storing each file separately as transactions can be batched through
[PTBs](https://docs.sui.io/concepts/transactions/prog-txn-blocks). Notably, this is compatible
with glob patterns offered by many shells, so you can for example run a command like `walrus store
*.png --epochs 100` to store all PNG files in the current directory.
- The `walrus` CLI now supports creating, funding, and extending *shared blobs* using the `walrus
share`, `walrus store --share`, and `walrus fund-shared-blob` commands. Shared blobs are an
example of collectively managed and funded blobs. See the [shared blobs
section](../usage/client-cli.md#shared-blobs) for more details.

## New WAL token

Along with the redeployment of Walrus, we have also deployed a fresh WAL contract. This
means that you cannot use any WAL token from the previous Testnet instance with the new Testnet
instance. You need to request new WAL tokens through the [Testnet WAL
faucet](../usage/setup.md#testnet-wal-faucet).

## Backwards-incompatible changes

One reason for a full redeployment is to allow us to make some changes that are
backwards-incompatible. Many of those are related to the contracts and thus less visible to users.
There are, however, some changes that may affect you.

### Configuration files

The format of the configuration files for storage nodes and clients has been changed. Make sure to
use the latest version of the configuration files, see the [configuration
section](../usage/setup.md#configuration).

### CLI options

Several CLI options of the `walrus` CLI have been changed. Notably, all "short" variants of options
(e.g., `-e` instead of `--epochs`) have been removed to prevent future confusion with new options.
Additionally, the `--epochs` flag is now mandatory for the `walrus store` command (this also affects
the [JSON API](../usage/json-api.md)).

Please refer to the CLI help (`walrus --help`, or `walrus <command> --help`) for further details.

### HTTP APIs

The paths, request, and response formats of the HTTP APIs have changed for the storage nodes, and
also the aggregator and publisher. Please refer to the section on the [HTTP
API](../usage/web-api.md) for further details.

## Effects on and actions required for existing Walrus Sites

The Walrus Sites contracts have not changed, which means that all corresponding objects on Sui are
still valid. However, the resources now point to blob IDs that do not yet exist on the new Testnet.
The easiest way to fix existing sites is to simply update them with the `--force` flag:

```sh
site-builder update --epochs <number of epochs> --force <path to site> <existing site object>
```

## New Move contracts & documentation

As part of the new Testnet release of Walrus, the Move smart contracts have been updated; the
deployed version can be found in the [`walrus-docs`
repository](https://github.com/MystenLabs/walrus-docs/tree/main/contracts).
15 changes: 7 additions & 8 deletions docs/client_config.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
system_object: 0x50b84b68eb9da4c6d904a929f43638481c09c03be6274b8569778fe085c1590d
staking_object: 0x37c0e4d7b36a2f64d51bba262a1791f844cfd88f31379f1b7c04244061d43914
walrus_package: 0x3d35ad1028562025f6f24336f0298d3775ba896bbbb63be7ad5b9fee8255dd89
exchange_object:
- 0x0e60a946a527902c90bbc71240435728cd6dc26b9e8debc69f09b71671c3029b
- 0x8a23a552895e341bca0106861786e014b5bb2f576bd7f76754226cc92266a0ee
- 0x7c469c2b189379bff42874742c292934c03cde9d0a2c20f293f1a32f8eece68c
- 0x59e7fa1b967c739ce676a7a3d8de444ac165f742421ba3b17656e2aee9fe541e
system_object: 0x98ebc47370603fe81d9e15491b2f1443d619d1dab720d586e429ed233e1255c1
staking_object: 0x20266a17b4f1a216727f3eef5772f8d486a9e3b5e319af80a5b75809c035561d
exchange_objects:
- 0x59ab926eb0d94d0d6d6139f11094ea7861914ad2ecffc7411529c60019133997
- 0x89127f53890840ab6c52fca96b4a5cf853d7de52318d236807ad733f976eef7b
- 0x9f9b4f113862e8b1a3591d7955fadd7c52ecc07cf24be9e3492ce56eb8087805
- 0xb60118f86ecb38ec79e74586f1bb184939640911ee1d63a84138d080632ee28a
4 changes: 2 additions & 2 deletions docs/dev-guide/dev-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ smaller chunks.

Blobs are stored for a certain number of *epochs*, as specified at the time they were stored. Walrus
storage nodes ensure that within these epochs a read succeeds. The current Testnet uses a short
epoch duration of one day for testing purposes, but Mainnet epochs are likely to be longer such as
many weeks each.
epoch duration of two days for testing purposes, but Mainnet epochs are planned to be multiple
weeks.

## Read

Expand Down
86 changes: 57 additions & 29 deletions docs/usage/client-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,20 @@ their meaning.

## Walrus system information

Information about the Walrus system is available through the `walrus info` command. For example,
`walrus info` gives an overview of current system parameters such as the current epoch, the number
of storage nodes and shards in the system, the maximum blob size, and the current cost in (Testnet)
WAL for storing blobs:
Information about the Walrus system is available through the `walrus info` command. It provides an
overview of current system parameters such as the current epoch, the number of storage nodes and
shards in the system, the maximum blob size, and the current cost in (Testnet) WAL for storing
blobs:

```console
$ walrus info

Walrus system information

Epochs and storage duration
Current epoch: 47
Epoch duration: 1day
Blobs can be stored for at most 200 epochs in the future.
Current epoch: 1
Epoch duration: 2day
Blobs can be stored for at most 183 epochs in the future.

Storage nodes
Number of storage nodes: 30
Expand Down Expand Up @@ -60,7 +60,7 @@ information on the storage nodes in the current and (if already selected) the ne
including their node IDs and stake and shard distribution can be viewed with the `--dev` argument:
`walrus info --dev`.

## Storing, querying status, and reading blobs
## Storing blobs

```admonish danger title="Public access"
**All blobs stored in Walrus are public and discoverable by all.** Therefore you must not use Walrus
Expand All @@ -71,43 +71,49 @@ confidentiality.
```admonish warning
It must be ensured that only a single process uses the Sui wallet for write actions (storing or
deleting). When using multiple instances of the client simultaneously, each of them must be pointed
to a different wallet.
to a different wallet. Note, it is possible to store multiple blobs with a single `walrus store`
command.
```

```admonish tip title="Obtaining Testnet WAL"
You can exchange Testnet SUI for Testnet WAL by running `walrus get-wal`. See the [setup
page](./setup.md#testnet-wal-faucet) for further details.
```

Storing blobs on Walrus can be achieved through the following command:
Storing one or multiple blobs on Walrus can be achieved through the following command:

```sh
walrus store <some file> --epochs <EPOCHS>
walrus store <FILES> --epochs <EPOCHS>
```

The CLI argument `--epochs <EPOCHS>` (or `-e`) indicates the number of epochs the blob should be
stored for. There is an upper limit on the number of epochs a blob can be stored for, which is 200
for the current Testnet deployment.
The mandatory CLI argument `--epochs <EPOCHS>` indicates the number of epochs the blob should be
stored for. There is an upper limit on the number of epochs a blob can be stored for, which is 183
for the current Testnet deployment (corresponding to one year). In addition to a positive integer,
you can also use `--epochs max` to store the blob for the maximum number of epochs.

```admonish warning
The `--epochs` argument is currently optional; if it is not specified, a warning is printed and the
default of 1 epoch is used. However, it will become a mandatory argument in the future.
```
You can store a single file or multiple files, separated by spaces. Notably, this is compatible
with glob patterns; for example, `walrus store *.png --epochs <EPOCHS>` will store all PNG files
in the current directory.

By default, the command will store the blob as a *permanent* blob. See the [section on deletable
blobs](#reclaiming-space-via-deletable-blobs) for more details on deletable blobs. Also, by default
an owned `Blob` object is created. It is possible to wrap this into a shared object, which can be
funded and extended by anyone, see the [shared blobs section](#shared-blobs).

```admonish tip title="Automatic optimizations"
When storing a blob, the client performs a number of automatic optimizations, including the
following:

- If the blob is already stored as a *permanent blob* on Walrus for a sufficient number of epochs
the command does not store it again. This behavior can be overwritten with the `--force` (or `-f`)
the command does not store it again. This behavior can be overwritten with the `--force`
CLI option, which stores the blob again and creates a fresh blob object on Sui belonging to the
wallet address.
- If the user's wallet has a compatible storage resource, this one is (re-)used instead of buying a
new one.
- If the user's wallet has a storage resource of suitable size and duration, it is (re-)used instead
of buying a new one.
- If the blob is already certified on Walrus but as a *deletable* blob or not for a sufficient
number of epochs, the command skips sending data to the storage nodes and just collects the
availability certificate
```
number of epochs, the command skips sending encoded blob data to the storage nodes and just
collects the availability certificate

## Querying blob status

The status of a blob can be queried through one of the following commands:

Expand All @@ -124,19 +130,21 @@ When the blob is available, the `blob-status` command also returns the `BlobCert
which consists of a transaction ID and a sequence number in the events emitted by the transaction.
The existence of this event certifies the availability of the blob.

## Reading blobs

Reading blobs from Walrus can be achieved through the following command:

```sh
walrus read <some blob ID>
```

By default the blob data is written to the standard output. The `--out <OUT>` CLI option (or `-o`)
can be used to specify an output file name. The `--rpc-url <URL>` (or `-r`) may be used to specify
By default the blob data is written to the standard output. The `--out <OUT>` CLI option
can be used to specify an output file name. The `--rpc-url <URL>` may be used to specify
a Sui RPC node to use instead of the one set in the wallet configuration or the default one.

## Reclaiming space via deletable blobs

By default `walrus store` uploads a blob and Walrus will keep it available until after its expiry
By default `walrus store` uploads a permanent blob available until after its expiry
epoch. Not even the uploader may delete it beforehand. However, optionally, the store command
may be invoked with the `--deletable` flag, to indicate the blob may be deleted before its expiry
by the owner of the Sui blob object representing the blob. Deletable blobs are indicated as such
Expand All @@ -152,7 +160,7 @@ Optionally the delete command can be invoked by specifying a `--file <PATH>` opt
blob ID from a file, or `--object-id <SUI_ID>` to delete the blob in the Sui blob object specified.

Before deleting a blob, the `walrus delete` command will ask for confirmation unless the `--yes`
(or `-y`) option is specified.
option is specified.

The `delete` command reclaims the storage object associated with the deleted blob, which is re-used
to store new blobs. The delete operation provides flexibility around managing storage costs and
Expand All @@ -173,6 +181,26 @@ It does not delete blobs from caches, slivers from past storage nodes, or copies
that could have been made by users before the blob was deleted.
```

## Shared blobs

*Shared blobs* are shared Sui objects wrapping "standard" `Blob` objects that can be funded and
whose lifetime can be extended by anyone. See the [shared blobs
contracts](https://github.com/MystenLabs/walrus-docs/blob/main/contracts/walrus/sources/system/shared_blob.move)
for further details.

You can create a shared blob from an existing `Blob` object you own with the `walrus share` command:

```sh
walrus share --blob-obj-id <BLOB_OBJ_ID>
```

The resulting shared blob can be directly funded by adding an `--amount`, or you can fund an
existing shared blob with the `walrus fund-shared-blob` command. Additionally, you can immediately
share a newly created blob by adding the `--share` option to the `walrus store` command.

You can use the `walrus extend` command to extend the lifetime of a shared blob object. Shared blobs
can only contain permanent blobs and cannot be deleted before their expiry.

## Blob ID utilities

The `walrus blob-id <FILE>` may be used to derive the blob ID of any file. The blob ID is a
Expand Down
3 changes: 2 additions & 1 deletion docs/usage/json-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ walrus json \
"config": "path/to/client_config.yaml",
"command": {
"store": {
"file": "README.md"
"files": ["README.md", "LICENSE"],
"epochs": 100
}
}
}'
Expand Down
Loading

0 comments on commit 91ec49a

Please sign in to comment.