Document not found (404)
-This URL is invalid, sorry. Please use the navigation bar or search to continue.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/favicon.png b/pr-preview/pr-16/favicon.png
deleted file mode 100644
index a5b1aa16..00000000
Binary files a/pr-preview/pr-16/favicon.png and /dev/null differ
diff --git a/pr-preview/pr-16/favicon.svg b/pr-preview/pr-16/favicon.svg
deleted file mode 100644
index 90e0ea58..00000000
--- a/pr-preview/pr-16/favicon.svg
+++ /dev/null
@@ -1,22 +0,0 @@
-
-
diff --git a/pr-preview/pr-16/fonts/OPEN-SANS-LICENSE.txt b/pr-preview/pr-16/fonts/OPEN-SANS-LICENSE.txt
deleted file mode 100644
index d6456956..00000000
--- a/pr-preview/pr-16/fonts/OPEN-SANS-LICENSE.txt
+++ /dev/null
@@ -1,202 +0,0 @@
-
- Apache License
- Version 2.0, January 2004
- http://www.apache.org/licenses/
-
- TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
- 1. Definitions.
-
- "License" shall mean the terms and conditions for use, reproduction,
- and distribution as defined by Sections 1 through 9 of this document.
-
- "Licensor" shall mean the copyright owner or entity authorized by
- the copyright owner that is granting the License.
-
- "Legal Entity" shall mean the union of the acting entity and all
- other entities that control, are controlled by, or are under common
- control with that entity. For the purposes of this definition,
- "control" means (i) the power, direct or indirect, to cause the
- direction or management of such entity, whether by contract or
- otherwise, or (ii) ownership of fifty percent (50%) or more of the
- outstanding shares, or (iii) beneficial ownership of such entity.
-
- "You" (or "Your") shall mean an individual or Legal Entity
- exercising permissions granted by this License.
-
- "Source" form shall mean the preferred form for making modifications,
- including but not limited to software source code, documentation
- source, and configuration files.
-
- "Object" form shall mean any form resulting from mechanical
- transformation or translation of a Source form, including but
- not limited to compiled object code, generated documentation,
- and conversions to other media types.
-
- "Work" shall mean the work of authorship, whether in Source or
- Object form, made available under the License, as indicated by a
- copyright notice that is included in or attached to the work
- (an example is provided in the Appendix below).
-
- "Derivative Works" shall mean any work, whether in Source or Object
- form, that is based on (or derived from) the Work and for which the
- editorial revisions, annotations, elaborations, or other modifications
- represent, as a whole, an original work of authorship. For the purposes
- of this License, Derivative Works shall not include works that remain
- separable from, or merely link (or bind by name) to the interfaces of,
- the Work and Derivative Works thereof.
-
- "Contribution" shall mean any work of authorship, including
- the original version of the Work and any modifications or additions
- to that Work or Derivative Works thereof, that is intentionally
- submitted to Licensor for inclusion in the Work by the copyright owner
- or by an individual or Legal Entity authorized to submit on behalf of
- the copyright owner. For the purposes of this definition, "submitted"
- means any form of electronic, verbal, or written communication sent
- to the Licensor or its representatives, including but not limited to
- communication on electronic mailing lists, source code control systems,
- and issue tracking systems that are managed by, or on behalf of, the
- Licensor for the purpose of discussing and improving the Work, but
- excluding communication that is conspicuously marked or otherwise
- designated in writing by the copyright owner as "Not a Contribution."
-
- "Contributor" shall mean Licensor and any individual or Legal Entity
- on behalf of whom a Contribution has been received by Licensor and
- subsequently incorporated within the Work.
-
- 2. Grant of Copyright License. Subject to the terms and conditions of
- this License, each Contributor hereby grants to You a perpetual,
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable
- copyright license to reproduce, prepare Derivative Works of,
- publicly display, publicly perform, sublicense, and distribute the
- Work and such Derivative Works in Source or Object form.
-
- 3. Grant of Patent License. Subject to the terms and conditions of
- this License, each Contributor hereby grants to You a perpetual,
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable
- (except as stated in this section) patent license to make, have made,
- use, offer to sell, sell, import, and otherwise transfer the Work,
- where such license applies only to those patent claims licensable
- by such Contributor that are necessarily infringed by their
- Contribution(s) alone or by combination of their Contribution(s)
- with the Work to which such Contribution(s) was submitted. If You
- institute patent litigation against any entity (including a
- cross-claim or counterclaim in a lawsuit) alleging that the Work
- or a Contribution incorporated within the Work constitutes direct
- or contributory patent infringement, then any patent licenses
- granted to You under this License for that Work shall terminate
- as of the date such litigation is filed.
-
- 4. Redistribution. You may reproduce and distribute copies of the
- Work or Derivative Works thereof in any medium, with or without
- modifications, and in Source or Object form, provided that You
- meet the following conditions:
-
- (a) You must give any other recipients of the Work or
- Derivative Works a copy of this License; and
-
- (b) You must cause any modified files to carry prominent notices
- stating that You changed the files; and
-
- (c) You must retain, in the Source form of any Derivative Works
- that You distribute, all copyright, patent, trademark, and
- attribution notices from the Source form of the Work,
- excluding those notices that do not pertain to any part of
- the Derivative Works; and
-
- (d) If the Work includes a "NOTICE" text file as part of its
- distribution, then any Derivative Works that You distribute must
- include a readable copy of the attribution notices contained
- within such NOTICE file, excluding those notices that do not
- pertain to any part of the Derivative Works, in at least one
- of the following places: within a NOTICE text file distributed
- as part of the Derivative Works; within the Source form or
- documentation, if provided along with the Derivative Works; or,
- within a display generated by the Derivative Works, if and
- wherever such third-party notices normally appear. The contents
- of the NOTICE file are for informational purposes only and
- do not modify the License. You may add Your own attribution
- notices within Derivative Works that You distribute, alongside
- or as an addendum to the NOTICE text from the Work, provided
- that such additional attribution notices cannot be construed
- as modifying the License.
-
- You may add Your own copyright statement to Your modifications and
- may provide additional or different license terms and conditions
- for use, reproduction, or distribution of Your modifications, or
- for any such Derivative Works as a whole, provided Your use,
- reproduction, and distribution of the Work otherwise complies with
- the conditions stated in this License.
-
- 5. Submission of Contributions. Unless You explicitly state otherwise,
- any Contribution intentionally submitted for inclusion in the Work
- by You to the Licensor shall be under the terms and conditions of
- this License, without any additional terms or conditions.
- Notwithstanding the above, nothing herein shall supersede or modify
- the terms of any separate license agreement you may have executed
- with Licensor regarding such Contributions.
-
- 6. Trademarks. This License does not grant permission to use the trade
- names, trademarks, service marks, or product names of the Licensor,
- except as required for reasonable and customary use in describing the
- origin of the Work and reproducing the content of the NOTICE file.
-
- 7. Disclaimer of Warranty. Unless required by applicable law or
- agreed to in writing, Licensor provides the Work (and each
- Contributor provides its Contributions) on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
- implied, including, without limitation, any warranties or conditions
- of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
- PARTICULAR PURPOSE. You are solely responsible for determining the
- appropriateness of using or redistributing the Work and assume any
- risks associated with Your exercise of permissions under this License.
-
- 8. Limitation of Liability. In no event and under no legal theory,
- whether in tort (including negligence), contract, or otherwise,
- unless required by applicable law (such as deliberate and grossly
- negligent acts) or agreed to in writing, shall any Contributor be
- liable to You for damages, including any direct, indirect, special,
- incidental, or consequential damages of any character arising as a
- result of this License or out of the use or inability to use the
- Work (including but not limited to damages for loss of goodwill,
- work stoppage, computer failure or malfunction, or any and all
- other commercial damages or losses), even if such Contributor
- has been advised of the possibility of such damages.
-
- 9. Accepting Warranty or Additional Liability. While redistributing
- the Work or Derivative Works thereof, You may choose to offer,
- and charge a fee for, acceptance of support, warranty, indemnity,
- or other liability obligations and/or rights consistent with this
- License. However, in accepting such obligations, You may act only
- on Your own behalf and on Your sole responsibility, not on behalf
- of any other Contributor, and only if You agree to indemnify,
- defend, and hold each Contributor harmless for any liability
- incurred by, or claims asserted against, such Contributor by reason
- of your accepting any such warranty or additional liability.
-
- END OF TERMS AND CONDITIONS
-
- APPENDIX: How to apply the Apache License to your work.
-
- To apply the Apache License to your work, attach the following
- boilerplate notice, with the fields enclosed by brackets "[]"
- replaced with your own identifying information. (Don't include
- the brackets!) The text should be enclosed in the appropriate
- comment syntax for the file format. We also recommend that a
- file or class name and description of purpose be included on the
- same "printed page" as the copyright notice for easier
- identification within third-party archives.
-
- Copyright [yyyy] [name of copyright owner]
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
diff --git a/pr-preview/pr-16/fonts/SOURCE-CODE-PRO-LICENSE.txt b/pr-preview/pr-16/fonts/SOURCE-CODE-PRO-LICENSE.txt
deleted file mode 100644
index 366206f5..00000000
--- a/pr-preview/pr-16/fonts/SOURCE-CODE-PRO-LICENSE.txt
+++ /dev/null
@@ -1,93 +0,0 @@
-Copyright 2010, 2012 Adobe Systems Incorporated (http://www.adobe.com/), with Reserved Font Name 'Source'. All Rights Reserved. Source is a trademark of Adobe Systems Incorporated in the United States and/or other countries.
-
-This Font Software is licensed under the SIL Open Font License, Version 1.1.
-This license is copied below, and is also available with a FAQ at:
-http://scripts.sil.org/OFL
-
-
------------------------------------------------------------
-SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
------------------------------------------------------------
-
-PREAMBLE
-The goals of the Open Font License (OFL) are to stimulate worldwide
-development of collaborative font projects, to support the font creation
-efforts of academic and linguistic communities, and to provide a free and
-open framework in which fonts may be shared and improved in partnership
-with others.
-
-The OFL allows the licensed fonts to be used, studied, modified and
-redistributed freely as long as they are not sold by themselves. The
-fonts, including any derivative works, can be bundled, embedded,
-redistributed and/or sold with any software provided that any reserved
-names are not used by derivative works. The fonts and derivatives,
-however, cannot be released under any other type of license. The
-requirement for fonts to remain under this license does not apply
-to any document created using the fonts or their derivatives.
-
-DEFINITIONS
-"Font Software" refers to the set of files released by the Copyright
-Holder(s) under this license and clearly marked as such. This may
-include source files, build scripts and documentation.
-
-"Reserved Font Name" refers to any names specified as such after the
-copyright statement(s).
-
-"Original Version" refers to the collection of Font Software components as
-distributed by the Copyright Holder(s).
-
-"Modified Version" refers to any derivative made by adding to, deleting,
-or substituting -- in part or in whole -- any of the components of the
-Original Version, by changing formats or by porting the Font Software to a
-new environment.
-
-"Author" refers to any designer, engineer, programmer, technical
-writer or other person who contributed to the Font Software.
-
-PERMISSION & CONDITIONS
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of the Font Software, to use, study, copy, merge, embed, modify,
-redistribute, and sell modified and unmodified copies of the Font
-Software, subject to the following conditions:
-
-1) Neither the Font Software nor any of its individual components,
-in Original or Modified Versions, may be sold by itself.
-
-2) Original or Modified Versions of the Font Software may be bundled,
-redistributed and/or sold with any software, provided that each copy
-contains the above copyright notice and this license. These can be
-included either as stand-alone text files, human-readable headers or
-in the appropriate machine-readable metadata fields within text or
-binary files as long as those fields can be easily viewed by the user.
-
-3) No Modified Version of the Font Software may use the Reserved Font
-Name(s) unless explicit written permission is granted by the corresponding
-Copyright Holder. This restriction only applies to the primary font name as
-presented to the users.
-
-4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
-Software shall not be used to promote, endorse or advertise any
-Modified Version, except to acknowledge the contribution(s) of the
-Copyright Holder(s) and the Author(s) or with their explicit written
-permission.
-
-5) The Font Software, modified or unmodified, in part or in whole,
-must be distributed entirely under this license, and must not be
-distributed under any other license. The requirement for fonts to
-remain under this license does not apply to any document created
-using the Font Software.
-
-TERMINATION
-This license becomes null and void if any of the above conditions are
-not met.
-
-DISCLAIMER
-THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
-OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
-COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
-INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
-DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
-OTHER DEALINGS IN THE FONT SOFTWARE.
diff --git a/pr-preview/pr-16/fonts/fonts.css b/pr-preview/pr-16/fonts/fonts.css
deleted file mode 100644
index 858efa59..00000000
--- a/pr-preview/pr-16/fonts/fonts.css
+++ /dev/null
@@ -1,100 +0,0 @@
-/* Open Sans is licensed under the Apache License, Version 2.0. See http://www.apache.org/licenses/LICENSE-2.0 */
-/* Source Code Pro is under the Open Font License. See https://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL */
-
-/* open-sans-300 - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: normal;
- font-weight: 300;
- src: local('Open Sans Light'), local('OpenSans-Light'),
- url('open-sans-v17-all-charsets-300.woff2') format('woff2');
-}
-
-/* open-sans-300italic - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: italic;
- font-weight: 300;
- src: local('Open Sans Light Italic'), local('OpenSans-LightItalic'),
- url('open-sans-v17-all-charsets-300italic.woff2') format('woff2');
-}
-
-/* open-sans-regular - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: normal;
- font-weight: 400;
- src: local('Open Sans Regular'), local('OpenSans-Regular'),
- url('open-sans-v17-all-charsets-regular.woff2') format('woff2');
-}
-
-/* open-sans-italic - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: italic;
- font-weight: 400;
- src: local('Open Sans Italic'), local('OpenSans-Italic'),
- url('open-sans-v17-all-charsets-italic.woff2') format('woff2');
-}
-
-/* open-sans-600 - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: normal;
- font-weight: 600;
- src: local('Open Sans SemiBold'), local('OpenSans-SemiBold'),
- url('open-sans-v17-all-charsets-600.woff2') format('woff2');
-}
-
-/* open-sans-600italic - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: italic;
- font-weight: 600;
- src: local('Open Sans SemiBold Italic'), local('OpenSans-SemiBoldItalic'),
- url('open-sans-v17-all-charsets-600italic.woff2') format('woff2');
-}
-
-/* open-sans-700 - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: normal;
- font-weight: 700;
- src: local('Open Sans Bold'), local('OpenSans-Bold'),
- url('open-sans-v17-all-charsets-700.woff2') format('woff2');
-}
-
-/* open-sans-700italic - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: italic;
- font-weight: 700;
- src: local('Open Sans Bold Italic'), local('OpenSans-BoldItalic'),
- url('open-sans-v17-all-charsets-700italic.woff2') format('woff2');
-}
-
-/* open-sans-800 - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: normal;
- font-weight: 800;
- src: local('Open Sans ExtraBold'), local('OpenSans-ExtraBold'),
- url('open-sans-v17-all-charsets-800.woff2') format('woff2');
-}
-
-/* open-sans-800italic - latin_vietnamese_latin-ext_greek-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Open Sans';
- font-style: italic;
- font-weight: 800;
- src: local('Open Sans ExtraBold Italic'), local('OpenSans-ExtraBoldItalic'),
- url('open-sans-v17-all-charsets-800italic.woff2') format('woff2');
-}
-
-/* source-code-pro-500 - latin_vietnamese_latin-ext_greek_cyrillic-ext_cyrillic */
-@font-face {
- font-family: 'Source Code Pro';
- font-style: normal;
- font-weight: 500;
- src: url('source-code-pro-v11-all-charsets-500.woff2') format('woff2');
-}
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-300.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-300.woff2
deleted file mode 100644
index 9f51be37..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-300.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-300italic.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-300italic.woff2
deleted file mode 100644
index 2f545448..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-300italic.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-600.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-600.woff2
deleted file mode 100644
index f503d558..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-600.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-600italic.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-600italic.woff2
deleted file mode 100644
index c99aabe8..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-600italic.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-700.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-700.woff2
deleted file mode 100644
index 421a1ab2..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-700.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-700italic.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-700italic.woff2
deleted file mode 100644
index 12ce3d20..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-700italic.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-800.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-800.woff2
deleted file mode 100644
index c94a223b..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-800.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-800italic.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-800italic.woff2
deleted file mode 100644
index eed7d3c6..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-800italic.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-italic.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-italic.woff2
deleted file mode 100644
index 398b68a0..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-italic.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-regular.woff2 b/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-regular.woff2
deleted file mode 100644
index 8383e94c..00000000
Binary files a/pr-preview/pr-16/fonts/open-sans-v17-all-charsets-regular.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/fonts/source-code-pro-v11-all-charsets-500.woff2 b/pr-preview/pr-16/fonts/source-code-pro-v11-all-charsets-500.woff2
deleted file mode 100644
index 72224568..00000000
Binary files a/pr-preview/pr-16/fonts/source-code-pro-v11-all-charsets-500.woff2 and /dev/null differ
diff --git a/pr-preview/pr-16/future.html b/pr-preview/pr-16/future.html
deleted file mode 100644
index 85bbb8bc..00000000
--- a/pr-preview/pr-16/future.html
+++ /dev/null
@@ -1,237 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Encoding, overheads, and verification
-We summarize here the basic encoding and cryptographic techniques used in Walrus.
--
-
- Storage nodes hold one or many shards in a storage epoch, out of a larger total (say 1000) -and each shard contains one blob sliver for each blob past PoA. Each shard is assigned to a -storage node in a storage epoch. -
- An erasure code encode algorithm takes a blob, -and encodes it as $K$ symbols, such that any fraction $p$ of symbols can be used to reconstruct -the blob. Each blob sliver contains a fixed number of such symbols. -
- We select $p<1/3$ so that a third of symbols and also slivers may be used to reconstruct the blob -by the decode algorithm. The matrix used to produce the erasure code is fixed and the same -for all blobs by the Walrus system, and encoders have no discretion about it. -
- Storage nodes manage one or more shards, and corresponding sliver of each blob are distributed -to all the storage shards. As a result, the overhead of the distributed store is ~5x that of -the blob itself, no matter how many shards we have. The encoding is systematic meaning that some -storage nodes hold part of the plain blob, allowing for fast random access reads. -
Each blob is also associated with some metadata including a blob ID to allow verification:
--
-
- A blob ID is computed as an authenticator of the set of all shard data and metadata (byte size, -encoding, blob hash). We hash a sliver representation in each of the shards and add the resulting -hashes into a Merkle tree. Then the root of the Merkle tree is the blob hash used to derive the -blob ID that identifies the blob in the system. -
- Each storage node may use the blob ID to check if some shard data belongs to a blob using the -authenticated structure corresponding to the blob hash (Merkle tree). A successful check means -that the data is indeed as intended by the writer of the blob (who, remember, may be corrupt). -
- When any party reconstructs a blob ID from shards data and slivers, or accepts any blob purporting -to be a specific blob ID, it must check that it encodes to the correct blob ID. This process -involves re-coding the blob using the erasure correction code, and re-deriving the blob ID to -check the blob indeed matches it. This prevents a malformed blob (i.e., incorrectly erasure coded) -from ever being read with a blob ID at any correct recipient. -
- A set of slivers above the reconstruction threshold belonging to a blob ID that are either -inconsistent or lead to the reconstruction of a different ID represent an incorrect encoding -(this may happen if the user that encoded the blob was malicious and encoded it incorrectly). -Storage nodes may delete slivers belonging to inconsistently encoded blobs, and upon request -return an inconsistency proof. -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/glossary.html b/pr-preview/pr-16/glossary.html
deleted file mode 100644
index cd227484..00000000
--- a/pr-preview/pr-16/glossary.html
+++ /dev/null
@@ -1,250 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Future discussion
-In this document, we left out details of the following features:
--
-
- Shard transfer and recovery upon storage epoch change. The encoding scheme used has been designed -to allow this operation to be efficient. A storage node needs to only get data of the same -magnitude to the missing sliver data to reconstruct them. -
- Details of light clients that can be used to sample availability. Individual clients may sample -the certified blobs from Sui metadata, and sample the availability of some slivers that they -store. On-chain bounties may be used to retrieve these slivers for missing blobs. -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/highlight.css b/pr-preview/pr-16/highlight.css
deleted file mode 100644
index ba57b82b..00000000
--- a/pr-preview/pr-16/highlight.css
+++ /dev/null
@@ -1,82 +0,0 @@
-/*
- * An increased contrast highlighting scheme loosely based on the
- * "Base16 Atelier Dune Light" theme by Bram de Haan
- * (http://atelierbram.github.io/syntax-highlighting/atelier-schemes/dune)
- * Original Base16 color scheme by Chris Kempson
- * (https://github.com/chriskempson/base16)
- */
-
-/* Comment */
-.hljs-comment,
-.hljs-quote {
- color: #575757;
-}
-
-/* Red */
-.hljs-variable,
-.hljs-template-variable,
-.hljs-attribute,
-.hljs-tag,
-.hljs-name,
-.hljs-regexp,
-.hljs-link,
-.hljs-name,
-.hljs-selector-id,
-.hljs-selector-class {
- color: #d70025;
-}
-
-/* Orange */
-.hljs-number,
-.hljs-meta,
-.hljs-built_in,
-.hljs-builtin-name,
-.hljs-literal,
-.hljs-type,
-.hljs-params {
- color: #b21e00;
-}
-
-/* Green */
-.hljs-string,
-.hljs-symbol,
-.hljs-bullet {
- color: #008200;
-}
-
-/* Blue */
-.hljs-title,
-.hljs-section {
- color: #0030f2;
-}
-
-/* Purple */
-.hljs-keyword,
-.hljs-selector-tag {
- color: #9d00ec;
-}
-
-.hljs {
- display: block;
- overflow-x: auto;
- background: #f6f7f6;
- color: #000;
-}
-
-.hljs-emphasis {
- font-style: italic;
-}
-
-.hljs-strong {
- font-weight: bold;
-}
-
-.hljs-addition {
- color: #22863a;
- background-color: #f0fff4;
-}
-
-.hljs-deletion {
- color: #b31d28;
- background-color: #ffeef0;
-}
diff --git a/pr-preview/pr-16/highlight.js b/pr-preview/pr-16/highlight.js
deleted file mode 100644
index 18d24345..00000000
--- a/pr-preview/pr-16/highlight.js
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- Highlight.js 10.1.1 (93fd0d73)
- License: BSD-3-Clause
- Copyright (c) 2006-2020, Ivan Sagalaev
-*/
-var hljs=function(){"use strict";function e(n){Object.freeze(n);var t="function"==typeof n;return Object.getOwnPropertyNames(n).forEach((function(r){!Object.hasOwnProperty.call(n,r)||null===n[r]||"object"!=typeof n[r]&&"function"!=typeof n[r]||t&&("caller"===r||"callee"===r||"arguments"===r)||Object.isFrozen(n[r])||e(n[r])})),n}class n{constructor(e){void 0===e.data&&(e.data={}),this.data=e.data}ignoreMatch(){this.ignore=!0}}function t(e){return e.replace(/&/g,"&").replace(//g,">").replace(/"/g,""").replace(/'/g,"'")}function r(e,...n){var t={};for(const n in e)t[n]=e[n];return n.forEach((function(e){for(const n in e)t[n]=e[n]})),t}function a(e){return e.nodeName.toLowerCase()}var i=Object.freeze({__proto__:null,escapeHTML:t,inherit:r,nodeStream:function(e){var n=[];return function e(t,r){for(var i=t.firstChild;i;i=i.nextSibling)3===i.nodeType?r+=i.nodeValue.length:1===i.nodeType&&(n.push({event:"start",offset:r,node:i}),r=e(i,r),a(i).match(/br|hr|img|input/)||n.push({event:"stop",offset:r,node:i}));return r}(e,0),n},mergeStreams:function(e,n,r){var i=0,s="",o=[];function l(){return e.length&&n.length?e[0].offset!==n[0].offset?e[0].offset
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Walrus Glossary
-To make communication as clear and efficient as possible, we make sure to use a single term for -every Walrus entity/concept and do not use any synonyms. The following table lists various -concepts, their canonical name, how they relate to / differ from other terms.
-Italicized terms in the description indicate other specific Walrus terms contained in the table.
-| Approved name | Description |
|---|---|
| storage node (SN) | entity storing data for Walrus; holds one or several shards |
| blob | single unstructured data object stored on Walrus |
| shard | (disjoint) subset of erasure-encoded data of all blobs; at every point in time, a shard is assigned to and stored on a single SN |
| sliver | erasure-encoded data of one shard corresponding to a single blob for one of the two encodings; this contains several erasure-encoded symbols of that blob but not the blob metadata |
| blob ID | cryptographic ID computed from a blob’s slivers |
| blob metadata | metadata of one blob; in particular, this contains a hash per shard to enable the authentication of slivers and recovery symbols |
| (end) user | any entity/person that wants to store or read blobs on/from Walrus; can act as a Walrus client itself or use the simple interface exposed by publishers and caches |
| publisher | service interacting with Sui and the SNs to store blobs on Walrus; offers a simple HTTP POST endpoint to end users |
| aggregator | service that reconstructs blobs by interacting with SNs and exposes a simple HTTP GET endpoint to end users |
| cache | an aggregator with additional caching capabilities |
| (Walrus) client | entity interacting directly with the SNs; this can be an aggregator/cache, a publisher, or an end user |
| (blob) reconstruction | decoding of the primary slivers to obtain the blob; includes re-encoding the blob and checking the Merkle proofs |
| (shard/sliver) recovery | process of an SN recovering a sliver or full shard by obtaining recovery symbols from other SNs |
| storage attestation | process where SNs exchange challenges and responses to demonstrate that they are storing their currently assigned shards |
| certificate of availability (CoA) | a blob ID with signatures of SNs holding at least $2f+1$ shards in a specific epoch |
| point of availability (PoA) | point in time when a CoA is submitted to Sui and the corresponding blob is guaranteed to be available until its expiration |
| inconsistency proof | set of several recovery symbols with their Merkle proofs such that the decoded sliver does not match the corresponding hash; this proves an incorrect/inconsistent encoding by the client |
| inconsistency certificate | an aggregated signature from 2/3 of SNs (weighted by their number of shards) that they have seen and stored an inconsistency proof for a blob ID |
| storage committee | the set of SNs for a storage epoch, including metadata about the shards they are responsible for and other metadata |
| member | an SN that is part of a committee at some epoch |
| storage epoch | the epoch for Walrus as distinct to the epoch for Sui |
| availability period | the period specified in storage epochs for which a blob is certified to be available on Walrus |
":e:f.tabReplace?e.replace(/\t/g,f.tabReplace):e):e}function E(e){let n=null;const t=function(e){var n=e.className+" ";n+=e.parentNode?e.parentNode.className:"";const t=f.languageDetectRe.exec(n);if(t){var r=T(t[1]);return r||(console.warn(g.replace("{}",t[1])),console.warn("Falling back to no-highlight mode for this block.",e)),r?t[1]:"no-highlight"}return n.split(/\s+/).find(e=>p(e)||T(e))}(e);if(p(t))return;S("before:highlightBlock",{block:e,language:t}),f.useBR?(n=document.createElement("div")).innerHTML=e.innerHTML.replace(/\n/g,"").replace(/
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/interacting.html b/pr-preview/pr-16/interacting.html
deleted file mode 100644
index 13d4e1ed..00000000
--- a/pr-preview/pr-16/interacting.html
+++ /dev/null
@@ -1,234 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Walrus
-Welcome to the developer documentation for Walrus, a decentralized storage and availability protocol -designed specifically for large binary files, or "blobs". Walrus focuses on providing a robust -solution for storing unstructured content on decentralized storage nodes while ensuring high -availability and reliability even in the presence of Byzantine faults.
-Features
--
-
-
-
Storage and retrieval: Walrus supports storage operations to write and read blobs. It also -allows anyone to prove that a blob has been stored and is available for retrieval at a later -time.
-
- -
-
Cost efficiency: By utilizing advanced error correction coding, Walrus maintains storage -costs at approximately five times the size of the stored blobs and encoded parts of each blob -are stored on each storage node. This is significantly more cost-effective compared to -traditional full replication methods and much more robust against failures compared to -protocols that only store each blob on a subset of storage nodes.
-
- -
-
Integration with Sui blockchain: Walrus leverages the Sui -for coordination, attesting availability and payments. Storage space can be owned as a resource on -Sui, split, merged, and transferred. Blob storage is represented using storage objects on Sui, and -smart contracts can check whether a blob is available and for how long.
-
- -
-
Flexible access: Users can interact with Walrus through a command-line interface (CLI), -software development kits (SDKs), and web2 HTTP technologies. Walrus is designed to work well -with traditional caches and content distribution networks (CDNs), while ensuring all operations -can also be run using local tools to maximize decentralization.
-
-
Architecture and operations
-Walrus's architecture ensures that content remains accessible and retrievable even when many -storage nodes are unavailable or malicious. Under the hood it uses modern error correction -techniques based on fast linear fountain codes, augmented to ensure resilience against Byzantine -faults, and a dynamically changing set of storage nodes. The core of Walrus remains simple, and -storage node management and blob certification leverages Sui smart contracts.
-This documentation is split into several parts. The first part provides an overview of the -objectives, security properties, and architecture of the Walrus system. The second part contains -concrete documentation on the usage of Walrus. At the end, we provide a glossary, -which defines key terms used throughout the project.
-Walrus is architected to provide a reliable and cost-effective solution for large-scale blob -storage, making it an ideal choice for applications requiring decentralized, affordable, durable, -and accessible data storage.
-Sources
-This documentation is built using mdBook from source files in -github.com/MystenLabs/walrus-docs/. Please report or -fix any errors you find in this documentation in that GitHub project.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/json-api.html b/pr-preview/pr-16/json-api.html
deleted file mode 100644
index 2d5a60df..00000000
--- a/pr-preview/pr-16/json-api.html
+++ /dev/null
@@ -1,257 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Interacting with Walrus
-We provide 3 ways to interact directly with the Walrus storage system:
--
-
- Through the walrus client command line interface (CLI). -
- Through a JSON API of the walrus CLI. -
- Through an HTTP API exposed by the walrus client daemon. -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/mark.min.js b/pr-preview/pr-16/mark.min.js
deleted file mode 100644
index 16362318..00000000
--- a/pr-preview/pr-16/mark.min.js
+++ /dev/null
@@ -1,7 +0,0 @@
-/*!***************************************************
-* mark.js v8.11.1
-* https://markjs.io/
-* Copyright (c) 2014–2018, Julian Kühnel
-* Released under the MIT license https://git.io/vwTVl
-*****************************************************/
-!function(e,t){"object"==typeof exports&&"undefined"!=typeof module?module.exports=t():"function"==typeof define&&define.amd?define(t):e.Mark=t()}(this,function(){"use strict";var e="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&"function"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?"symbol":typeof e},t=function(e,t){if(!(e instanceof t))throw new TypeError("Cannot call a class as a function")},n=function(){function e(e,t){for(var n=0;n
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- JSON mode
-All Walrus client commands (except, currently, the info command) are available in JSON mode.
-In this mode, all the command line flags of the original CLI command can be specified in JSON
-format. The JSON mode therefore simplifies programmatic access to the CLI.
For example, to store a blob, run:
-walrus json \
- '{
- "config": "working_dir/client_config.yaml",
- "command": {
- "store": {
- "file": "README.md"
- }
- }
- }'
-or, to read a blob knowing the blob ID:
-walrus json \
- '{
- "config": "working_dir/client_config.yaml",
- "command": {
- "read": {
- "blob_id": "4BKcDC0Ih5RJ8R0tFMz3MZVNZV8b2goT6_JiEEwNHQo"
- }
- }
- }'
-The json command also accepts input from stdin.
The output of a json command will itself be JSON-formatted, again to simplify parsing the results
-in a programmatic way. For example, the JSON output can be piped to the jq command for parsing and
-manually extracting relevant fields.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/operations-off-chain.html b/pr-preview/pr-16/operations-off-chain.html
deleted file mode 100644
index 99d2643a..00000000
--- a/pr-preview/pr-16/operations-off-chain.html
+++ /dev/null
@@ -1,324 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Objectives and use-cases
-Walrus supports operations to store and retrieve blobs, and to prove and verify their availability. -It ensures content survives storage nodes suffering Byzantine faults and remains available and -retrievable. It provides APIs to access the stored content over a CLI, SDKs and over web2 HTTP -technologies, and supports contend delivery infrastructures like caches and content distribution -networks (CDNs).
-Under the hood, storage cost is a small fixed multiple of the size of blobs (around 5x) thanks to -advanced error correction coding, in contrast to the full replication of data traditional to -blockchains such as >100x for data stored in Sui objects. As a result much bigger resources, up to -several GiB, may be stored on Walrus at substantially lower cost compared to Sui or other -blockchains. Since encoded blobs are stored on all storage nodes Walrus also provides superior -robustness compared with designs with a small amount of replicas storing the full blob.
-Walrus uses the Sui chain for coordination and payments. Available storage is represented as Sui -objects that can be acquired, owned, split, merged and transferred. Storage space can be tied to -a stored blob for a period of time. And the resulting Sui object may be used to prove -availability on chain in smart contracts, or off chain using light clients.
-In the next chapter we discuss in details the above operations relating to storage, -retrieval and availability.
-In the future, we plan to include in Walrus some minimal governance to allow storage nodes to -change between storage epochs. Walrus is also compatible with periodic payments for continued -storage. We also plan to implement storage attestation based on challenges to get confidence that -that blobs are stored or at least available. Walrus also allows light-nodes that store small parts -of blobs to get rewards for proving availability and assisting recovery. We will cover these -topics in later documents. We also provide details of the encoding scheme in a separate document.
-Non-objectives
-There are a few things that Walrus explicitly is not:
--
-
- Walrus does not reimplement a CDN that might be geo-replicated or have less than tens of -milliseconds of latency. Instead, it ensures that traditional CDNs are usable and compatible with -Walrus caches. -
- Walrus does not re-implement a full smart contracts platform with consensus or execution. It -relies on Sui smart contracts when necessary, to manage Walrus resources and processes including -payments, storage epochs etc. -
- Walrus supports storage of any blob including encrypted blobs, however Walrus itself is not the -distributed key management infrastructure that manages and distributed encryption or decryption -keys to support a full private storage eco-system. It can however provide the storage layer of for -such infrastructures. -
Use-cases
-App builders may use Walrus in conjunction with Sui to build experiences that require large -amounts of data to be stored in a decentralized manner and possibly certified as available:
--
-
- Storage of media for NFT or dapps: Walrus can directly store and serve media such as images, -sounds, sprites, videos, other game assets, etc. This is publicly available media that can be -accessed using HTTP requests at caches to create multimedia dapps. -
- AI related use cases: Walrus can store clean data sets of training data, datasets with a -known and verified provenance, models weights, and proofs of correct training for AI models. -Or it may be used to store and ensure the availability of an AI model output. -
- Storage of long term archival of blockchain history: Walrus can be used as a lower cost -decentralized store to store blockchain history. For Sui this can include sequences of -checkpoints with all associated transaction and effects content. As well as historic snapshots -of the blockchain state, code or binaries. -
- Support availability for L2s: Walrus allows parties to certify the availability of blobs, as -required by L2s that need data to be stored and be attested as available to all. This may also -include availability of extra audit data such as validity proofs, zero knowledge proofs of -correct execution or large fraud proofs. -
- Support a full decentralized web experience: Walrus can host full decentralized web -experiences including all resources (such as js, css, html, media). These can provide content but -also host the UX of dapps to build dapps with fully decentralized front end and back ends on -chain. It brings the full "web" back in "web3". -
- Support subscription models for media: Creators can store encrypted media on Walrus and only -provide access via decryption keys to parties that have paid a subscription fee or have paid for -contents. (Note that Walrus provides the storage, encryption and decryption needs to happen off -Walrus). -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/operations-sui.html b/pr-preview/pr-16/operations-sui.html
deleted file mode 100644
index 6d97cf38..00000000
--- a/pr-preview/pr-16/operations-sui.html
+++ /dev/null
@@ -1,276 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Off-chain operations
-Walrus operations happen off Sui, but may interact with the Sui flows defining the resource life -cycle.
-Write paths
-
Systems overview of writes, illustrated above:
--
-
- A user acquires a storage resource of appropriate size and duration on-chain, either by directly -buying it on the Walrus system object, or a secondary market. A user can split, merge, and -transfer storage acquired storage resources. -
- When a user wants to write a blob, it first erasure codes it using encode, and computes its -blob ID. Then they can perform the following steps itself, or use a publisher to perform steps -on their behalf. -
- The user goes on chain (Sui) and updates a storage resource to register the blob ID with the -appropriate size and lifetime desired. This emits an event, received by storage nodes. Once the -user receives it then continues the upload. -
- The user sends each of the blob slivers and metadata to the storage nodes that currently -manages the corresponding shards. -
- A storage node managing a shard receives a sliver and checks it against the blob ID of the overall -blob. It also checks that there is a blob resource with that blob ID that is authorized to store -a blob. If correct, then it signs a statement that it holds the sliver for blob ID (and metadata) -and returns it to the user. -
- The user puts together the signatures returned from storage nodes into an availability certificate -and sends it on chain. When successfully checked an availability event for the blob ID is emitted, -and all other storage nodes seek to download any missing shards for the blob ID. This event being -emitted on Sui is the Point of Availability (PoA) for the blob ID. -
- After the PoA, and without user involvement, storage nodes sync and recover any missing slivers -that are certified. -
The user waits for 2/3 of shards signatures to return a certificate of availability. The rate of the -code is below 1/3 allowing for reconstruction if even 1/3 of shards only return the sliver. Since at -most 1/3 of the storage nodes can fail, this ensures reconstruction if a reader requests slivers -from all storage nodes that have signed the ID of the blob. Note that the full process can be -mediated by a publisher, that receives a blob and drives the process to completion.
-Refresh availability
-Since no content data is required to refresh the period of storage, refresh is conducted fully on -chain within the protocol. To request an extension to the availability period of a blob, a user -provides an appropriate storage resource. Upon success this emits an event that storage nodes -receive to extend the period each sliver is stored for.
-Inconsistent resource flow
-When a correct storage node tries to reconstruct a shard it may fail if the encoding of a blob ID -past PoA was incorrect, and will be able to extract an inconsistency proof for the blob ID. It then -generates a inconsistency certificate and uploads it on chain. The flow is as follows:
--
-
- A storage node fails to reconstruct a shard, and generates an inconsistency proof. -
- The storage node sends the blob ID and inconsistency proof to all storage nodes of the storage -epoch, and gets a signature, that it aggregates to an inconsistency certificate. -
- The storage node sends the inconsistency certificate to the Walrus smart contract, that checks it -and emits a inconsistent resource event. -
- Upon receiving a inconsistent resource event correct storage nodes delete sliver data and only -keep a metadata record to return None for the blob ID for the availability period. No storage -attestation challenges are issued for this blob ID. -
Note that a blob ID that is inconsistent will always resolve to None upon reading: this is due to -the read process running the decoding algorithm, and then re-encoding to check the blob ID is -correctly derived from a consistent encoding. This means that an inconsistency proof only reveals a -true fact to storage nodes (that may not otherwise have ran decoding), and does not change the -output of read in any case.
-Note however that partial reads leveraging the systematic nature of the encoding may return partial -reads for inconsistently encoded files. Thus if consistency and availability of reads is important -dapps should do full reads rather than partial reads.
-Read paths
-A user can read stored blobs either directly or through a cache. We discuss here the direct user -journey since this is also the operation of the cache in case of a cache miss. We assume that most -reads will happen through caches, for blobs that are hot, and will not result in requests to -storage nodes.
--
-
- The reader gets the metadata for the blob ID from any storage node, and authenticates it using -the blob ID. -
- The reader then sends a request for the shards corresponding to blob ID to storage nodes, and -waits for f+1 to respond. Sufficient requests are sent in parallel to ensure low latency for -reads. -
- The reader authenticates the slivers returned with the blob ID, reconstructs the blob, and decides -whether the contents are a valid blob or inconsistent. -
- Optionally, for a cache, the result is cached and can be served without re-construction for some -time, until it is removed from the cache. Requests for the blob to the cache return the blob -contents, or a proof the blob is inconsistently encoded. -
Challenge mechanism for storage attestation
-During an epoch a correct storage node challenges all shards to provide blob slivers past PoA:
--
-
- The list of available blobs for the period is determined by the sequence of Sui events up -to the past period. Inconsistent blobs are not challenged, and a record proving this status -can be returned instead. -
- A challenge sequence is determined by providing a seed to the challenged shard. The sequence is -then computed based both on the seed AND the content of each challenged blob ID. This creates a -sequential read dependency. -
- The response to the challenge provides the sequence of shard contents for the blob IDs in a -timely manner. -
- The challenger node uses thresholds to determine whether the challenge was passed, and reports -the result on chain. -
- The challenge / response communication is authenticated. -
Challenges provide some reassurance that the storage node actually can recover shard data in a -probabilistic manner, avoiding storage nodes getting payment without any evidence they may retrieve -shard data. The sequential nature of the challenge and some reasonable timeout also ensure that -the process is timely.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/operations.html b/pr-preview/pr-16/operations.html
deleted file mode 100644
index b14acd28..00000000
--- a/pr-preview/pr-16/operations.html
+++ /dev/null
@@ -1,231 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Operations on Sui
-Walrus uses Sui smart contracts to coordinate storage operations as resources that have a lifetime, -and payments. As well as to facilitate governance to determine the storage nodes holding each -storage shard. We outline here these operations and refer to them below as part of the read / write -paths. As a reminder, only blob metadata is ever exposed to Sui or its validators, and the content -of blobs is always stored off-chain on Walrus storage nodes and caches. The storage nodes or caches -do not have to overlap with any Sui infra (validators etc), and the storage epochs may be of -different lengths and not have the same start / end times as Sui epochs.
-Storage resource life cycle on Sui
-A number of Sui smart contracts hold the metadata of the Walrus system and all its entities.
--
-
- A Walrus system object holds the committee of storage nodes for the current storage epoch. The -system object also holds the total available space on Walrus and the price per unit of storage (1 -KiB). These values are determined by 2/3 agreement between the storage nodes for the storage -epoch. Users can pay to purchase storage space for some time duration. These space resources may -be split, merged and transferred. Later they can be used to place a blob ID into Walrus. -
- The storage fund holds funds for storing blobs over one, multiple storage epochs or -perpetually. When purchasing storage space from the system object users pay into the storage fund -separated over multiple storage epochs, and payments are made each epoch to storage nodes -according to performance (see below). -
- A user acquires some storage through the contracts or transfer, and can assign to it a blob ID, -signifying they wish to store this blob ID into it. This emits a Move resource event that -both caches and storage nodes listen to to expect and authorize off-chain storage operations. -
- Eventually a user holds an off-chain availability certificate from storage nodes for a blob -ID. The user uploads the certificate on chain to signal that the blob ID is available for an -availability period. The certificate is checked against the latest Walrus committee, -and an availability event is emitted for the blob ID if correct. This is the PoA for the -blob. -
- In case a blob ID is not correctly encoded a inconsistency proof certificate may be uploaded -on chain at a later time, and an inconsistent blob event is emitted signaling to all that the -blob ID read results will always return None. This indicates that its slivers may be deleted by -storage nodes, except for an indicator to return None. -
Users writing to Walrus, need to perform Sui transactions to acquire storage and certify blobs. -Users creating or consuming proofs for attestations of blob availability read the chain -only to prove or verify emission of events. A node that reads Walrus resources only needs to read -the blockchain to get committee metadata once per epoch, and then they request slivers directly -from storage nodes by blob ID to perform reads.
-Governance operations on Sui
-Each Walrus storage epoch is represented by the Walrus system object that contains a storage -committee and various metadata or storage nodes like the mapping between shards and storage nodes, -available space and current costs. User may go to the system object for the period and buy some -storage amount for one or more storage epochs. At each storage epoch there is a price for storage, -and the payment provided becomes part of a storage fund for all the storage epochs that span -the storage bought. There is a maximum number of storage epochs in the future for which storage can -be bought (~2 years). Storage is a resource that can be split and merged, and transferred.
-At the end of the storage epoch part of the funds in the storage fund need to be allocated to -storage nodes. The idea here is for storage nodes to perform light audits of each others, -and suggest which nodes are to be paid based on the performance of these audits.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/overview.html b/pr-preview/pr-16/overview.html
deleted file mode 100644
index d90bb8a4..00000000
--- a/pr-preview/pr-16/overview.html
+++ /dev/null
@@ -1,230 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Operations
-Walrus operations can be separated in interactions with the Sui chain, which -is used by Walrus for coordination and governance, and off-chain -interactions between clients and storage nodes.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/print.html b/pr-preview/pr-16/print.html
deleted file mode 100644
index 36763a18..00000000
--- a/pr-preview/pr-16/print.html
+++ /dev/null
@@ -1,713 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- System overview
-This chapter provides an overview of the security properties, -architecture, and encoding mechanisms of the Walrus system.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/properties.html b/pr-preview/pr-16/properties.html
deleted file mode 100644
index bb3e5b86..00000000
--- a/pr-preview/pr-16/properties.html
+++ /dev/null
@@ -1,259 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Walrus
-Welcome to the developer documentation for Walrus, a decentralized storage and availability protocol -designed specifically for large binary files, or "blobs". Walrus focuses on providing a robust -solution for storing unstructured content on decentralized storage nodes while ensuring high -availability and reliability even in the presence of Byzantine faults.
-Features
--
-
-
-
Storage and retrieval: Walrus supports storage operations to write and read blobs. It also -allows anyone to prove that a blob has been stored and is available for retrieval at a later -time.
-
- -
-
Cost efficiency: By utilizing advanced error correction coding, Walrus maintains storage -costs at approximately five times the size of the stored blobs and encoded parts of each blob -are stored on each storage node. This is significantly more cost-effective compared to -traditional full replication methods and much more robust against failures compared to -protocols that only store each blob on a subset of storage nodes.
-
- -
-
Integration with Sui blockchain: Walrus leverages the Sui -for coordination, attesting availability and payments. Storage space can be owned as a resource on -Sui, split, merged, and transferred. Blob storage is represented using storage objects on Sui, and -smart contracts can check whether a blob is available and for how long.
-
- -
-
Flexible access: Users can interact with Walrus through a command-line interface (CLI), -software development kits (SDKs), and web2 HTTP technologies. Walrus is designed to work well -with traditional caches and content distribution networks (CDNs), while ensuring all operations -can also be run using local tools to maximize decentralization.
-
-
Architecture and operations
-Walrus's architecture ensures that content remains accessible and retrievable even when many -storage nodes are unavailable or malicious. Under the hood it uses modern error correction -techniques based on fast linear fountain codes, augmented to ensure resilience against Byzantine -faults, and a dynamically changing set of storage nodes. The core of Walrus remains simple, and -storage node management and blob certification leverages Sui smart contracts.
-This documentation is split into several parts. The first part provides an overview of the -objectives, security properties, and architecture of the Walrus system. The second part contains -concrete documentation on the usage of Walrus. At the end, we provide a glossary, -which defines key terms used throughout the project.
-Walrus is architected to provide a reliable and cost-effective solution for large-scale blob -storage, making it an ideal choice for applications requiring decentralized, affordable, durable, -and accessible data storage.
-Sources
-This documentation is built using mdBook from source files in -github.com/MystenLabs/walrus-docs/. Please report or -fix any errors you find in this documentation in that GitHub project.
-Objectives and use-cases
-Walrus supports operations to store and retrieve blobs, and to prove and verify their availability. -It ensures content survives storage nodes suffering Byzantine faults and remains available and -retrievable. It provides APIs to access the stored content over a CLI, SDKs and over web2 HTTP -technologies, and supports contend delivery infrastructures like caches and content distribution -networks (CDNs).
-Under the hood, storage cost is a small fixed multiple of the size of blobs (around 5x) thanks to -advanced error correction coding, in contrast to the full replication of data traditional to -blockchains such as >100x for data stored in Sui objects. As a result much bigger resources, up to -several GiB, may be stored on Walrus at substantially lower cost compared to Sui or other -blockchains. Since encoded blobs are stored on all storage nodes Walrus also provides superior -robustness compared with designs with a small amount of replicas storing the full blob.
-Walrus uses the Sui chain for coordination and payments. Available storage is represented as Sui -objects that can be acquired, owned, split, merged and transferred. Storage space can be tied to -a stored blob for a period of time. And the resulting Sui object may be used to prove -availability on chain in smart contracts, or off chain using light clients.
-In the next chapter we discuss in details the above operations relating to storage, -retrieval and availability.
-In the future, we plan to include in Walrus some minimal governance to allow storage nodes to -change between storage epochs. Walrus is also compatible with periodic payments for continued -storage. We also plan to implement storage attestation based on challenges to get confidence that -that blobs are stored or at least available. Walrus also allows light-nodes that store small parts -of blobs to get rewards for proving availability and assisting recovery. We will cover these -topics in later documents. We also provide details of the encoding scheme in a separate document.
-Non-objectives
-There are a few things that Walrus explicitly is not:
--
-
- Walrus does not reimplement a CDN that might be geo-replicated or have less than tens of -milliseconds of latency. Instead, it ensures that traditional CDNs are usable and compatible with -Walrus caches. -
- Walrus does not re-implement a full smart contracts platform with consensus or execution. It -relies on Sui smart contracts when necessary, to manage Walrus resources and processes including -payments, storage epochs etc. -
- Walrus supports storage of any blob including encrypted blobs, however Walrus itself is not the -distributed key management infrastructure that manages and distributed encryption or decryption -keys to support a full private storage eco-system. It can however provide the storage layer of for -such infrastructures. -
Use-cases
-App builders may use Walrus in conjunction with Sui to build experiences that require large -amounts of data to be stored in a decentralized manner and possibly certified as available:
--
-
- Storage of media for NFT or dapps: Walrus can directly store and serve media such as images, -sounds, sprites, videos, other game assets, etc. This is publicly available media that can be -accessed using HTTP requests at caches to create multimedia dapps. -
- AI related use cases: Walrus can store clean data sets of training data, datasets with a -known and verified provenance, models weights, and proofs of correct training for AI models. -Or it may be used to store and ensure the availability of an AI model output. -
- Storage of long term archival of blockchain history: Walrus can be used as a lower cost -decentralized store to store blockchain history. For Sui this can include sequences of -checkpoints with all associated transaction and effects content. As well as historic snapshots -of the blockchain state, code or binaries. -
- Support availability for L2s: Walrus allows parties to certify the availability of blobs, as -required by L2s that need data to be stored and be attested as available to all. This may also -include availability of extra audit data such as validity proofs, zero knowledge proofs of -correct execution or large fraud proofs. -
- Support a full decentralized web experience: Walrus can host full decentralized web -experiences including all resources (such as js, css, html, media). These can provide content but -also host the UX of dapps to build dapps with fully decentralized front end and back ends on -chain. It brings the full "web" back in "web3". -
- Support subscription models for media: Creators can store encrypted media on Walrus and only -provide access via decryption keys to parties that have paid a subscription fee or have paid for -contents. (Note that Walrus provides the storage, encryption and decryption needs to happen off -Walrus). -
System overview
-This chapter provides an overview of the security properties, -architecture, and encoding mechanisms of the Walrus system.
-Walrus assurance and security properties
-The properties below hold true subject to the assumption that for all storage epochs 2/3 of shards -are operated by storage nodes that faithfully and correctly follow the Walrus protocol.
-Each blob is encoded using error correction into slivers and a blob ID is cryptographically -derived. For a given blob ID there is a point of availability (PoA) and an availability -period, observable through an event on the Sui chain. The following properties relate to the PoA:
--
-
- After the PoA, for a blob ID, any correct user that performs a read within the availability -period will eventually terminate and get a value V which is either blob contents F or None. -
- After PoA if two correct users perform a read and get V and V’ then V = V’. -
- A correct user with an appropriate storage resource can always perform store for a blob F with a -blob ID and advance the protocol until the PoA. -
- A read after PoA for a blob F stored by a correct user, will result in F. -
Some assurance properties ensure the correct internal processes of Walrus storage nodes. -For the purposes of defining these, an inconsistency proof proves that a blob ID was -stored by an incorrect user (and was incorrectly encoded).
--
-
- After PoA and for a blob ID stored by a correct user, a storage node is always able to recover -the correct sliver for its shards for this blob ID. -
- After PoA if a correct storage node cannot recover a sliver, it can produce an inconsistency proof -for the blob ID. -
- If a blob ID is stored by a correct user, an inconsistently proof cannot be derived for it. -
- A read by a correct user for a blob ID for which an inconsistency proof may exist returns None. -
Note that there is no delete operation and a blob ID past PoA will be available for the full -availability period.
-As a rule of thumb: before PoA it is the responsibility of a client to ensure the availability of -a blob and its upload to Walrus. After PoA it is the responsibility of Walrus as a system to -maintain the availability of the blob ID as part of its operation for the full availability period -remaining. Emission of the event corresponding to the PoA for a blob ID attests its -availability.
-Basic architecture and security assumptions
-The key actors in the Walrus architecture are the following:
--
-
- Users through clients want to store and read blobs. They are ready to pay for service -when it comes to writes, and when it comes to non-best-effort reads. Users also want to prove -the availability of a blob to third parties without the cost of sending or receiving the full -blob. Users may be malicious in various ways: they may wish to not pay for services, prove the -availability of an unavailable blobs, or modify / delete blobs without authorization, try to -exhaust resources of storage nodes, etc. -
- Storage nodes hold one or many shards within a storage epoch. Each blob is erasure -encoded in many slivers and slivers from each stored blob become part of all shards. A shard -at any storage epoch is associated with a storage node that actually stores all slivers of -the shard, and is ready to serve them. The assignment of storage nodes to shards within -storage epochs is controlled by a Sui smart contract and we assume that more than 2/3 of the -shards are managed by correct storage nodes within each storage epoch. This means that we must -tolerate up to 1/3 Byzantine storage nodes within each storage epoch and across storage epochs. -
- All clients and storage nodes operate a blockchain client (specifically on Sui), and mediate -payments, resources (space), mapping of shards to storage nodes, and metadata through blockchain -smart contracts. Users interact with the blockchain to get storage resources and certify stored -blobs, and storage nodes listen to the blockchain events to coordinate their operations. -
Walrus supports any additional number of optional infrastructure actors that can operate in a -permissionless way:
--
-
- Caches are clients that store one or more full blobs and make them available to users -over traditional web2 (HTTP, etc) technologies. They are optional in that end-users may also -operate a local cache, and perform Walrus reads over web2 technologies locally. However, cache -infrastructures may also act as CDNs, share the cost of blob reconstruction over many requests, -have better connectivity, etc. A client can always verify that reads from such infrastructures -are correct. -
- Publishers are clients that help end-users store a blob using web2 technologies, and -using less bandwidth and custom logic. They in effect receive the blob to be published. over -traditional web2 protocols (e.g., HTTP), and perform the Walrus store protocol on their behalf, -including the encoding, distribution of slivers to shards, creation of certificate of certificate, -and other on-chain actions. They are optional in that a user may directly interact with both Sui -and storage nodes to store blobs directly. An end user can always verify that a publisher -performed their duties correctly by attesting availability. -
Caches, publishers, and end-users are not considered trusted components of the system, and they may -deviate from the protocol arbitrarily. However, some of the security properties of Walrus only hold -for honest end-users that use honest intermediaries (caches and publishers). We provide means for -end-users to audit the correct operation of both caches and publishers.
-Encoding, overheads, and verification
-We summarize here the basic encoding and cryptographic techniques used in Walrus.
--
-
- Storage nodes hold one or many shards in a storage epoch, out of a larger total (say 1000) -and each shard contains one blob sliver for each blob past PoA. Each shard is assigned to a -storage node in a storage epoch. -
- An erasure code encode algorithm takes a blob, -and encodes it as $K$ symbols, such that any fraction $p$ of symbols can be used to reconstruct -the blob. Each blob sliver contains a fixed number of such symbols. -
- We select $p<1/3$ so that a third of symbols and also slivers may be used to reconstruct the blob -by the decode algorithm. The matrix used to produce the erasure code is fixed and the same -for all blobs by the Walrus system, and encoders have no discretion about it. -
- Storage nodes manage one or more shards, and corresponding sliver of each blob are distributed -to all the storage shards. As a result, the overhead of the distributed store is ~5x that of -the blob itself, no matter how many shards we have. The encoding is systematic meaning that some -storage nodes hold part of the plain blob, allowing for fast random access reads. -
Each blob is also associated with some metadata including a blob ID to allow verification:
--
-
- A blob ID is computed as an authenticator of the set of all shard data and metadata (byte size, -encoding, blob hash). We hash a sliver representation in each of the shards and add the resulting -hashes into a Merkle tree. Then the root of the Merkle tree is the blob hash used to derive the -blob ID that identifies the blob in the system. -
- Each storage node may use the blob ID to check if some shard data belongs to a blob using the -authenticated structure corresponding to the blob hash (Merkle tree). A successful check means -that the data is indeed as intended by the writer of the blob (who, remember, may be corrupt). -
- When any party reconstructs a blob ID from shards data and slivers, or accepts any blob purporting -to be a specific blob ID, it must check that it encodes to the correct blob ID. This process -involves re-coding the blob using the erasure correction code, and re-deriving the blob ID to -check the blob indeed matches it. This prevents a malformed blob (i.e., incorrectly erasure coded) -from ever being read with a blob ID at any correct recipient. -
- A set of slivers above the reconstruction threshold belonging to a blob ID that are either -inconsistent or lead to the reconstruction of a different ID represent an incorrect encoding -(this may happen if the user that encoded the blob was malicious and encoded it incorrectly). -Storage nodes may delete slivers belonging to inconsistently encoded blobs, and upon request -return an inconsistency proof. -
Operations
-Walrus operations can be separated in interactions with the Sui chain, which -is used by Walrus for coordination and governance, and off-chain -interactions between clients and storage nodes.
-Operations on Sui
-Walrus uses Sui smart contracts to coordinate storage operations as resources that have a lifetime, -and payments. As well as to facilitate governance to determine the storage nodes holding each -storage shard. We outline here these operations and refer to them below as part of the read / write -paths. As a reminder, only blob metadata is ever exposed to Sui or its validators, and the content -of blobs is always stored off-chain on Walrus storage nodes and caches. The storage nodes or caches -do not have to overlap with any Sui infra (validators etc), and the storage epochs may be of -different lengths and not have the same start / end times as Sui epochs.
-Storage resource life cycle on Sui
-A number of Sui smart contracts hold the metadata of the Walrus system and all its entities.
--
-
- A Walrus system object holds the committee of storage nodes for the current storage epoch. The -system object also holds the total available space on Walrus and the price per unit of storage (1 -KiB). These values are determined by 2/3 agreement between the storage nodes for the storage -epoch. Users can pay to purchase storage space for some time duration. These space resources may -be split, merged and transferred. Later they can be used to place a blob ID into Walrus. -
- The storage fund holds funds for storing blobs over one, multiple storage epochs or -perpetually. When purchasing storage space from the system object users pay into the storage fund -separated over multiple storage epochs, and payments are made each epoch to storage nodes -according to performance (see below). -
- A user acquires some storage through the contracts or transfer, and can assign to it a blob ID, -signifying they wish to store this blob ID into it. This emits a Move resource event that -both caches and storage nodes listen to to expect and authorize off-chain storage operations. -
- Eventually a user holds an off-chain availability certificate from storage nodes for a blob -ID. The user uploads the certificate on chain to signal that the blob ID is available for an -availability period. The certificate is checked against the latest Walrus committee, -and an availability event is emitted for the blob ID if correct. This is the PoA for the -blob. -
- In case a blob ID is not correctly encoded a inconsistency proof certificate may be uploaded -on chain at a later time, and an inconsistent blob event is emitted signaling to all that the -blob ID read results will always return None. This indicates that its slivers may be deleted by -storage nodes, except for an indicator to return None. -
Users writing to Walrus, need to perform Sui transactions to acquire storage and certify blobs. -Users creating or consuming proofs for attestations of blob availability read the chain -only to prove or verify emission of events. A node that reads Walrus resources only needs to read -the blockchain to get committee metadata once per epoch, and then they request slivers directly -from storage nodes by blob ID to perform reads.
-Governance operations on Sui
-Each Walrus storage epoch is represented by the Walrus system object that contains a storage -committee and various metadata or storage nodes like the mapping between shards and storage nodes, -available space and current costs. User may go to the system object for the period and buy some -storage amount for one or more storage epochs. At each storage epoch there is a price for storage, -and the payment provided becomes part of a storage fund for all the storage epochs that span -the storage bought. There is a maximum number of storage epochs in the future for which storage can -be bought (~2 years). Storage is a resource that can be split and merged, and transferred.
-At the end of the storage epoch part of the funds in the storage fund need to be allocated to -storage nodes. The idea here is for storage nodes to perform light audits of each others, -and suggest which nodes are to be paid based on the performance of these audits.
-Off-chain operations
-Walrus operations happen off Sui, but may interact with the Sui flows defining the resource life -cycle.
-Write paths
-
Systems overview of writes, illustrated above:
--
-
- A user acquires a storage resource of appropriate size and duration on-chain, either by directly -buying it on the Walrus system object, or a secondary market. A user can split, merge, and -transfer storage acquired storage resources. -
- When a user wants to write a blob, it first erasure codes it using encode, and computes its -blob ID. Then they can perform the following steps itself, or use a publisher to perform steps -on their behalf. -
- The user goes on chain (Sui) and updates a storage resource to register the blob ID with the -appropriate size and lifetime desired. This emits an event, received by storage nodes. Once the -user receives it then continues the upload. -
- The user sends each of the blob slivers and metadata to the storage nodes that currently -manages the corresponding shards. -
- A storage node managing a shard receives a sliver and checks it against the blob ID of the overall -blob. It also checks that there is a blob resource with that blob ID that is authorized to store -a blob. If correct, then it signs a statement that it holds the sliver for blob ID (and metadata) -and returns it to the user. -
- The user puts together the signatures returned from storage nodes into an availability certificate -and sends it on chain. When successfully checked an availability event for the blob ID is emitted, -and all other storage nodes seek to download any missing shards for the blob ID. This event being -emitted on Sui is the Point of Availability (PoA) for the blob ID. -
- After the PoA, and without user involvement, storage nodes sync and recover any missing slivers -that are certified. -
The user waits for 2/3 of shards signatures to return a certificate of availability. The rate of the -code is below 1/3 allowing for reconstruction if even 1/3 of shards only return the sliver. Since at -most 1/3 of the storage nodes can fail, this ensures reconstruction if a reader requests slivers -from all storage nodes that have signed the ID of the blob. Note that the full process can be -mediated by a publisher, that receives a blob and drives the process to completion.
-Refresh availability
-Since no content data is required to refresh the period of storage, refresh is conducted fully on -chain within the protocol. To request an extension to the availability period of a blob, a user -provides an appropriate storage resource. Upon success this emits an event that storage nodes -receive to extend the period each sliver is stored for.
-Inconsistent resource flow
-When a correct storage node tries to reconstruct a shard it may fail if the encoding of a blob ID -past PoA was incorrect, and will be able to extract an inconsistency proof for the blob ID. It then -generates a inconsistency certificate and uploads it on chain. The flow is as follows:
--
-
- A storage node fails to reconstruct a shard, and generates an inconsistency proof. -
- The storage node sends the blob ID and inconsistency proof to all storage nodes of the storage -epoch, and gets a signature, that it aggregates to an inconsistency certificate. -
- The storage node sends the inconsistency certificate to the Walrus smart contract, that checks it -and emits a inconsistent resource event. -
- Upon receiving a inconsistent resource event correct storage nodes delete sliver data and only -keep a metadata record to return None for the blob ID for the availability period. No storage -attestation challenges are issued for this blob ID. -
Note that a blob ID that is inconsistent will always resolve to None upon reading: this is due to -the read process running the decoding algorithm, and then re-encoding to check the blob ID is -correctly derived from a consistent encoding. This means that an inconsistency proof only reveals a -true fact to storage nodes (that may not otherwise have ran decoding), and does not change the -output of read in any case.
-Note however that partial reads leveraging the systematic nature of the encoding may return partial -reads for inconsistently encoded files. Thus if consistency and availability of reads is important -dapps should do full reads rather than partial reads.
-Read paths
-A user can read stored blobs either directly or through a cache. We discuss here the direct user -journey since this is also the operation of the cache in case of a cache miss. We assume that most -reads will happen through caches, for blobs that are hot, and will not result in requests to -storage nodes.
--
-
- The reader gets the metadata for the blob ID from any storage node, and authenticates it using -the blob ID. -
- The reader then sends a request for the shards corresponding to blob ID to storage nodes, and -waits for f+1 to respond. Sufficient requests are sent in parallel to ensure low latency for -reads. -
- The reader authenticates the slivers returned with the blob ID, reconstructs the blob, and decides -whether the contents are a valid blob or inconsistent. -
- Optionally, for a cache, the result is cached and can be served without re-construction for some -time, until it is removed from the cache. Requests for the blob to the cache return the blob -contents, or a proof the blob is inconsistently encoded. -
Challenge mechanism for storage attestation
-During an epoch a correct storage node challenges all shards to provide blob slivers past PoA:
--
-
- The list of available blobs for the period is determined by the sequence of Sui events up -to the past period. Inconsistent blobs are not challenged, and a record proving this status -can be returned instead. -
- A challenge sequence is determined by providing a seed to the challenged shard. The sequence is -then computed based both on the seed AND the content of each challenged blob ID. This creates a -sequential read dependency. -
- The response to the challenge provides the sequence of shard contents for the blob IDs in a -timely manner. -
- The challenger node uses thresholds to determine whether the challenge was passed, and reports -the result on chain. -
- The challenge / response communication is authenticated. -
Challenges provide some reassurance that the storage node actually can recover shard data in a -probabilistic manner, avoiding storage nodes getting payment without any evidence they may retrieve -shard data. The sequential nature of the challenge and some reasonable timeout also ensure that -the process is timely.
-Future discussion
-In this document, we left out details of the following features:
--
-
- Shard transfer and recovery upon storage epoch change. The encoding scheme used has been designed -to allow this operation to be efficient. A storage node needs to only get data of the same -magnitude to the missing sliver data to reconstruct them. -
- Details of light clients that can be used to sample availability. Individual clients may sample -the certified blobs from Sui metadata, and sample the availability of some slivers that they -store. On-chain bounties may be used to retrieve these slivers for missing blobs. -
Interacting with Walrus
-We provide 3 ways to interact directly with the Walrus storage system:
--
-
- Through the walrus client command line interface (CLI). -
- Through a JSON API of the walrus CLI. -
- Through an HTTP API exposed by the walrus client daemon. -
Using the Walrus client
-The walrus binary can be used to interact with Walrus as a client. To use it, you need a Walrus
-configuration and a Sui wallet.
-Detailed usage information is available through
walrus --help
-Storing and reading blobs from Walrus can be achieved through the following commands:
-CONFIG=working_dir/client_config.yaml # adjust for your configuration file
-walrus -c $CONFIG store <some file> # store a file
-walrus -c $CONFIG read <some blob ID> # read a blob
-JSON mode
-All Walrus client commands (except, currently, the info command) are available in JSON mode.
-In this mode, all the command line flags of the original CLI command can be specified in JSON
-format. The JSON mode therefore simplifies programmatic access to the CLI.
For example, to store a blob, run:
-walrus json \
- '{
- "config": "working_dir/client_config.yaml",
- "command": {
- "store": {
- "file": "README.md"
- }
- }
- }'
-or, to read a blob knowing the blob ID:
-walrus json \
- '{
- "config": "working_dir/client_config.yaml",
- "command": {
- "read": {
- "blob_id": "4BKcDC0Ih5RJ8R0tFMz3MZVNZV8b2goT6_JiEEwNHQo"
- }
- }
- }'
-The json command also accepts input from stdin.
The output of a json command will itself be JSON-formatted, again to simplify parsing the results
-in a programmatic way. For example, the JSON output can be piped to the jq command for parsing and
-manually extracting relevant fields.
Client Daemon mode & HTTP API
-In addition to the CLI mode, the Walrus client offers a daemon mode. In this mode, it runs a -simple web server offering HTTP interfaces to store and read blobs.
-Starting the daemon
-You can run the daemon with -different sets of API endpoints through one of the following commands:
-ADDRESS="127.0.0.1:31415" # bind the daemon to localhost and port 31415
-walrus -c $CONFIG aggregator -b $ADDRESS # run an aggregator to read blobs
-walrus -c $CONFIG publisher -b $ADDRESS # run a publisher to store blobs
-walrus -c $CONFIG daemon -b $ADDRESS # run a daemon combining an aggregator and a publisher
-The aggregator provides all read APIs, the publisher all the store APIs, and daemon provides both. -Note that the aggregator does not perform Sui on-chain actions, and therefore consumes no gas. -However, the publisher does perform actions on-chain and will consume gas. It is therefore important -to ensure only authorized parties may access it, or other measures to manage gas costs.
-HTTP API Usage
-You can then interact with the daemon through simple HTTP requests. For example, with -cURL, you can store blobs as follows:
-curl -X PUT "http://$ADDRESS/v1/store" -d "some string" # store the string `some string` for 1 storage epoch
-curl -X PUT "http://$ADDRESS/v1/store?epochs=5" -d @"some/file" # store file `some/file` for 5 storage epochs
-Blobs may be read using the following cURL command:
-curl "http://$ADDRESS/v1/<some blob ID>" # read a blob from Walrus (with aggregator or daemon)
-Modern browsers will attempt to sniff the content type for such resources, and will generally do a -good job of inferring content types for media. However, the aggregator on purpose prevents such -sniffing from inferring dangerous executable types such as javascript or style sheet types.
-Walrus Glossary
-To make communication as clear and efficient as possible, we make sure to use a single term for -every Walrus entity/concept and do not use any synonyms. The following table lists various -concepts, their canonical name, how they relate to / differ from other terms.
-Italicized terms in the description indicate other specific Walrus terms contained in the table.
-| Approved name | Description |
|---|---|
| storage node (SN) | entity storing data for Walrus; holds one or several shards |
| blob | single unstructured data object stored on Walrus |
| shard | (disjoint) subset of erasure-encoded data of all blobs; at every point in time, a shard is assigned to and stored on a single SN |
| sliver | erasure-encoded data of one shard corresponding to a single blob for one of the two encodings; this contains several erasure-encoded symbols of that blob but not the blob metadata |
| blob ID | cryptographic ID computed from a blob’s slivers |
| blob metadata | metadata of one blob; in particular, this contains a hash per shard to enable the authentication of slivers and recovery symbols |
| (end) user | any entity/person that wants to store or read blobs on/from Walrus; can act as a Walrus client itself or use the simple interface exposed by publishers and caches |
| publisher | service interacting with Sui and the SNs to store blobs on Walrus; offers a simple HTTP POST endpoint to end users |
| aggregator | service that reconstructs blobs by interacting with SNs and exposes a simple HTTP GET endpoint to end users |
| cache | an aggregator with additional caching capabilities |
| (Walrus) client | entity interacting directly with the SNs; this can be an aggregator/cache, a publisher, or an end user |
| (blob) reconstruction | decoding of the primary slivers to obtain the blob; includes re-encoding the blob and checking the Merkle proofs |
| (shard/sliver) recovery | process of an SN recovering a sliver or full shard by obtaining recovery symbols from other SNs |
| storage attestation | process where SNs exchange challenges and responses to demonstrate that they are storing their currently assigned shards |
| certificate of availability (CoA) | a blob ID with signatures of SNs holding at least $2f+1$ shards in a specific epoch |
| point of availability (PoA) | point in time when a CoA is submitted to Sui and the corresponding blob is guaranteed to be available until its expiration |
| inconsistency proof | set of several recovery symbols with their Merkle proofs such that the decoded sliver does not match the corresponding hash; this proves an incorrect/inconsistent encoding by the client |
| inconsistency certificate | an aggregated signature from 2/3 of SNs (weighted by their number of shards) that they have seen and stored an inconsistency proof for a blob ID |
| storage committee | the set of SNs for a storage epoch, including metadata about the shards they are responsible for and other metadata |
| member | an SN that is part of a committee at some epoch |
| storage epoch | the epoch for Walrus as distinct to the epoch for Sui |
| availability period | the period specified in storage epochs for which a blob is certified to be available on Walrus |
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/pr-preview/pr-16/searcher.js b/pr-preview/pr-16/searcher.js
deleted file mode 100644
index dc03e0a0..00000000
--- a/pr-preview/pr-16/searcher.js
+++ /dev/null
@@ -1,483 +0,0 @@
-"use strict";
-window.search = window.search || {};
-(function search(search) {
- // Search functionality
- //
- // You can use !hasFocus() to prevent keyhandling in your key
- // event handlers while the user is typing their search.
-
- if (!Mark || !elasticlunr) {
- return;
- }
-
- //IE 11 Compatibility from https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/startsWith
- if (!String.prototype.startsWith) {
- String.prototype.startsWith = function(search, pos) {
- return this.substr(!pos || pos < 0 ? 0 : +pos, search.length) === search;
- };
- }
-
- var search_wrap = document.getElementById('search-wrapper'),
- searchbar = document.getElementById('searchbar'),
- searchbar_outer = document.getElementById('searchbar-outer'),
- searchresults = document.getElementById('searchresults'),
- searchresults_outer = document.getElementById('searchresults-outer'),
- searchresults_header = document.getElementById('searchresults-header'),
- searchicon = document.getElementById('search-toggle'),
- content = document.getElementById('content'),
-
- searchindex = null,
- doc_urls = [],
- results_options = {
- teaser_word_count: 30,
- limit_results: 30,
- },
- search_options = {
- bool: "AND",
- expand: true,
- fields: {
- title: {boost: 1},
- body: {boost: 1},
- breadcrumbs: {boost: 0}
- }
- },
- mark_exclude = [],
- marker = new Mark(content),
- current_searchterm = "",
- URL_SEARCH_PARAM = 'search',
- URL_MARK_PARAM = 'highlight',
- teaser_count = 0,
-
- SEARCH_HOTKEY_KEYCODE = 83,
- ESCAPE_KEYCODE = 27,
- DOWN_KEYCODE = 40,
- UP_KEYCODE = 38,
- SELECT_KEYCODE = 13;
-
- function hasFocus() {
- return searchbar === document.activeElement;
- }
-
- function removeChildren(elem) {
- while (elem.firstChild) {
- elem.removeChild(elem.firstChild);
- }
- }
-
- // Helper to parse a url into its building blocks.
- function parseURL(url) {
- var a = document.createElement('a');
- a.href = url;
- return {
- source: url,
- protocol: a.protocol.replace(':',''),
- host: a.hostname,
- port: a.port,
- params: (function(){
- var ret = {};
- var seg = a.search.replace(/^\?/,'').split('&');
- var len = seg.length, i = 0, s;
- for (;i
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Walrus assurance and security properties
-The properties below hold true subject to the assumption that for all storage epochs 2/3 of shards -are operated by storage nodes that faithfully and correctly follow the Walrus protocol.
-Each blob is encoded using error correction into slivers and a blob ID is cryptographically -derived. For a given blob ID there is a point of availability (PoA) and an availability -period, observable through an event on the Sui chain. The following properties relate to the PoA:
--
-
- After the PoA, for a blob ID, any correct user that performs a read within the availability -period will eventually terminate and get a value V which is either blob contents F or None. -
- After PoA if two correct users perform a read and get V and V’ then V = V’. -
- A correct user with an appropriate storage resource can always perform store for a blob F with a -blob ID and advance the protocol until the PoA. -
- A read after PoA for a blob F stored by a correct user, will result in F. -
Some assurance properties ensure the correct internal processes of Walrus storage nodes. -For the purposes of defining these, an inconsistency proof proves that a blob ID was -stored by an incorrect user (and was incorrectly encoded).
--
-
- After PoA and for a blob ID stored by a correct user, a storage node is always able to recover -the correct sliver for its shards for this blob ID. -
- After PoA if a correct storage node cannot recover a sliver, it can produce an inconsistency proof -for the blob ID. -
- If a blob ID is stored by a correct user, an inconsistently proof cannot be derived for it. -
- A read by a correct user for a blob ID for which an inconsistency proof may exist returns None. -
Note that there is no delete operation and a blob ID past PoA will be available for the full -availability period.
-As a rule of thumb: before PoA it is the responsibility of a client to ensure the availability of -a blob and its upload to Walrus. After PoA it is the responsibility of Walrus as a system to -maintain the availability of the blob ID as part of its operation for the full availability period -remaining. Emission of the event corresponding to the PoA for a blob ID attests its -availability.
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Client Daemon mode & HTTP API
-In addition to the CLI mode, the Walrus client offers a daemon mode. In this mode, it runs a -simple web server offering HTTP interfaces to store and read blobs.
-Starting the daemon
-You can run the daemon with -different sets of API endpoints through one of the following commands:
-ADDRESS="127.0.0.1:31415" # bind the daemon to localhost and port 31415
-walrus -c $CONFIG aggregator -b $ADDRESS # run an aggregator to read blobs
-walrus -c $CONFIG publisher -b $ADDRESS # run a publisher to store blobs
-walrus -c $CONFIG daemon -b $ADDRESS # run a daemon combining an aggregator and a publisher
-The aggregator provides all read APIs, the publisher all the store APIs, and daemon provides both. -Note that the aggregator does not perform Sui on-chain actions, and therefore consumes no gas. -However, the publisher does perform actions on-chain and will consume gas. It is therefore important -to ensure only authorized parties may access it, or other measures to manage gas costs.
-HTTP API Usage
-You can then interact with the daemon through simple HTTP requests. For example, with -cURL, you can store blobs as follows:
-curl -X PUT "http://$ADDRESS/v1/store" -d "some string" # store the string `some string` for 1 storage epoch
-curl -X PUT "http://$ADDRESS/v1/store?epochs=5" -d @"some/file" # store file `some/file` for 5 storage epochs
-Blobs may be read using the following cURL command:
-curl "http://$ADDRESS/v1/<some blob ID>" # read a blob from Walrus (with aggregator or daemon)
-Modern browsers will attempt to sniff the content type for such resources, and will generally do a -good job of inferring content types for media. However, the aggregator on purpose prevents such -sniffing from inferring dangerous executable types such as javascript or style sheet types.
- -