[DOCs]: New docs framework (#5017)

* started todo list for rework * startd long todo list * startd long todo list * remove ts docs from ts sdk dir * started new docs draft * rearranged code example dir structure * modified code component filepaths * first pass rust sdk * small shift typescript org * updated todo list * consolidating images folders in one place * first pass @ operator docs * gen updates * sdk in its own dir * first pass developers structure * first pass network structure * structure * add licensing * moved old docs -> old_docs dir for clarity when devving * moving around new docs - think this is the final dir structure * updated todo list * new autodoc version (#4781) * Update rework_todo.md * quick first sketch of landing page * rework of structure of developers * added arch and concepts stubs * add new bits to todo list * new list * tweak to overview * mixnet node overview * tweak overview * first pass new arch * first pass concepts * first pass traffic * more network docs * moved some chain files to the dev portal stubs * removed old reference to archive * note to client * concepts 2nd pass * crypto first proper pass, sphinx * stub for not p2p * structure change * traffic 2nd pass * misc * hid root index * overhaul arch * overhaul arch * add links + tweaks * update todo list * updating nyx section * added zknym docs * added zknym docs * note on where to find deployed info * smart contracts done * started moving integrations docs over from ts sdk * pass @ integration page * todo for the tldr overview * added ffi stub files * updated todo list * move sdks to developers * initial pass at new clients overview for developers * rework intro * add echo serv to tools * sidebar autocollapse * integration overview work + tools * concepts overview for devporta * stub * more for networking pages * added to concepts in dev portal * updated arch * crypto overview page * typo fix * add credential stub * first pass concepts done * start reorg of rust sdk docs * reorg + added FFI table * added no scroll to inline code * finished ffi overview page * first pass @ rest of rust sdk doc * first pass ffi * tweaks * added testnet example + note to custom topology example overview * stripped unnecessary stuff from TS * tweaks to ffi * updated faq * first pass tcpproxy * commit before moving image dir * moved images/ to correct place * started on client redo * chain first pass * moved cli wallet out of tools * first pass new ws client * new chain info, left todo links in * links * more links * chain registry * added echo server to tools * rust sdk links * ts sdk links * final linkchecks * redo acks diagram as mermaid * add mermaid flow diagram * added links for codecs + full flow diagram * removed todo * remove forced dark mode * diagram + concepts overview * small correction re tcpproxy ffi * remove diagram title * new sock5 diagram, minor client docs tweaks * diagrams * change order in list * added note for standalone: can be accessed via sdk * tweaks * replaced old diagram with mermaid * fixed link * hardcoded import version for the moment * update deps * remove test component * recreated tools dir * remove tools dir moved to wrong palce * prebuild and predev script for autodoc commands * make script own command instead of prebuild * made code blocks sh * updated autogenerated docs * temp * auto commit generated command files * add link to autodoc generated files * updated autodoc for committing changing else exit * auto commit generated command files * updated readme * make subcommand headers smaller * removed mdbook related scripts * update readme * update readme * removed backups of root meta.json * cherry pick yana commits + some extra config in theme * update readme * update theme: width of page and padding * some more themeing * changed erroneous note * docs redirects first pass * tweaking * new pages + rest of redirects for old docs/ * brought in archive + done rewrites for devportal * cherry pick yana landingpage * tweaked landing page component * changed theme of mermaid diagram to match everything else * updated todo list * [DOCs]: Operators rework to next.js (#4930) * initialise operators guides v2 * new introduction page * add variables csv and page * add baseurl to allow short path * add sandbox page * added building from source page * add binary pages * add preliminary steps * clean preliminary steps dir * syntax edit * syntax edit * add configuration page * create new proxy configuration page * create new proxy configuration page * create bonding.mdx page * correct images path * syntax edit * add new validator setup page * add api setup page * add nyx configuration page * add nym node and maintenance pages * finish maintenance and add nymvisor conf page * add manual upgrade page * add nymvisor upgrade page * add performance testing page and dir * add node api check page * add explore nym scripts page * add testing pages * fix menu issue by moving snippets to coomponents * add all troubleshooting pages * add general faq page * add nym node faq page * add nyx faq page * revamp legal forum to community counsel and add all pages * rewire relative paths to new structure * simplify setup and remove lock file * syntax fix * rm package.json * re add package.json, rm package-lock.json * removed old books from commit * address review comments --------- Co-authored-by: mfahampshire <[email protected]> Co-authored-by: mx <[email protected]> * tweak client links * also moved matrix images to correct place * Max/fix links new docs framework (#4989) * tweak client links * standardise images in public/ * old images move to public/archive * rename overview to more descriptive * links (#4990) * links * removed todos * updated todo list * minor themeing * operator redirects * pick yana's edits: remove specified callout theming * added todo comments for old ts sdk redirects * [new/docs/operators]: Create archive section - PR ready to merge (#5004) * [new-docs/operators] : Fix callout syntax (#5006) * fix callout syntax from color to type * correct callout from danger to warning * update footer * updated footer * finalised rewrites * tweaks to clients and reintroduced old examples page * update todo * Max/individual command autodocs (#5015) * auto commit generated command files * added to autodoc.sh: build all binaries before running * autodoc move individual command outputs to components * Max/individual command autodocs (#5018) * updated autodoc script * updated autodoc script for fix + reintroduced gitignore file for generated markdown * auto commit generated command files * auto commit generated command files * added command-outputs to autodoc script * fix merge conflicts * repush components * remove old docs dirs * auto commit generated command files * auto commit generated command files * updated messages paradigm with the standalone proxies * [NEW-DOCs/operators]: Command output, accordion, api scraping & all final tasks (#5026) * add custom scripts, create prebuild to import data to pages * update after latest prebuild * auto commit generated command files * add accordion component * add changbelog page * add node_api_check outputs * finish all command outputs * more accordions beautifications * finish accordion * PR ready to go * address review comments --------- Co-authored-by: mfahampshire <[email protected]> * Adjust padding * Fix responsive design * cherry pick yana landingpage flex update * reremove old docs * added dependencies to readme * pushing build attempt changes * fix merge errors, path errors, dump uselss dinosaurs - BUILT THE F*N DOCS w success * moved prebuild to its own script * generate timenow * auto commit generated command files * remove comment * auto commit generated command files * auto commit generated command files * auto commit generated command files * build from new configs * add mdx type as explicit dep * remove rc from version in package * change predev script * update readme with scripts * update general info * add license * auto commit generated command files * add updated components * removed old examples page for the moment * remove old list will reintroduce hidden behind gitignore for future * reintroduce todo list behind gitignore * added standalone tcpproxy binary info * nothing change for redeploy test * make build standalone * updated readme * working on new cd * remove export * updated ci/cd for docs * added ci script for dist * hide text on laptop wide screen * add pnpm to ci/cd * add pnpm version to ci/cd * add default dir to ci/cd * change path to script * update projct name ci * lint ci branch ignore * add basePath to next.config.js * update doc rewrites * revert basePath addition * update basePath * add mobile styles * fix responsive style * remove old ts sdk docs workflow * temp remove autodoc from workspace * update sidebar for clarity: crypto = cryptography * ignore documentation in pr-validation workflow --------- Co-authored-by: Yana <[email protected]> Co-authored-by: import this <[email protected]> Co-authored-by: fmtabbara <[email protected]>
nymtech · Nov 4, 2024 · d6599b2 · d6599b2
1 parent 5e0417e
commit d6599b2
Show file tree

Hide file tree

Showing 673 changed files with 36,416 additions and 14,844 deletions.
diff --git a/.github/workflows/cd-docs.yml b/.github/workflows/cd-docs.yml
@@ -6,20 +6,27 @@ on:
 jobs:
   build:
     runs-on: arc-ubuntu-20.04
+    defaults:
+      run:
+        working-directory: documentation/docs
     steps:
       - uses: actions/checkout@v4
       - name: Install Dependencies (Linux)
         run: sudo apt-get update && sudo apt-get install -y build-essential curl wget libssl-dev libudev-dev squashfs-tools protobuf-compiler git python3 && sudo apt-get update --fix-missing
       - name: Install pip3
-        run: sudo apt install -y python3-pip  
+        run: sudo apt install -y python3-pip
       - name: Install Python3 modules
         run: sudo pip3 install pandas tabulate
       - name: Install rsync
         run: sudo apt-get install rsync
       - uses: rlespinasse/[email protected]
+      - name: Setup pnpm
+        uses: pnpm/[email protected]
+        with:
+          version: 9
       - uses: actions/setup-node@v4
         with:
-          node-version: 18
+          node-version: 20
       - name: Install Rust stable
         uses: actions-rs/toolchain@v1
         with:
@@ -29,33 +36,13 @@ jobs:
         with:
           command: build
           args: --workspace --release
-      - name: Install mdbook and plugins
-        run: cd documentation && ./install_mdbook_deps.sh
-      - name: Remove existing Nym config directory (`~/.nym/`)
-        run: cd documentation && ./remove_existing_config.sh
-        continue-on-error: false
-# This is the original flow
-#      - name: Build all projects in documentation/ & move to ~/dist/docs/
-#        run: cd documentation && ./build_all_to_dist.sh
-
-# This is a workaround replacement which builds on the last working commit b332a6b55668f60988e36961f3f62a794ba82ddb and then on current branch
-      - name: Save current branch to ~/current_branch
-        run: git rev-parse --abbrev-ref HEAD > ~/current_branch
-      - name: Git pull, reset & switch to b332a6b55668f60988e36961f3f62a794ba82ddb
-        run: git pull && git reset --hard && git checkout b332a6b55668f60988e36961f3f62a794ba82ddb
-      - name: Build all projects in documentation/ & move to ~/dist/docs/ from b332a6b55668f60988e36961f3f62a794ba82ddb
-        run: cd documentation && ./build_all_to_dist.sh
 
-      - name: Switch to current branch
-        run: git checkout $echo "$(cat ~/current_branch)"
-      - name: Build all projects in documentation/ & move to ~/dist/docs/ on current branch
-        run: cd documentation && ./build_all_to_dist.sh && rm ~/current_branch
-
-# End of replacemet
-
-      - name: Post process
-        run: cd documentation && ./post_process.sh
-        continue-on-error: false
+      - name: Install project dependencies
+        run: pnpm i
+      - name: Build project
+        run: pnpm run build
+      - name: Move files to /dist/
+        run: ../scripts/move-to-dist.sh
 
       - name: Create Vercel project file
         uses: mobiledevops/secret-to-file-action@v1

diff --git a/.github/workflows/ci-docs.yml b/.github/workflows/ci-docs.yml
@@ -3,28 +3,35 @@ name: ci-docs
 on:
   workflow_dispatch:
   push:
-    branches-ignore: master
+    branches-ignore: [master]
     paths:
-      - 'documentation/docs/**'
-      - '.github/workflows/ci-docs.yml'
+      - "documentation/docs/**"
+      - ".github/workflows/ci-docs.yml"
 
 jobs:
   build:
     runs-on: arc-ubuntu-20.04
+    defaults:
+      run:
+        working-directory: documentation/docs
     steps:
       - uses: actions/checkout@v4
       - name: Install Dependencies (Linux)
         run: sudo apt-get update && sudo apt-get install -y build-essential curl wget libssl-dev libudev-dev squashfs-tools protobuf-compiler git python3 && sudo apt-get update --fix-missing
       - name: Install pip3
-        run: sudo apt install -y python3-pip  
+        run: sudo apt install -y python3-pip
       - name: Install Python3 modules
         run: sudo pip3 install pandas tabulate
       - name: Install rsync
         run: sudo apt-get install rsync
       - uses: rlespinasse/[email protected]
+      - name: Setup pnpm
+        uses: pnpm/[email protected]
+        with:
+          version: 9
       - uses: actions/setup-node@v4
         with:
-          node-version: 18
+          node-version: 20
       - name: Install Rust stable
         uses: actions-rs/toolchain@v1
         with:
@@ -34,30 +41,13 @@ jobs:
         with:
           command: build
           args: --workspace --release
-      - name: Install mdbook and plugins
-        run: cd documentation && ./install_mdbook_deps.sh
-      - name: Remove existing Nym config directory (`~/.nym/`)
-        run: cd documentation && ./remove_existing_config.sh
-        continue-on-error: false
-
-# This is the original flow
-#      - name: Build all projects in documentation/ & move to ~/dist/docs/
-#        run: cd documentation && ./build_all_to_dist.sh
-
-# This is a workaround replacement which builds on the last working commit b332a6b55668f60988e36961f3f62a794ba82ddb and then on current branch
-      - name: Save current branch to ~/current_branch
-        run: git rev-parse --abbrev-ref HEAD > ~/current_branch
-      - name: Git pull, reset & switch to b332a6b55668f60988e36961f3f62a794ba82ddb
-        run: git pull && git reset --hard && git checkout b332a6b55668f60988e36961f3f62a794ba82ddb
-      - name: Build all projects in documentation/ & move to ~/dist/docs/ from b332a6b55668f60988e36961f3f62a794ba82ddb
-        run: cd documentation && ./build_all_to_dist.sh
-
-      - name: Switch to current branch
-        run: git checkout $echo "$(cat ~/current_branch)"
-      - name: Build all projects in documentation/ & move to ~/dist/docs/ on current branch
-        run: cd documentation && ./build_all_to_dist.sh && rm ~/current_branch
 
-# End of replacemet
+      - name: Install project dependencies
+        run: pnpm i
+      - name: Build project
+        run: pnpm run build
+      - name: Move files to /dist/
+        run: ../scripts/move-to-dist.sh
 
       - name: Deploy branch to CI www
         continue-on-error: true
@@ -68,5 +58,5 @@ jobs:
           SOURCE: "dist/docs/"
           REMOTE_HOST: ${{ secrets.CI_WWW_REMOTE_HOST }}
           REMOTE_USER: ${{ secrets.CI_WWW_REMOTE_USER }}
-          TARGET: ${{ secrets.CI_WWW_REMOTE_TARGET }}/docs-${{ env.GITHUB_REF_SLUG }}
+          TARGET: ${{ secrets.CI_WWW_REMOTE_TARGET }}/docs-nextra-${{ env.GITHUB_REF_SLUG }}
           EXCLUDE: "/node_modules/"
diff --git a/.github/workflows/ci-sdk-docs-typescript.yml b/.github/workflows/ci-sdk-docs-typescript.yml
diff --git a/.github/workflows/pr-validation.yml b/.github/workflows/pr-validation.yml
@@ -4,7 +4,9 @@ on:
   pull_request:
     branches:
       - develop
-      - 'release/**'
+      - "release/**"
+    paths-ignore:
+      - "documentation/**"
     types:
       - labeled
       - unlabeled

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -98,6 +98,7 @@ members = [
     "common/wasm/utils",
     "common/wireguard",
     "common/wireguard-types",
+    # "documentation/autodoc",
     "explorer-api",
     "explorer-api/explorer-api-requests",
     "explorer-api/explorer-client",
@@ -203,7 +204,7 @@ axum-extra = "0.9.4"
 base64 = "0.22.1"
 bincode = "1.3.3"
 bip39 = { version = "2.0.0", features = ["zeroize"] }
-bit-vec = "0.7.0" # can we unify those?
+bit-vec = "0.7.0"                                     # can we unify those?
 bitvec = "1.0.0"
 blake3 = "1.5.4"
 bloomfilter = "1.0.14"

diff --git a/documentation/.gitignore b/documentation/.gitignore
@@ -0,0 +1 @@
+todo.md
diff --git a/documentation/README.md b/documentation/README.md
@@ -1,30 +1,53 @@
-# Documentation
+# Nym Docs v2
+
+This is v2 of the nym docs, condensed from various mdbooks projects that we had previously.
+
+These docs are hosted at [nymtech.net/docs](www.nymtech.net/docs).
 
 ## Doc projects
-Each directory contains a readme with more information about running and contributing to the projects. Each is built with [`mdbook`](https://rust-lang.github.io/mdBook/index.html) - use `mdbook serve` to build and serve them (defaults to `localhost:3000`).
-* `docs` contains technical documentation hosted at [https://nymtech.net/docs](https://nymtech.net/docs)
-* `dev-portal` contains developer documentation hosted at [https://nymtech.net/developers](https://nymtech.net/developers)
-* `operators` contains node setup and maintenance guides hosted at [https://nymtech.net/operators](https://nymtech.net/operators)
+`docs/pages/` contains several subdirs, each hosting a subsection of the docs:
+* `network` contains key concepts, cryptosystems, architecture.
+* `developers` contains key concepts for developers, required architecture, and Rust/Typescript SDK docs.
+* `operators` contains node setup and maintenance guides.
+
+## Local development
+### Dependencies
+Our `prebuild` script relies on the following:
+- `python`
+- `pip`
+- [`pandas`](https://pandas.pydata.org/)
+- [`tabulate`](https://pypi.org/project/tabulate/)
+- `jq`
+
+Otherwise make sure to have `node` installed.
 
-> If you are looking for the Typescript SDK documentation located at [sdk.nymtech.net](https://sdk.nymtech.net) this can be found in `../sdk/typescript/docs/`
+### Serve Local (Hot Reload)
+```sh
+pnpm i
+pnpm run dev
+```
 
-## Contribution 
-* If you wish to add to the documentation please create a PR against this repo. 
-* If you are **adding a plugin dependency** make sure to also **add that to the list of plugins in `install_mdbook_deps.sh` line 12**. 
+Open `http://localhost:3000`.
 
-## Scripts 
-* `bump_versions.sh` allows you to update the ~~`platform_release_version` and~~ `wallet_release_version` variable~~s~~ in the `book.toml` of each mdbook project at once. You can also optionally update the `minimum_rust_version` as well. Helpful for lazy-updating when cutting a new version of the docs. 
+## Build
+```sh
+pnpm run build
+```
 
-* The following scripts are used by the `ci-dev.yml` and `cd-dev.yml` scripts (located in `../.github/workflows/`):
-  * `build_all_to_dist.sh` is used for building all mdbook projects and moving the rendered html to `../dist/` to be rsynced with various servers. 
-  * `install_mdbook_deps.sh` checks for an existing install of mdbook (and plugins), uninstalls them, and then installs them on a clean slate. This is to avoid weird dependency clashes if relying on an existing mdbook version. 
-  * `post_process.sh` is used to post process CSS/image/href links for serving several mdbooks from a subdirectory. 
-  * `removed_existing_config.sh` is used to check for existing nym client/node config files on the CI/CD server and remove it if it exists. This is to mitigate issues with `mdbook-cmdrun` where e.g. a node is already initialised, and the command fails.   
+The static output will be in `./out`;
 
-## CI/CD 
-Deployment of the docs is partially automated and partially manual. 
-* `ci-docs.yml` will run on pushes to all branches **apart from `master`**
-* `cd-docs.yml` must be run manually. This pushes to a staging branch which then must be manually promoted to production. 
+## Contribution
+* If you wish to add to the documentation please create a PR against this repo, with a `patch` against `develop`.
+
+## Scripts
+* There are several autogenerated command files for clients and binaries. These are generated with `generate:commands`, which runs the `autodoc` rust binary, moves the files to their required places, and then if there is an update, commits them to git. This is for remote deployments.
+* Python scripts TODO
+
+### Autodoc
+`autodoc` is a script that generates markdown files containing commands and their output (both command and `--help` output). For the moment the binaries and their commands are manually configured in the script.
+
+## CI/CD
+TODO
 
 ## Licensing and copyright information
 This is a monorepo and components that make up Nym as a system are licensed individually, so for accurate information, please check individual files.
@@ -36,4 +59,3 @@ As a general approach, licensing is as follows this pattern:
 * Nym applications and binaries are [GPL-3.0-only](https://www.gnu.org/licenses/)
 
 * Used libraries and different components are [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) or [MIT](https://mit-license.org/)
-
diff --git a/documentation/autodoc/.gitignore b/documentation/autodoc/.gitignore
@@ -0,0 +1 @@
+autodoc-generated-markdown/*