doc: Add an introduction and a design overview to the user guide (#674)

In preparation for #334 (and other how-tos).

This adds a bit more introduction and context to the user guide that
should be independent on the actual API used (Rust/TS) and also provides
stubs for the How-Tos on the parsing/navigating the syntax tree.

When I tried to write these specific how-tos, I thought that we should
be able to define, at least from the higher level, what the CST is, how
the process looks like and add some more context (e.g. multiple versions
and why we need to create a specific version for the language, first).
Also this helps establish a bit more structure to the guide that should
help with writing more documentation/guides as we go.

Ironically, http://shader-slang.com/slang/user-guide/index.html helped
me structure this a bit as a guide that the user is welcomed to and what
they might benefit from (intro, context) before diving deeper into the
guides/context.

README.md

@@ -17,7 +17,7 @@ Written in Rust and distributed in multiple languages.
 
 This repository maintains the source code and release process for these projects:
 
--   [Slang Rust Crate + CLI](https://nomicfoundation.github.io/slang/user-guide/cargo-crate/)
+-   [Slang Rust Crate + CLI](https://nomicfoundation.github.io/slang/user-guide/rust-crate/)
 -   [Slang TypeScript Package](https://nomicfoundation.github.io/slang/user-guide/npm-package/)
 -   [Solidity Language Specification](https://nomicfoundation.github.io/slang/solidity-specification/)
 

crates/solidity/outputs/cargo/crate/README.md

@@ -13,7 +13,7 @@ A modular set of compiler APIs empowering the next generation of Solidity code a
 Written in Rust and distributed in multiple languages.
 
 -   [Announcement Post](https://medium.com/nomic-foundation-blog/slang-rethnet-2ad465fd7880)
--   [User Guide](https://nomicfoundation.github.io/slang/user-guide/cargo-crate/)
+-   [User Guide](https://nomicfoundation.github.io/slang/user-guide/rust-crate/)
 
 <br/>
 

documentation/public/user-guide/NAV.md

@@ -1,3 +1,5 @@
 -   [User Guide](./index.md)
--   [Cargo Crate](./cargo-crate/index.md)
--   [NPM Package](./npm-package/index.md)
+-   [Introduction](./introduction.md)
+-   [Concepts](./concepts.md)
+-   [Rust Crate](./rust-crate/)
+-   [NPM Package](./npm-package/)

documentation/public/user-guide/concepts.md

@@ -0,0 +1,11 @@
+At its core, Slang is a collection of APIs that are meant to analyze the source code, starting with the source code itself and ending with a rich structure that can be reasoned about. This is a departure from the classic approach of "black-box" compilers, which are handed the input and only their output can be observed.
+
+## Concrete Syntax Tree (CST)
+
+At the time of writing, Slang is capable of parsing the source code into a Concrete Syntax Tree (CST; also sometimes called "full-fidelity"), which is a tree structure of the program that also includes things like punctuation or whitespace.
+
+This is done by using the (standard) approach of lexical analysis followed by syntax analysis - the source text as a sequence of characters is recognized into a sequence of tokens (lexical analysis), which then in turn is _parsed_ into the CST.
+
+The resulting CST is a regular tree data structure that you can visit. The tree nodes are represented by the [`Node`](https://docs.rs/slang_solidity/latest/slang_solidity/cst/enum.Node.html) structure. _Rule_ nodes (aka _non-terminals_) represent sub-trees, while the _token_ nodes (aka _terminals_) are leaves and represent a lexical token (i.e. an identifier, keyword, punctuation) in the source.
+
+To help navigate the tree, we define and expose a [`Cursor`](https://docs.rs/slang_solidity/latest/slang_solidity/cursor/struct.Cursor.html) API in both Rust and TypeScript.

documentation/public/user-guide/index.md

@@ -1,4 +1,14 @@
 # User Guide
 
--   [Cargo Crate](./cargo-crate/index.md)
+Welcome to the Slang user guide! This aims to be an introduction to Slang itself, its concepts and also contains a collection of guides how you can achieve basic tasks with it.
+
+To get a good grasp on the concepts used in Slang, see the [Concepts](./concepts.md) section.
+
+## Distributions
+
+Slang itself is written in Rust and we maintain a public Rust package on [crates.io](https://crates.io/crates/slang_solidity). At the moment, we also distribute it as an [npm package](https://www.npmjs.com/package/@nomicfoundation/slang) with TypeScript interface. In the future, we plan on expanding the language bindings with Python and possibly more.
+
+For the guides related to specific distributions, see below:
+
+-   [Rust Crate](./rust-crate/index.md)
 -   [NPM Package](./npm-package/index.md)

documentation/public/user-guide/introduction.md

@@ -0,0 +1,18 @@
+## What is Slang?
+
+Slang is intended to be a modular Solidity compiler, specifically targeting code analysis and developer tooling. This means servicing tools with domain-specific APIs and, in general, facilitating working with and analyzing the Solidity source code. If you're in the editor writing Solidity or performing linting or additional validation, there's a chance that you are, or could be, running Slang!
+
+## What Slang is not?
+
+First and foremost, it is not a replacement for `solc`, the standard Solidity compiler. We do not plan at the moment to support emitting optimized EVM bytecode for use in production. Secondly, it does not perform formal verification of contracts or Solidity logic in general. However, other tools that serve this purpose are intended to be built on top of it.
+
+## Supporting multiple versions
+
+The Solidity programming language has evolved quite a bit since its inception. Some features were introduced, some changed, while some eventually became obsolete and were removed altogether.
+
+While it's good for a programming language to evolve and better serve the needs of its users, not being able to easily upgrade or re-deploy existing contracts poses a unique challenge. Developer tooling must be able to understand and consume older contracts that are still being used on the blockchain, written in older versions of Solidity.
+
+Because of that, Slang must be able to reason about different versions of Solidity; how the language grammar, name binding rules, and semantics have changed across different versions. One of our goals is to document differences as part of our [Solidity Specification](../solidity-specification/index.md).
+
+This is why, instead of having to download separate versions of the tool for each Solidity version, you can access the Slang language APIs by simply specifying the Solidity version that you want to work with.  
+See [language-specific guides](../#distributions) for more examples.

documentation/public/user-guide/npm-package/NAV.md

@@ -0,0 +1,2 @@
+-   [Index](./index.md)
+-   [How to parse a Solidity file](./how-to-parse-a-file/)

documentation/public/user-guide/npm-package/how-to-parse-a-file/index.md

@@ -0,0 +1,3 @@
+# How to parse a Solidity file
+
+--8<-- "crates/solidity/inputs/language/snippets/under-construction.md"

documentation/public/user-guide/npm-package/index.md

@@ -1,5 +1,7 @@
 # NPM Package
 
+[![npm](https://img.shields.io/npm/v/@nomicfoundation/slang?label=NPM%20Package&logo=npm&logoColor=white)](https://www.npmjs.com/package/@nomicfoundation/slang)
+
 ## Installation
 
 Start by adding the Slang package as a dependency to your project:

documentation/public/user-guide/rust-crate/NAV.md

@@ -0,0 +1,2 @@
+-   [Index](./index.md)
+-   [How to parse a Solidity file](./how-to-parse-a-file/)

documentation/public/user-guide/rust-crate/how-to-parse-a-file/index.md

@@ -0,0 +1,3 @@
+# How to parse a Solidity file
+
+--8<-- "crates/solidity/inputs/language/snippets/under-construction.md"

documentation/public/user-guide/cargo-crate/index.md → documentation/public/user-guide/rust-crate/index.md

@@ -1,4 +1,11 @@
-# Cargo Crate
+# Rust Crate
+
+[![crate](https://img.shields.io/crates/v/slang_solidity?label=Rust%20Crate&logo=rust&logoColor=white)](https://crates.io/crates/slang_solidity)
+[![docs.rs](https://img.shields.io/docsrs/slang_solidity?logo=docs.rs&label=docs.rs)](https://docs.rs/slang_solidity/latest/slang_solidity)
+
+The Rust package is published to crates.io as [`slang_solidity`](https://crates.io/crates/slang_solidity) ([docs](https://docs.rs/slang_solidity/latest/slang_solidity/)).
+
+It can be used both as a regular Rust dependency and as a standalone CLI (installable with Cargo).
 
 ## Installation