Update docs (#3478)

* Updated docs

* Updated docs

* Update docs

README.md

@@ -7,47 +7,44 @@
 
 # Armada
 
-Armada is a multi-[Kubernetes](https://kubernetes.io/docs/concepts/overview/) cluster batch job scheduler.
+Armada is a system built on top of [Kubernetes](https://kubernetes.io/docs/concepts/overview/) for running batch workloads. With Armada as middleware for batch, Kubernetes can be a common substrate for batch and service workloads.  Armada is used in production and can run millions of jobs per day across tens of thousands of nodes. 
 
-Armada is designed to address the following issues:
+Armada addresses the following limitations of Kubernetes:
 
-1. A single Kubernetes cluster can not be scaled indefinitely, and managing very large Kubernetes clusters is [challenging](https://openai.com/blog/scaling-kubernetes-to-7500-nodes/). Hence, Armada is a multi-cluster scheduler built on top of several Kubernetes clusters.
-2. Achieving very high throughput using the in-cluster storage backend, etcd, is [challenging](https://etcd.io/docs/v3.5/op-guide/performance/). Hence, queueing and scheduling is performed partly out-of-cluster using a specialized storage layer.
+1. Scaling a single Kubernetes cluster beyond a certain size is [challenging](https://openai.com/blog/scaling-kubernetes-to-7500-nodes/). Hence, Armada is designed to effectively schedule jobs across many Kubernetes clusters. Many thousands of nodes can be managed by Armada in this way.
+2. Achieving very high throughput using the in-cluster storage backend, etcd, is [challenging](https://etcd.io/docs/v3.5/op-guide/performance/). Hence, Armada performs queueing and scheduling out-of-cluster using a specialized storage layer. This allows Armada to maintain queues composed of millions of jobs.
+3. The default [kube-scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) is not suitable for batch. Instead, Armada includes a novel multi-Kubernetes cluster scheduler with support for important batch scheduling features, such as:
+   * Fair queuing and scheduling across multiple users. Based on dominant resource fairness.
+   * Resource and job scheduling rate limits.
+   * Gang-scheduling, i.e., atomically scheduling sets of related jobs.
+   * Job preemption, both to run urgent jobs in a timely fashion and to balance resource allocation between users.
 
-Armada is designed primarily for machine learning, AI, and data analytics workloads, and to:
+Armada also provides features to help manage large compute clusters effectively, including:
 
-- Manage compute clusters composed of tens of thousands of nodes in total.
-- Schedule a thousand or more pods per second, on average.
-- Enqueue tens of thousands of jobs over a few seconds.
-- Divide resources fairly between users.
-- Provide visibility for users and admins.
-- Ensure near-constant uptime.
+* Detailed analytics exposed via [Prometheus](https://prometheus.io/) showing how the system behaves and how resources are allocated.
+* Automatically removing nodes exhibiting high failure rates from consideration for scheduling.
+* A mechanism to earmark nodes for a particular set of jobs, but allowing them to be used by other jobs when not used for their primary purpose.
 
-Armada is a [CNCF](https://www.cncf.io/) Sandbox project used in production at [G-Research](https://www.gresearch.co.uk/).
+Armada is designed with the enterprise in mind; all components are secure and highly available.
 
-For an overview of Armada, see these videos:
+Armada is a [CNCF](https://www.cncf.io/) Sandbox project and is used in production at [G-Research](https://www.gresearch.co.uk/).
+
+For an overview of Armada, see the following videos:
 
 - [Armada - high-throughput batch scheduling](https://www.youtube.com/watch?v=FT8pXYciD9A)
 - [Building Armada - Running Batch Jobs at Massive Scale on Kubernetes](https://www.youtube.com/watch?v=B3WPxw3OUl4)
 
-Armada adheres to the CNCF [Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
+The Armada project adheres to the CNCF [Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
 
 ## Documentation
 
-For an overview of the architecture and design of Armada, and instructions for submitting jobs, see:
-
+For documentation, see the following:
 
-- [Components overview](./docs/design/relationships_diagram.md)
-- [Scheduler](./docs/design/scheduler.md)
-- [Architecture](./docs/design/architecture.md)
+- [System overview](./docs/system_overview.md)
+- [Scheduler](./docs/scheduler.md)
 - [User guide](./docs/user.md)
 - [Quickstart](./docs/quickstart/index.md)
-
-For a full developer guide, see:
-
 - [Development guide](./docs/developer.md)
-
-For API reference, see:
 - [API Documentation](./docs/developer/api.md)
 
 We expect readers of the documentation to have a basic understanding of Docker and Kubernetes; see, e.g., the following links:
@@ -56,6 +53,7 @@ We expect readers of the documentation to have a basic understanding of Docker a
 - [Kubernetes overview](https://kubernetes.io/docs/concepts/overview/)
 
 ## Contributions
+
 Thank you for considering contributing to Armada!
 We want everyone to feel that they can contribute to the Armada Project.
 Your contributions are valuable, whether it's fixing a bug, implementing a new feature, improving documentation, or suggesting enhancements.

docs/consistency.md

@@ -0,0 +1,22 @@
+# A note on consistency
+
+The data stream approach taken by Armada is not the only way to maintain consistency across views. Here, we compare this approach with the two other possible solutions.
+
+Armada stores its state across several databases. Whenever Armada receives an API call to update its state, all those databases need to be updated. However, if each database were to be updated independently it is possible for some of those updates to succeed while others fail, leading to an inconsistent application state. It would require complex logic to detect and correct for such partial failures. However, even with such logic we could not guarantee that the application state is consistent; if Armada crashes before it has had time to correct for the partial failure the application may remain in an inconsistent state.
+
+There are three commonly used approaches to address this issue:
+
+* Store all state in a single database with support for transactions. Changes are submitted atomically and are rolled back in case of failure; there are no partial failures.
+* Distributed transaction frameworks (e.g., X/Open XA), which extend the notation of transactions to operations involving several databases.
+* Ordered idempotent updates.
+
+The first approach results in tight coupling between components and would limit us to a single database technology. Adding a new component (e.g., a new dashboard) could break existing component since all operations part of the transaction are rolled back if one fails. The second approach allows us to use multiple databases (as long as they support the distributed transaction framework), but components are still tightly coupled since they have to be part of the same transaction. Further, there are performance concerns associated with these options, since transactions may not be easily scalable. Hence, we use the third approach, which we explain next.
+
+First, note that if we can replay the sequence of state transitions that led to the current state, in case of a crash we can recover the correct state by truncating the database and replaying all transitions from the beginning of time. Because operations are ordered, this always results in the same end state. If we also, for each database, store the id of the most recent transition successfully applied to that database, we only need to replay transitions more recent than that. This saves us from having to start over from a clean database; because we know where we left off we can keep going from there. For this to work, we need transactions but not distributed transactions. Essentially, applying a transition already written to the database results in a no-op, i.e., the updates are idempotent (meaning that applying the same update twice has the same effect as applying it once).
+
+The two principal drawbacks of this approach are:
+
+* Eventual consistency: Whereas the first two approaches result in a system that is always consistent, with the third approach, because databases are updated independently, there will be some replication lag during which some part of the state may be inconsistent.
+* Timeliness: There is some delay between submitting a change and that change being reflected in the application state.
+
+Working around eventual consistency requires some care, but is not impossible. For example, it is fine for the UI to show the a job as "running" for a few seconds after the job has finished before showing "completed". Regarding timeliness, it is not a problem if there is a few seconds delay between a job being submitted and the job being considered for queueing. However, poor timeliness may lead to clients (i.e., the entity submitting jobs to the system) not being able to read their own writes for some time, which may lead to confusion (i.e., there may be some delay between a client submitting a job a that job showing as "pending"). This issue can be worked around by storing the set of submitted jobs in-memory either at the client or at the API endpoint.