+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial1_Basic_QA_Pipeline.ipynb)
+
+Question Answering can be used in a variety of use cases. A very common one: Using it to navigate through complex knowledge bases or long documents ("search setting").
+
+A "knowledge base" could for example be your website, an internal wiki or a collection of financial reports.
+In this tutorial we will work on a slightly different domain: "Game of Thrones".
+
+Let's see how we can use a bunch of Wikipedia articles to answer a variety of questions about the
+marvellous seven kingdoms.
+
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+
+```python
+from haystack.utils import clean_wiki_text, convert_files_to_dicts, fetch_archive_from_http, print_answers
+from haystack.nodes import FARMReader, TransformersReader
+```
+
+## Document Store
+
+Haystack finds answers to queries within the documents stored in a `DocumentStore`. The current implementations of `DocumentStore` include `ElasticsearchDocumentStore`, `FAISSDocumentStore`, `SQLDocumentStore`, and `InMemoryDocumentStore`.
+
+**Here:** We recommended Elasticsearch as it comes preloaded with features like [full-text queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html), [BM25 retrieval](https://www.elastic.co/elasticon/conf/2016/sf/improved-text-scoring-with-bm25), and [vector storage for text embeddings](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/dense-vector.html).
+
+**Alternatives:** If you are unable to setup an Elasticsearch instance, then follow the [Tutorial 3](https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.ipynb) for using SQL/InMemory document stores.
+
+**Hint**: This tutorial creates a new document store instance with Wikipedia articles on Game of Thrones. However, you can configure Haystack to work with your existing document stores.
+
+### Start an Elasticsearch server
+You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (e.g. in Colab notebooks), then you can manually download and execute Elasticsearch from source.
+
+
+```python
+# Recommended: Start Elasticsearch using Docker via the Haystack utility function
+from haystack.utils import launch_es
+
+launch_es()
+```
+
+
+```python
+# In Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+
+```python
+# Connect to Elasticsearch
+
+from haystack.document_stores import ElasticsearchDocumentStore
+
+document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")
+```
+
+## Preprocessing of documents
+
+Haystack provides a customizable pipeline for:
+ - converting files into texts
+ - cleaning texts
+ - splitting texts
+ - writing them to a Document Store
+
+In this tutorial, we download Wikipedia articles about Game of Thrones, apply a basic cleaning function, and index them in Elasticsearch.
+
+
+```python
+# Let's first fetch some documents that we want to query
+# Here: 517 Wikipedia articles for Game of Thrones
+doc_dir = "data/tutorial1"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt1.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Convert files to dicts
+# You can optionally supply a cleaning function that is applied to each doc (e.g. to remove footers)
+# It must take a str as input, and return a str.
+dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
+
+# We now have a list of dictionaries that we can write to our document store.
+# If your texts come from a different source (e.g. a DB), you can of course skip convert_files_to_dicts() and create the dictionaries yourself.
+# The default format here is:
+# {
+# 'content': "
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install -q git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,faiss]
+```
+
+
+```python
+from haystack.utils import convert_files_to_dicts, fetch_archive_from_http, clean_wiki_text
+from haystack.nodes import Seq2SeqGenerator
+```
+
+### Document Store
+
+FAISS is a library for efficient similarity search on a cluster of dense vectors.
+The `FAISSDocumentStore` uses a SQL(SQLite in-memory be default) database under-the-hood
+to store the document text and other meta data. The vector embeddings of the text are
+indexed on a FAISS Index that later is queried for searching answers.
+The default flavour of FAISSDocumentStore is "Flat" but can also be set to "HNSW" for
+faster search at the expense of some accuracy. Just set the faiss_index_factor_str argument in the constructor.
+For more info on which suits your use case: https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
+
+
+```python
+from haystack.document_stores import FAISSDocumentStore
+
+document_store = FAISSDocumentStore(embedding_dim=128, faiss_index_factory_str="Flat")
+```
+
+### Cleaning & indexing documents
+
+Similarly to the previous tutorials, we download, convert and index some Game of Thrones articles to our DocumentStore
+
+
+```python
+# Let's first get some files that we want to use
+doc_dir = "data/tutorial12"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt12.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Convert files to dicts
+dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
+
+# Now, let's write the dicts containing documents to our DB.
+document_store.write_documents(dicts)
+```
+
+### Initalize Retriever and Reader/Generator
+
+#### Retriever
+
+We use a `DensePassageRetriever` and we invoke `update_embeddings` to index the embeddings of documents in the `FAISSDocumentStore`
+
+
+
+
+```python
+from haystack.nodes import DensePassageRetriever
+
+retriever = DensePassageRetriever(
+ document_store=document_store,
+ query_embedding_model="vblagoje/dpr-question_encoder-single-lfqa-wiki",
+ passage_embedding_model="vblagoje/dpr-ctx_encoder-single-lfqa-wiki",
+)
+
+document_store.update_embeddings(retriever)
+```
+
+Before we blindly use the `DensePassageRetriever` let's empirically test it to make sure a simple search indeed finds the relevant documents.
+
+
+```python
+from haystack.utils import print_documents
+from haystack.pipelines import DocumentSearchPipeline
+
+p_retrieval = DocumentSearchPipeline(retriever)
+res = p_retrieval.run(query="Tell me something about Arya Stark?", params={"Retriever": {"top_k": 10}})
+print_documents(res, max_text_len=512)
+```
+
+#### Reader/Generator
+
+Similar to previous Tutorials we now initalize our reader/generator.
+
+Here we use a `Seq2SeqGenerator` with the *vblagoje/bart_lfqa* model (see: https://huggingface.co/vblagoje/bart_lfqa)
+
+
+
+
+```python
+generator = Seq2SeqGenerator(model_name_or_path="vblagoje/bart_lfqa")
+```
+
+### Pipeline
+
+With a Haystack `Pipeline` you can stick together your building blocks to a search pipeline.
+Under the hood, `Pipelines` are Directed Acyclic Graphs (DAGs) that you can easily customize for your own use cases.
+To speed things up, Haystack also comes with a few predefined Pipelines. One of them is the `GenerativeQAPipeline` that combines a retriever and a reader/generator to answer our questions.
+You can learn more about `Pipelines` in the [docs](https://haystack.deepset.ai/docs/latest/pipelinesmd).
+
+
+```python
+from haystack.pipelines import GenerativeQAPipeline
+
+pipe = GenerativeQAPipeline(generator, retriever)
+```
+
+## Voilà! Ask a question!
+
+
+```python
+pipe.run(
+ query="How did Arya Stark's character get portrayed in a television adaptation?", params={"Retriever": {"top_k": 3}}
+)
+```
+
+
+```python
+pipe.run(query="Why is Arya Stark an unusual character?", params={"Retriever": {"top_k": 3}})
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/13.md b/docs/v1.3.0/_src/tutorials/tutorials/13.md
new file mode 100644
index 0000000000..2fc143b1bb
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/13.md
@@ -0,0 +1,187 @@
+
+
+# Question Generation
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial13_Question_generation.ipynb)
+
+This is a bare bones tutorial showing what is possible with the QuestionGenerator Nodes and Pipelines which automatically
+generate questions which the question generation model thinks can be answered by a given document.
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+
+```python
+# Imports needed to run this notebook
+
+from pprint import pprint
+from tqdm import tqdm
+from haystack.nodes import QuestionGenerator, ElasticsearchRetriever, FARMReader
+from haystack.document_stores import ElasticsearchDocumentStore
+from haystack.pipelines import (
+ QuestionGenerationPipeline,
+ RetrieverQuestionGenerationPipeline,
+ QuestionAnswerGenerationPipeline,
+)
+from haystack.utils import launch_es, print_questions
+```
+
+Let's start an Elasticsearch instance with one of the options below:
+
+
+```python
+# Option 1: Start Elasticsearch service via Docker
+launch_es()
+```
+
+
+```python
+# Option 2: In Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+Let's initialize some core components
+
+
+```python
+text1 = "Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python's design philosophy emphasizes code readability with its notable use of significant whitespace."
+text2 = "Princess Arya Stark is the third child and second daughter of Lord Eddard Stark and his wife, Lady Catelyn Stark. She is the sister of the incumbent Westerosi monarchs, Sansa, Queen in the North, and Brandon, King of the Andals and the First Men. After narrowly escaping the persecution of House Stark by House Lannister, Arya is trained as a Faceless Man at the House of Black and White in Braavos, using her abilities to avenge her family. Upon her return to Westeros, she exacts retribution for the Red Wedding by exterminating the Frey male line."
+text3 = "Dry Cleaning are an English post-punk band who formed in South London in 2018.[3] The band is composed of vocalist Florence Shaw, guitarist Tom Dowse, bassist Lewis Maynard and drummer Nick Buxton. They are noted for their use of spoken word primarily in lieu of sung vocals, as well as their unconventional lyrics. Their musical stylings have been compared to Wire, Magazine and Joy Division.[4] The band released their debut single, 'Magic of Meghan' in 2019. Shaw wrote the song after going through a break-up and moving out of her former partner's apartment the same day that Meghan Markle and Prince Harry announced they were engaged.[5] This was followed by the release of two EPs that year: Sweet Princess in August and Boundary Road Snacks and Drinks in October. The band were included as part of the NME 100 of 2020,[6] as well as DIY magazine's Class of 2020.[7] The band signed to 4AD in late 2020 and shared a new single, 'Scratchcard Lanyard'.[8] In February 2021, the band shared details of their debut studio album, New Long Leg. They also shared the single 'Strong Feelings'.[9] The album, which was produced by John Parish, was released on 2 April 2021.[10]"
+
+docs = [{"content": text1}, {"content": text2}, {"content": text3}]
+
+# Initialize document store and write in the documents
+document_store = ElasticsearchDocumentStore()
+document_store.write_documents(docs)
+
+# Initialize Question Generator
+question_generator = QuestionGenerator()
+```
+
+## Question Generation Pipeline
+
+The most basic version of a question generator pipeline takes a document as input and outputs generated questions
+which the the document can answer.
+
+
+```python
+question_generation_pipeline = QuestionGenerationPipeline(question_generator)
+for idx, document in enumerate(document_store):
+
+ print(f"\n * Generating questions for document {idx}: {document.content[:100]}...\n")
+ result = question_generation_pipeline.run(documents=[document])
+ print_questions(result)
+```
+
+## Retriever Question Generation Pipeline
+
+This pipeline takes a query as input. It retrieves relevant documents and then generates questions based on these.
+
+
+```python
+retriever = ElasticsearchRetriever(document_store=document_store)
+rqg_pipeline = RetrieverQuestionGenerationPipeline(retriever, question_generator)
+
+print(f"\n * Generating questions for documents matching the query 'Arya Stark'\n")
+result = rqg_pipeline.run(query="Arya Stark")
+print_questions(result)
+```
+
+## Question Answer Generation Pipeline
+
+This pipeline takes a document as input, generates questions on it, and attempts to answer these questions using
+a Reader model
+
+
+```python
+reader = FARMReader("deepset/roberta-base-squad2")
+qag_pipeline = QuestionAnswerGenerationPipeline(question_generator, reader)
+for idx, document in enumerate(tqdm(document_store)):
+
+ print(f"\n * Generating questions and answers for document {idx}: {document.content[:100]}...\n")
+ result = qag_pipeline.run(documents=[document])
+ print_questions(result)
+```
+
+## Translated Question Answer Generation Pipeline
+Trained models for Question Answer Generation are not available in many languages other than English. Haystack
+provides a workaround for that issue by machine-translating a pipeline's inputs and outputs with the
+TranslationWrapperPipeline. The following example generates German questions and answers on a German text
+document - by using an English model for Question Answer Generation.
+
+
+```python
+# Fill the document store with a German document.
+text1 = "Python ist eine interpretierte Hochsprachenprogrammiersprache für allgemeine Zwecke. Sie wurde von Guido van Rossum entwickelt und 1991 erstmals veröffentlicht. Die Design-Philosophie von Python legt den Schwerpunkt auf die Lesbarkeit des Codes und die Verwendung von viel Leerraum (Whitespace)."
+docs = [{"content": text1}]
+document_store.delete_documents()
+document_store.write_documents(docs)
+
+# Load machine translation models
+from haystack.nodes import TransformersTranslator
+
+in_translator = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-de-en")
+out_translator = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-en-de")
+
+# Wrap the previously defined QuestionAnswerGenerationPipeline
+from haystack.pipelines import TranslationWrapperPipeline
+
+pipeline_with_translation = TranslationWrapperPipeline(
+ input_translator=in_translator, output_translator=out_translator, pipeline=qag_pipeline
+)
+
+for idx, document in enumerate(tqdm(document_store)):
+ print(f"\n * Generating questions and answers for document {idx}: {document.content[:100]}...\n")
+ result = pipeline_with_translation.run(documents=[document])
+ print_questions(result)
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/14.md b/docs/v1.3.0/_src/tutorials/tutorials/14.md
new file mode 100644
index 0000000000..4182aada4a
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/14.md
@@ -0,0 +1,362 @@
+
+
+# Query Classifier Tutorial
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial14_Query_Classifier.ipynb)
+
+In this tutorial we introduce the query classifier the goal of introducing this feature was to optimize the overall flow of Haystack pipeline by detecting the nature of user queries. Now, the Haystack can detect primarily three types of queries using both light-weight SKLearn Gradient Boosted classifier or Transformer based more robust classifier. The three categories of queries are as follows:
+
+
+### 1. Keyword Queries:
+Such queries don't have semantic meaning and merely consist of keywords. For instance these three are the examples of keyword queries.
+
+* arya stark father
+* jon snow country
+* arya stark younger brothers
+
+### 2. Interrogative Queries:
+In such queries users usually ask a question, regardless of presence of "?" in the query the goal here is to detect the intent of the user whether any question is asked or not in the query. For example:
+
+* who is the father of arya stark ?
+* which country was jon snow filmed ?
+* who are the younger brothers of arya stark ?
+
+### 3. Declarative Queries:
+Such queries are variation of keyword queries, however, there is semantic relationship between words. Fo example:
+
+* Arya stark was a daughter of a lord.
+* Jon snow was filmed in a country in UK.
+* Bran was brother of a princess.
+
+In this tutorial, you will learn how the `TransformersQueryClassifier` and `SklearnQueryClassifier` classes can be used to intelligently route your queries, based on the nature of the user query. Also, you can choose between a lightweight Gradients boosted classifier or a transformer based classifier.
+
+Furthermore, there are two types of classifiers you can use out of the box from Haystack.
+1. Keyword vs Statement/Question Query Classifier
+2. Statement vs Question Query Classifier
+
+As evident from the name the first classifier detects the keywords search queries and semantic statements like sentences/questions. The second classifier differentiates between question based queries and declarative sentences.
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+These lines are to install Haystack through pip
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+
+# Install pygraphviz
+!apt install libgraphviz-dev
+!pip install pygraphviz
+
+# In Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+If running from Colab or a no Docker environment, you will want to start Elasticsearch from source
+
+## Initialization
+
+Here are some core imports
+
+Then let's fetch some data (in this case, pages from the Game of Thrones wiki) and prepare it so that it can
+be used indexed into our `DocumentStore`
+
+
+```python
+from haystack.utils import print_answers, fetch_archive_from_http, convert_files_to_dicts, clean_wiki_text, launch_es
+from haystack.pipelines import Pipeline, RootNode
+from haystack.document_stores import ElasticsearchDocumentStore
+from haystack.nodes import (
+ ElasticsearchRetriever,
+ DensePassageRetriever,
+ FARMReader,
+ TransformersQueryClassifier,
+ SklearnQueryClassifier,
+)
+
+# Download and prepare data - 517 Wikipedia articles for Game of Thrones
+doc_dir = "data/tutorial14"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt14.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# convert files to dicts containing documents that can be indexed to our datastore
+got_dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
+
+# Initialize DocumentStore and index documents
+launch_es()
+document_store = ElasticsearchDocumentStore()
+document_store.delete_documents()
+document_store.write_documents(got_dicts)
+
+# Initialize Sparse retriever
+es_retriever = ElasticsearchRetriever(document_store=document_store)
+
+# Initialize dense retriever
+dpr_retriever = DensePassageRetriever(document_store)
+document_store.update_embeddings(dpr_retriever, update_existing_embeddings=False)
+
+reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
+```
+
+## Keyword vs Question/Statement Classifier
+
+The keyword vs question/statement query classifier essentially distinguishes between the keyword queries and statements/questions. So you can intelligently route to different retrieval nodes based on the nature of the query. Using this classifier can potentially yield the following benefits:
+
+* Getting better search results (e.g. by routing only proper questions to DPR / QA branches and not keyword queries)
+* Less GPU costs (e.g. if 50% of your traffic is only keyword queries you could just use elastic here and save the GPU resources for the other 50% of traffic with semantic queries)
+
+
+
+
+Below, we define a `SklQueryClassifier` and show how to use it:
+
+Read more about the trained model and dataset used [here](https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier/readme.txt)
+
+
+```python
+# Here we build the pipeline
+sklearn_keyword_classifier = Pipeline()
+sklearn_keyword_classifier.add_node(component=SklearnQueryClassifier(), name="QueryClassifier", inputs=["Query"])
+sklearn_keyword_classifier.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["QueryClassifier.output_1"])
+sklearn_keyword_classifier.add_node(component=es_retriever, name="ESRetriever", inputs=["QueryClassifier.output_2"])
+sklearn_keyword_classifier.add_node(component=reader, name="QAReader", inputs=["ESRetriever", "DPRRetriever"])
+sklearn_keyword_classifier.draw("pipeline_classifier.png")
+```
+
+
+```python
+# Run only the dense retriever on the full sentence query
+res_1 = sklearn_keyword_classifier.run(query="Who is the father of Arya Stark?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_1, details="minimum")
+
+# Run only the sparse retriever on a keyword based query
+res_2 = sklearn_keyword_classifier.run(query="arya stark father")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_2, details="minimum")
+```
+
+
+```python
+# Run only the dense retriever on the full sentence query
+res_3 = sklearn_keyword_classifier.run(query="which country was jon snow filmed ?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_3, details="minimum")
+
+# Run only the sparse retriever on a keyword based query
+res_4 = sklearn_keyword_classifier.run(query="jon snow country")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_4, details="minimum")
+```
+
+
+```python
+# Run only the dense retriever on the full sentence query
+res_5 = sklearn_keyword_classifier.run(query="who are the younger brothers of arya stark ?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_5, details="minimum")
+
+# Run only the sparse retriever on a keyword based query
+res_6 = sklearn_keyword_classifier.run(query="arya stark younger brothers")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_6, details="minimum")
+```
+
+## Transformer Keyword vs Question/Statement Classifier
+
+Firstly, it's essential to understand the trade-offs between SkLearn and Transformer query classifiers. The transformer classifier is more accurate than SkLearn classifier however, it requires more memory and most probably GPU for faster inference however the transformer size is roughly `50 MBs`. Whereas, SkLearn is less accurate however is much more faster and doesn't require GPU for inference.
+
+Below, we define a `TransformersQueryClassifier` and show how to use it:
+
+Read more about the trained model and dataset used [here](https://huggingface.co/shahrukhx01/bert-mini-finetune-question-detection)
+
+
+```python
+# Here we build the pipeline
+transformer_keyword_classifier = Pipeline()
+transformer_keyword_classifier.add_node(
+ component=TransformersQueryClassifier(), name="QueryClassifier", inputs=["Query"]
+)
+transformer_keyword_classifier.add_node(
+ component=dpr_retriever, name="DPRRetriever", inputs=["QueryClassifier.output_1"]
+)
+transformer_keyword_classifier.add_node(component=es_retriever, name="ESRetriever", inputs=["QueryClassifier.output_2"])
+transformer_keyword_classifier.add_node(component=reader, name="QAReader", inputs=["ESRetriever", "DPRRetriever"])
+transformer_keyword_classifier.draw("pipeline_classifier.png")
+```
+
+
+```python
+# Run only the dense retriever on the full sentence query
+res_1 = transformer_keyword_classifier.run(query="Who is the father of Arya Stark?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_1, details="minimum")
+
+# Run only the sparse retriever on a keyword based query
+res_2 = transformer_keyword_classifier.run(query="arya stark father")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_2, details="minimum")
+```
+
+
+```python
+# Run only the dense retriever on the full sentence query
+res_3 = transformer_keyword_classifier.run(query="which country was jon snow filmed ?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_3, details="minimum")
+
+# Run only the sparse retriever on a keyword based query
+res_4 = transformer_keyword_classifier.run(query="jon snow country")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_4, details="minimum")
+```
+
+
+```python
+# Run only the dense retriever on the full sentence query
+res_5 = transformer_keyword_classifier.run(query="who are the younger brothers of arya stark ?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_5, details="minimum")
+
+# Run only the sparse retriever on a keyword based query
+res_6 = transformer_keyword_classifier.run(query="arya stark younger brothers")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_6, details="minimum")
+```
+
+## Question vs Statement Classifier
+
+One possible use case of this classifier could be to route queries after the document retrieval to only send questions to QA reader and in case of declarative sentence, just return the DPR/ES results back to user to enhance user experience and only show answers when user explicitly asks it.
+
+
+
+
+Below, we define a `TransformersQueryClassifier` and show how to use it:
+
+Read more about the trained model and dataset used [here](https://huggingface.co/shahrukhx01/question-vs-statement-classifier)
+
+
+```python
+# Here we build the pipeline
+transformer_question_classifier = Pipeline()
+transformer_question_classifier.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["Query"])
+transformer_question_classifier.add_node(
+ component=TransformersQueryClassifier(model_name_or_path="shahrukhx01/question-vs-statement-classifier"),
+ name="QueryClassifier",
+ inputs=["DPRRetriever"],
+)
+transformer_question_classifier.add_node(component=reader, name="QAReader", inputs=["QueryClassifier.output_1"])
+transformer_question_classifier.draw("question_classifier.png")
+
+# Run only the QA reader on the question query
+res_1 = transformer_question_classifier.run(query="Who is the father of Arya Stark?")
+print("DPR Results" + "\n" + "=" * 15)
+print_answers(res_1, details="minimum")
+
+# Show only DPR results
+res_2 = transformer_question_classifier.run(query="Arya Stark was the daughter of a Lord.")
+print("ES Results" + "\n" + "=" * 15)
+print_answers(res_2, details="minimum")
+```
+
+## Standalone Query Classifier
+Below we run queries classifiers standalone to better understand their outputs on each of the three types of queries
+
+
+```python
+# Here we create the keyword vs question/statement query classifier
+from haystack.pipelines import TransformersQueryClassifier
+
+queries = [
+ "arya stark father",
+ "jon snow country",
+ "who is the father of arya stark",
+ "which country was jon snow filmed?",
+]
+
+keyword_classifier = TransformersQueryClassifier()
+
+for query in queries:
+ result = keyword_classifier.run(query=query)
+ if result[1] == "output_1":
+ category = "question/statement"
+ else:
+ category = "keyword"
+
+ print(f"Query: {query}, raw_output: {result}, class: {category}")
+```
+
+
+```python
+# Here we create the question vs statement query classifier
+from haystack.pipelines import TransformersQueryClassifier
+
+queries = [
+ "Lord Eddard was the father of Arya Stark.",
+ "Jon Snow was filmed in United Kingdom.",
+ "who is the father of arya stark?",
+ "Which country was jon snow filmed in?",
+]
+
+question_classifier = TransformersQueryClassifier(model_name_or_path="shahrukhx01/question-vs-statement-classifier")
+
+for query in queries:
+ result = question_classifier.run(query=query)
+ if result[1] == "output_1":
+ category = "question"
+ else:
+ category = "statement"
+
+ print(f"Query: {query}, raw_output: {result}, class: {category}")
+```
+
+## Conclusion
+
+The query classifier gives you more possibility to be more creative with the pipelines and use different retrieval nodes in a flexible fashion. Moreover, as in the case of Question vs Statement classifier you can also choose the queries which you want to send to the reader.
+
+Finally, you also have the possible of bringing your own classifier and plugging it into either `TransformersQueryClassifier(model_name_or_path="
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+
+# The TaPAs-based TableReader requires the torch-scatter library
+!pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
+
+# Install pygraphviz for visualization of Pipelines
+!apt install libgraphviz-dev
+!pip install pygraphviz
+```
+
+### Start an Elasticsearch server
+You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (e.g. in Colab notebooks), then you can manually download and execute Elasticsearch from source.
+
+
+```python
+# Recommended: Start Elasticsearch using Docker via the Haystack utility function
+from haystack.utils import launch_es
+
+launch_es()
+```
+
+
+```python
+# In Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+
+```python
+# Connect to Elasticsearch
+from haystack.document_stores import ElasticsearchDocumentStore
+
+# We want to use a small model producing 512-dimensional embeddings, so we need to set embedding_dim to 512
+document_index = "document"
+document_store = ElasticsearchDocumentStore(
+ host="localhost", username="", password="", index=document_index, embedding_dim=512
+)
+```
+
+## Add Tables to DocumentStore
+To quickly demonstrate the capabilities of the `TableTextRetriever` and the `TableReader` we use a subset of 1000 tables of the [Open Table-and-Text Question Answering (OTT-QA) dataset](https://github.com/wenhuchen/OTT-QA).
+
+Just as text passages, tables are represented as `Document` objects in Haystack. The content field, though, is a pandas DataFrame instead of a string.
+
+
+```python
+# Let's first fetch some tables that we want to query
+# Here: 1000 tables from OTT-QA
+from haystack.utils import fetch_archive_from_http
+
+doc_dir = "data/tutorial15"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/ottqa_sample.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+```
+
+
+```python
+# Add the tables to the DocumentStore
+
+import json
+from haystack import Document
+import pandas as pd
+
+
+def read_ottqa_tables(filename):
+ processed_tables = []
+ with open(filename) as tables:
+ tables = json.load(tables)
+ for key, table in tables.items():
+ current_columns = table["header"]
+ current_rows = table["data"]
+ current_df = pd.DataFrame(columns=current_columns, data=current_rows)
+ current_doc_title = table["title"]
+ current_section_title = table["section_title"]
+ document = Document(
+ content=current_df,
+ content_type="table",
+ meta={"title": current_doc_title, "section_title": current_section_title},
+ id=key,
+ )
+ processed_tables.append(document)
+
+ return processed_tables
+
+
+tables = read_ottqa_tables(f"{doc_dir}/ottqa_tables_sample.json")
+document_store.write_documents(tables, index=document_index)
+
+# Showing content field and meta field of one of the Documents of content_type 'table'
+print(tables[0].content)
+print(tables[0].meta)
+```
+
+## Initalize Retriever, Reader, & Pipeline
+
+### Retriever
+
+Retrievers help narrowing down the scope for the Reader to a subset of tables where a given question could be answered.
+They use some simple but fast algorithm.
+
+**Here:** We use the `TableTextRetriever` capable of retrieving relevant content among a database
+of texts and tables using dense embeddings. It is an extension of the `DensePassageRetriever` and consists of three encoders (one query encoder, one text passage encoder and one table encoder) that create embeddings in the same vector space. More details on the `TableTextRetriever` and how it is trained can be found in [this paper](https://arxiv.org/abs/2108.04049).
+
+**Alternatives:**
+
+- `ElasticsearchRetriever` that uses BM25 algorithm
+
+
+
+```python
+from haystack.nodes.retriever import TableTextRetriever
+
+retriever = TableTextRetriever(
+ document_store=document_store,
+ query_embedding_model="deepset/bert-small-mm_retrieval-question_encoder",
+ passage_embedding_model="deepset/bert-small-mm_retrieval-passage_encoder",
+ table_embedding_model="deepset/bert-small-mm_retrieval-table_encoder",
+ embed_meta_fields=["title", "section_title"],
+)
+```
+
+
+```python
+# Add table embeddings to the tables in DocumentStore
+document_store.update_embeddings(retriever=retriever)
+```
+
+
+```python
+## Alternative: ElasticsearchRetriever
+# from haystack.nodes.retriever import ElasticsearchRetriever
+# retriever = ElasticsearchRetriever(document_store=document_store)
+```
+
+
+```python
+# Try the Retriever
+from haystack.utils import print_documents
+
+retrieved_tables = retriever.retrieve("How many twin buildings are under construction?", top_k=5)
+# Get highest scored table
+print(retrieved_tables[0].content)
+```
+
+### Reader
+The `TableReader` is based on TaPas, a transformer-based language model capable of grasping the two-dimensional structure of a table. It scans the tables returned by the retriever and extracts the anser. The available TableReader models can be found [here](https://huggingface.co/models?pipeline_tag=table-question-answering&sort=downloads).
+
+**Notice**: The `TableReader` will return an answer for each table, even if the query cannot be answered by the table. Furthermore, the confidence scores are not useful as of now, given that they will *always* be very high (i.e. 1 or close to 1).
+
+
+```python
+from haystack.nodes import TableReader
+
+reader = TableReader(model_name_or_path="google/tapas-base-finetuned-wtq", max_seq_len=512)
+```
+
+
+```python
+# Try the TableReader on one Table (highest-scored retrieved table from previous section)
+
+table_doc = document_store.get_document_by_id("List_of_tallest_twin_buildings_and_structures_in_the_world_1")
+print(table_doc.content)
+```
+
+
+```python
+from haystack.utils import print_answers
+
+prediction = reader.predict(query="How many twin buildings are under construction?", documents=[table_doc])
+print_answers(prediction, details="all")
+```
+
+The offsets in the `offsets_in_document` and `offsets_in_context` field indicate the table cells that the model predicts to be part of the answer. They need to be interpreted on the linearized table, i.e., a flat list containing all of the table cells.
+
+In the `Answer`'s meta field, you can find the aggreagtion operator used to construct the answer (in this case `COUNT`) and the answer cells as strings.
+
+
+```python
+print(f"Predicted answer: {prediction['answers'][0].answer}")
+print(f"Meta field: {prediction['answers'][0].meta}")
+```
+
+### Pipeline
+The Retriever and the Reader can be sticked together to a pipeline in order to first retrieve relevant tables and then extract the answer.
+
+**Notice**: Given that the `TableReader` does not provide useful confidence scores and returns an answer for each of the tables, the sorting of the answers might be not helpful.
+
+
+```python
+# Initialize pipeline
+from haystack import Pipeline
+
+table_qa_pipeline = Pipeline()
+table_qa_pipeline.add_node(component=retriever, name="TableTextRetriever", inputs=["Query"])
+table_qa_pipeline.add_node(component=reader, name="TableReader", inputs=["TableTextRetriever"])
+```
+
+
+```python
+prediction = table_qa_pipeline.run("How many twin buildings are under construction?")
+print_answers(prediction, details="minimum")
+```
+
+# Open-Domain QA on Text and Tables
+With haystack, you not only have the possibility to do QA on texts or tables, solely, but you can also use both texts and tables as your source of information.
+
+To demonstrate this, we add 1,000 sample text passages from the OTT-QA dataset.
+
+
+```python
+# Add 1,000 text passages from OTT-QA to our document store.
+
+
+def read_ottqa_texts(filename):
+ processed_passages = []
+ with open(filename) as passages:
+ passages = json.load(passages)
+ for title, content in passages.items():
+ title = title[6:]
+ title = title.replace("_", " ")
+ document = Document(content=content, content_type="text", meta={"title": title})
+ processed_passages.append(document)
+
+ return processed_passages
+
+
+passages = read_ottqa_texts(f"{doc_dir}/ottqa_texts_sample.json")
+document_store.write_documents(passages, index=document_index)
+```
+
+
+```python
+document_store.update_embeddings(retriever=retriever, update_existing_embeddings=False)
+```
+
+## Pipeline for QA on Combination of Text and Tables
+We are using one node for retrieving both texts and tables, the `TableTextRetriever`. In order to do question-answering on the Documents coming from the `TableTextRetriever`, we need to route Documents of type `"text"` to a `FARMReader` (or alternatively `TransformersReader`) and Documents of type `"table"` to a `TableReader`.
+
+To achieve this, we make use of two additional nodes:
+- `SplitDocumentList`: Splits the List of Documents retrieved by the `TableTextRetriever` into two lists containing only Documents of type `"text"` or `"table"`, respectively.
+- `JoinAnswers`: Takes Answers coming from two different Readers (in this case `FARMReader` and `TableReader`) and joins them to a single list of Answers.
+
+
+```python
+from haystack.nodes import FARMReader, RouteDocuments, JoinAnswers
+
+text_reader = FARMReader("deepset/roberta-base-squad2")
+# In order to get meaningful scores from the TableReader, use "deepset/tapas-large-nq-hn-reader" or
+# "deepset/tapas-large-nq-reader" as TableReader models. The disadvantage of these models is, however,
+# that they are not capable of doing aggregations over multiple table cells.
+table_reader = TableReader("deepset/tapas-large-nq-hn-reader")
+route_documents = RouteDocuments()
+join_answers = JoinAnswers()
+```
+
+
+```python
+text_table_qa_pipeline = Pipeline()
+text_table_qa_pipeline.add_node(component=retriever, name="TableTextRetriever", inputs=["Query"])
+text_table_qa_pipeline.add_node(component=route_documents, name="RouteDocuments", inputs=["TableTextRetriever"])
+text_table_qa_pipeline.add_node(component=text_reader, name="TextReader", inputs=["RouteDocuments.output_1"])
+text_table_qa_pipeline.add_node(component=table_reader, name="TableReader", inputs=["RouteDocuments.output_2"])
+text_table_qa_pipeline.add_node(component=join_answers, name="JoinAnswers", inputs=["TextReader", "TableReader"])
+```
+
+
+```python
+# Let's have a look on the structure of the combined Table an Text QA pipeline.
+from IPython import display
+
+text_table_qa_pipeline.draw()
+display.Image("pipeline.png")
+```
+
+
+```python
+# Example query whose answer resides in a text passage
+predictions = text_table_qa_pipeline.run(query="Who is Aleksandar Trifunovic?")
+```
+
+
+```python
+# We can see both text passages and tables as contexts of the predicted answers.
+print_answers(predictions, details="minimum")
+```
+
+
+```python
+# Example query whose answer resides in a table
+predictions = text_table_qa_pipeline.run(query="What is Cuba's national tree?")
+```
+
+
+```python
+# We can see both text passages and tables as contexts of the predicted answers.
+print_answers(predictions, details="minimum")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
+
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/16.md b/docs/v1.3.0/_src/tutorials/tutorials/16.md
new file mode 100644
index 0000000000..c622091ae7
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/16.md
@@ -0,0 +1,266 @@
+
+
+# Extending your Metadata using DocumentClassifiers at Index Time
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial16_Document_Classifier_at_Index_Time.ipynb)
+
+With DocumentClassifier it's possible to automatically enrich your documents with categories, sentiments, topics or whatever metadata you like. This metadata could be used for efficient filtering or further processing. Say you have some categories your users typically filter on. If the documents are tagged manually with these categories, you could automate this process by training a model. Or you can leverage the full power and flexibility of zero shot classification. All you need to do is pass your categories to the classifier, no labels required. This tutorial shows how to integrate it in your indexing pipeline.
+
+DocumentClassifier adds the classification result (label and score) to Document's meta property.
+Hence, we can use it to classify documents at index time. \
+The result can be accessed at query time: for example by applying a filter for "classification.label".
+
+This tutorial will show you how to integrate a classification model into your preprocessing steps and how you can filter for this additional metadata at query time. In the last section we show how to put it all together and create an indexing pipeline.
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab, ocr]
+
+!wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.03.tar.gz
+!tar -xvf xpdf-tools-linux-4.03.tar.gz && sudo cp xpdf-tools-linux-4.03/bin64/pdftotext /usr/local/bin
+
+# Install pygraphviz
+!apt install libgraphviz-dev
+!pip install pygraphviz
+```
+
+
+```python
+# Here are the imports we need
+from haystack.document_stores.elasticsearch import ElasticsearchDocumentStore
+from haystack.nodes import PreProcessor, TransformersDocumentClassifier, FARMReader, ElasticsearchRetriever
+from haystack.schema import Document
+from haystack.utils import convert_files_to_dicts, fetch_archive_from_http, print_answers
+```
+
+
+```python
+# This fetches some sample files to work with
+
+doc_dir = "data/tutorial16"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/preprocessing_tutorial16.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+```
+
+## Read and preprocess documents
+
+
+
+```python
+# note that you can also use the document classifier before applying the PreProcessor, e.g. before splitting your documents
+
+all_docs = convert_files_to_dicts(dir_path=doc_dir)
+preprocessor_sliding_window = PreProcessor(split_overlap=3, split_length=10, split_respect_sentence_boundary=False)
+docs_sliding_window = preprocessor_sliding_window.process(all_docs)
+```
+
+## Apply DocumentClassifier
+
+We can enrich the document metadata at index time using any transformers document classifier model. While traditional classification models are trained to predict one of a few "hard-coded" classes and required a dedicated training dataset, zero-shot classification is super flexible and you can easily switch the classes the model should predict on the fly. Just supply them via the labels param.
+Here we use a zero shot model that is supposed to classify our documents in 'music', 'natural language processing' and 'history'. Feel free to change them for whatever you like to classify. \
+These classes can later on be accessed at query time.
+
+
+```python
+doc_classifier = TransformersDocumentClassifier(
+ model_name_or_path="cross-encoder/nli-distilroberta-base",
+ task="zero-shot-classification",
+ labels=["music", "natural language processing", "history"],
+ batch_size=16,
+)
+```
+
+
+```python
+# we can also use any other transformers model besides zero shot classification
+
+# doc_classifier_model = 'bhadresh-savani/distilbert-base-uncased-emotion'
+# doc_classifier = TransformersDocumentClassifier(model_name_or_path=doc_classifier_model, batch_size=16, use_gpu=-1)
+```
+
+
+```python
+# we could also specifiy a different field we want to run the classification on
+
+# doc_classifier = TransformersDocumentClassifier(model_name_or_path="cross-encoder/nli-distilroberta-base",
+# task="zero-shot-classification",
+# labels=["music", "natural language processing", "history"],
+# batch_size=16, use_gpu=-1,
+# classification_field="description")
+```
+
+
+```python
+# convert to Document using a fieldmap for custom content fields the classification should run on
+docs_to_classify = [Document.from_dict(d) for d in docs_sliding_window]
+```
+
+
+```python
+# classify using gpu, batch_size makes sure we do not run out of memory
+classified_docs = doc_classifier.predict(docs_to_classify)
+```
+
+
+```python
+# let's see how it looks: there should be a classification result in the meta entry containing labels and scores.
+print(classified_docs[0].to_dict())
+```
+
+## Indexing
+
+
+```python
+# In Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+
+```python
+# Connect to Elasticsearch
+document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")
+```
+
+
+```python
+# Now, let's write the docs to our DB.
+document_store.delete_all_documents()
+document_store.write_documents(classified_docs)
+```
+
+
+```python
+# check if indexed docs contain classification results
+test_doc = document_store.get_all_documents()[0]
+print(
+ f'document {test_doc.id} with content \n\n{test_doc.content}\n\nhas label {test_doc.meta["classification"]["label"]}'
+)
+```
+
+## Querying the data
+
+All we have to do to filter for one of our classes is to set a filter on "classification.label".
+
+
+```python
+# Initialize QA-Pipeline
+from haystack.pipelines import ExtractiveQAPipeline
+
+retriever = ElasticsearchRetriever(document_store=document_store)
+reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)
+pipe = ExtractiveQAPipeline(reader, retriever)
+```
+
+
+```python
+## Voilà! Ask a question while filtering for "music"-only documents
+prediction = pipe.run(
+ query="What is heavy metal?",
+ params={"Retriever": {"top_k": 10, "filters": {"classification.label": ["music"]}}, "Reader": {"top_k": 5}},
+)
+```
+
+
+```python
+print_answers(prediction, details="high")
+```
+
+## Wrapping it up in an indexing pipeline
+
+
+```python
+from pathlib import Path
+from haystack.pipelines import Pipeline
+from haystack.nodes import TextConverter, PreProcessor, FileTypeClassifier, PDFToTextConverter, DocxToTextConverter
+```
+
+
+```python
+file_type_classifier = FileTypeClassifier()
+text_converter = TextConverter()
+pdf_converter = PDFToTextConverter()
+docx_converter = DocxToTextConverter()
+
+indexing_pipeline_with_classification = Pipeline()
+indexing_pipeline_with_classification.add_node(
+ component=file_type_classifier, name="FileTypeClassifier", inputs=["File"]
+)
+indexing_pipeline_with_classification.add_node(
+ component=text_converter, name="TextConverter", inputs=["FileTypeClassifier.output_1"]
+)
+indexing_pipeline_with_classification.add_node(
+ component=pdf_converter, name="PdfConverter", inputs=["FileTypeClassifier.output_2"]
+)
+indexing_pipeline_with_classification.add_node(
+ component=docx_converter, name="DocxConverter", inputs=["FileTypeClassifier.output_4"]
+)
+indexing_pipeline_with_classification.add_node(
+ component=preprocessor_sliding_window,
+ name="Preprocessor",
+ inputs=["TextConverter", "PdfConverter", "DocxConverter"],
+)
+indexing_pipeline_with_classification.add_node(
+ component=doc_classifier, name="DocumentClassifier", inputs=["Preprocessor"]
+)
+indexing_pipeline_with_classification.add_node(
+ component=document_store, name="DocumentStore", inputs=["DocumentClassifier"]
+)
+indexing_pipeline_with_classification.draw("index_time_document_classifier.png")
+
+document_store.delete_documents()
+txt_files = [f for f in Path(doc_dir).iterdir() if f.suffix == ".txt"]
+pdf_files = [f for f in Path(doc_dir).iterdir() if f.suffix == ".pdf"]
+docx_files = [f for f in Path(doc_dir).iterdir() if f.suffix == ".docx"]
+indexing_pipeline_with_classification.run(file_paths=txt_files)
+indexing_pipeline_with_classification.run(file_paths=pdf_files)
+indexing_pipeline_with_classification.run(file_paths=docx_files)
+
+document_store.get_all_documents()[0]
+```
+
+
+```python
+# we can store this pipeline and use it from the REST-API
+indexing_pipeline_with_classification.save_to_yaml("indexing_pipeline_with_classification.yaml")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
+
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/2.md b/docs/v1.3.0/_src/tutorials/tutorials/2.md
new file mode 100644
index 0000000000..bf2314f007
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/2.md
@@ -0,0 +1,157 @@
+
+
+# Fine-tuning a Model on Your Own Data
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial2_Finetune_a_model_on_your_data.ipynb)
+
+For many use cases it is sufficient to just use one of the existing public models that were trained on SQuAD or other public QA datasets (e.g. Natural Questions).
+However, if you have domain-specific questions, fine-tuning your model on custom examples will very likely boost your performance.
+While this varies by domain, we saw that ~ 2000 examples can easily increase performance by +5-20%.
+
+This tutorial shows you how to fine-tune a pretrained model on your own dataset.
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+
+```python
+from haystack.nodes import FARMReader
+from haystack.utils import fetch_archive_from_http
+```
+
+
+## Create Training Data
+
+There are two ways to generate training data
+
+1. **Annotation**: You can use the [annotation tool](https://haystack.deepset.ai/guides/annotation) to label your data, i.e. highlighting answers to your questions in a document. The tool supports structuring your workflow with organizations, projects, and users. The labels can be exported in SQuAD format that is compatible for training with Haystack.
+
+
+
+2. **Feedback**: For production systems, you can collect training data from direct user feedback via Haystack's [REST API interface](https://github.com/deepset-ai/haystack#rest-api). This includes a customizable user feedback API for providing feedback on the answer returned by the API. The API provides a feedback export endpoint to obtain the feedback data for fine-tuning your model further.
+
+
+## Fine-tune your model
+
+Once you have collected training data, you can fine-tune your base models.
+We initialize a reader as a base model and fine-tune it on our own custom dataset (should be in SQuAD-like format).
+We recommend using a base model that was trained on SQuAD or a similar QA dataset before to benefit from Transfer Learning effects.
+
+**Recommendation**: Run training on a GPU.
+If you are using Colab: Enable this in the menu "Runtime" > "Change Runtime type" > Select "GPU" in dropdown.
+Then change the `use_gpu` arguments below to `True`
+
+
+```python
+reader = FARMReader(model_name_or_path="distilbert-base-uncased-distilled-squad", use_gpu=True)
+data_dir = "data/squad20"
+# data_dir = "PATH/TO_YOUR/TRAIN_DATA"
+reader.train(data_dir=data_dir, train_filename="dev-v2.0.json", use_gpu=True, n_epochs=1, save_dir="my_model")
+```
+
+
+```python
+# Saving the model happens automatically at the end of training into the `save_dir` you specified
+# However, you could also save a reader manually again via:
+reader.save(directory="my_model")
+```
+
+
+```python
+# If you want to load it at a later point, just do:
+new_reader = FARMReader(model_name_or_path="my_model")
+```
+
+## Distill your model
+In this case, we have used "distilbert-base-uncased" as our base model. This model was trained using a process called distillation. In this process, a bigger model is trained first and is used to train a smaller model which increases its accuracy. This is why "distilbert-base-uncased" can achieve quite competitive performance while being very small.
+
+Sometimes, however, you can't use an already distilled model and have to distil it yourself. For this case, haystack has implemented [distillation features](https://haystack.deepset.ai/guides/model-distillation).
+
+### Augmenting your training data
+To get the most out of model distillation, we recommend increasing the size of your training data by using data augmentation. You can do this by running the [`augment_squad.py` script](https://github.com/deepset-ai/haystack/blob/master/haystack/utils/augment_squad.py):
+
+
+```python
+# Downloading script
+!wget https://raw.githubusercontent.com/deepset-ai/haystack/master/haystack/utils/augment_squad.py
+
+doc_dir = "data/tutorial2"
+
+# Downloading smaller glove vector file (only for demonstration purposes)
+glove_url = "https://nlp.stanford.edu/data/glove.6B.zip"
+fetch_archive_from_http(url=glove_url, output_dir=doc_dir)
+
+# Downloading very small dataset to make tutorial faster (please use a bigger dataset for real use cases)
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/squad_small.json.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Just replace the path with your dataset and adjust the output (also please remove glove path to use bigger glove vector file)
+!python augment_squad.py --squad_path squad_small.json --output_path augmented_dataset.json --multiplication_factor 2 --glove_path glove.6B.300d.txt
+```
+
+In this case, we use a multiplication factor of 2 to keep this example lightweight. Usually you would use a factor like 20 depending on the size of your training data. Augmenting this small dataset with a multiplication factor of 2, should take about 5 to 10 minutes to run on one V100 GPU.
+
+### Running distillation
+Distillation in haystack is done in two steps: First, you run intermediate layer distillation on the augmented dataset to ensure the two models behave similarly. After that, you run the prediction layer distillation on the non-augmented dataset to optimize the model for your specific task.
+
+If you want, you can leave out the intermediate layer distillation step and only run the prediction layer distillation. This way you also do not need to perform data augmentation. However, this will make the model significantly less accurate.
+
+
+```python
+# Loading a fine-tuned model as teacher e.g. "deepset/bert-base-uncased-squad2"
+teacher = FARMReader(model_name_or_path="my_model", use_gpu=True)
+
+# You can use any pre-trained language model as teacher that uses the same tokenizer as the teacher model.
+# The number of the layers in the teacher model also needs to be a multiple of the number of the layers in the student.
+student = FARMReader(model_name_or_path="huawei-noah/TinyBERT_General_6L_768D", use_gpu=True)
+
+student.distil_intermediate_layers_from(teacher, data_dir=".", train_filename="augmented_dataset.json", use_gpu=True)
+student.distil_prediction_layer_from(teacher, data_dir="data/squad20", train_filename="dev-v2.0.json", use_gpu=True)
+
+student.save(directory="my_distilled_model")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/3.md b/docs/v1.3.0/_src/tutorials/tutorials/3.md
new file mode 100644
index 0000000000..f785e9c776
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/3.md
@@ -0,0 +1,230 @@
+
+
+# Build a QA System Without Elasticsearch
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.ipynb)
+
+Haystack provides alternatives to Elasticsearch for developing quick prototypes.
+
+You can use an `InMemoryDocumentStore` or a `SQLDocumentStore`(with SQLite) as the document store.
+
+If you are interested in more feature-rich Elasticsearch, then please refer to the Tutorial 1.
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+
+```python
+from haystack.utils import clean_wiki_text, convert_files_to_dicts, fetch_archive_from_http, print_answers
+from haystack.nodes import FARMReader, TransformersReader
+```
+
+## Document Store
+
+
+
+```python
+# In-Memory Document Store
+from haystack.document_stores import InMemoryDocumentStore
+
+document_store = InMemoryDocumentStore()
+```
+
+
+```python
+# SQLite Document Store
+# from haystack.document_stores import SQLDocumentStore
+# document_store = SQLDocumentStore(url="sqlite:///qa.db")
+```
+
+## Preprocessing of documents
+
+Haystack provides a customizable pipeline for:
+ - converting files into texts
+ - cleaning texts
+ - splitting texts
+ - writing them to a Document Store
+
+In this tutorial, we download Wikipedia articles on Game of Thrones, apply a basic cleaning function, and index them in Elasticsearch.
+
+
+```python
+# Let's first get some documents that we want to query
+# Here: 517 Wikipedia articles for Game of Thrones
+doc_dir = "data/tutorial3"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt3.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# convert files to dicts containing documents that can be indexed to our datastore
+# You can optionally supply a cleaning function that is applied to each doc (e.g. to remove footers)
+# It must take a str as input, and return a str.
+dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
+
+# We now have a list of dictionaries that we can write to our document store.
+# If your texts come from a different source (e.g. a DB), you can of course skip convert_files_to_dicts() and create the dictionaries yourself.
+# The default format here is: {"name": "
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+
+```python
+from haystack.document_stores import ElasticsearchDocumentStore
+
+from haystack.nodes import EmbeddingRetriever
+import pandas as pd
+import requests
+```
+
+### Start an Elasticsearch server
+You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (eg., in Colab notebooks), then you can manually download and execute Elasticsearch from source.
+
+
+```python
+# Recommended: Start Elasticsearch using Docker via the Haystack utility function
+from haystack.utils import launch_es, fetch_archive_from_http
+
+launch_es()
+```
+
+
+```python
+# In Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+### Init the DocumentStore
+In contrast to Tutorial 1 (extractive QA), we:
+
+* specify the name of our `text_field` in Elasticsearch that we want to return as an answer
+* specify the name of our `embedding_field` in Elasticsearch where we'll store the embedding of our question and that is used later for calculating our similarity to the incoming user question
+* set `excluded_meta_data=["question_emb"]` so that we don't return the huge embedding vectors in our search results
+
+
+```python
+from haystack.document_stores import ElasticsearchDocumentStore
+
+document_store = ElasticsearchDocumentStore(
+ host="localhost",
+ username="",
+ password="",
+ index="document",
+ embedding_field="question_emb",
+ embedding_dim=384,
+ excluded_meta_data=["question_emb"],
+)
+```
+
+### Create a Retriever using embeddings
+Instead of retrieving via Elasticsearch's plain BM25, we want to use vector similarity of the questions (user question vs. FAQ ones).
+We can use the `EmbeddingRetriever` for this purpose and specify a model that we use for the embeddings.
+
+
+```python
+retriever = EmbeddingRetriever(
+ document_store=document_store, embedding_model="sentence-transformers/all-MiniLM-L6-v2", use_gpu=True
+)
+```
+
+### Prepare & Index FAQ data
+We create a pandas dataframe containing some FAQ data (i.e curated pairs of question + answer) and index those in elasticsearch.
+Here: We download some question-answer pairs related to COVID-19
+
+
+```python
+# Download
+doc_dir = "data/tutorial4"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/small_faq_covid.csv.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Get dataframe with columns "question", "answer" and some custom metadata
+df = pd.read_csv(f"{doc_dir}/small_faq_covid.csv")
+# Minimal cleaning
+df.fillna(value="", inplace=True)
+df["question"] = df["question"].apply(lambda x: x.strip())
+print(df.head())
+
+# Get embeddings for our questions from the FAQs
+questions = list(df["question"].values)
+df["question_emb"] = retriever.embed_queries(texts=questions)
+df = df.rename(columns={"question": "content"})
+
+# Convert Dataframe to list of dicts and index them in our DocumentStore
+docs_to_index = df.to_dict(orient="records")
+document_store.write_documents(docs_to_index)
+```
+
+### Ask questions
+Initialize a Pipeline (this time without a reader) and ask questions
+
+
+```python
+from haystack.pipelines import FAQPipeline
+
+pipe = FAQPipeline(retriever=retriever)
+```
+
+
+```python
+from haystack.utils import print_answers
+
+prediction = pipe.run(query="How is the virus spreading?", params={"Retriever": {"top_k": 10}})
+print_answers(prediction, details="medium")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/5.md b/docs/v1.3.0/_src/tutorials/tutorials/5.md
new file mode 100644
index 0000000000..9c038dcf65
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/5.md
@@ -0,0 +1,375 @@
+
+
+# Evaluation of a Pipeline and its Components
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial5_Evaluation.ipynb)
+
+To be able to make a statement about the quality of results a question-answering pipeline or any other pipeline in haystack produces, it is important to evaluate it. Furthermore, evaluation allows determining which components of the pipeline can be improved.
+The results of the evaluation can be saved as CSV files, which contain all the information to calculate additional metrics later on or inspect individual predictions.
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+## Start an Elasticsearch server
+You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (eg., in Colab notebooks), then you can manually download and execute Elasticsearch from source.
+
+
+```python
+# If Docker is available: Start Elasticsearch as docker container
+# from haystack.utils import launch_es
+# launch_es()
+
+# Alternative in Colab / No Docker environments: Start Elasticsearch from source
+! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
+! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
+! chown -R daemon:daemon elasticsearch-7.9.2
+
+import os
+from subprocess import Popen, PIPE, STDOUT
+
+es_server = Popen(
+ ["elasticsearch-7.9.2/bin/elasticsearch"], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1) # as daemon
+)
+# wait until ES has started
+! sleep 30
+```
+
+## Fetch, Store And Preprocess the Evaluation Dataset
+
+
+```python
+from haystack.utils import fetch_archive_from_http
+
+# Download evaluation data, which is a subset of Natural Questions development set containing 50 documents with one question per document and multiple annotated answers
+doc_dir = "data/tutorial5"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/nq_dev_subset_v2.json.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+```
+
+
+```python
+# make sure these indices do not collide with existing ones, the indices will be wiped clean before data is inserted
+doc_index = "tutorial5_docs"
+label_index = "tutorial5_labels"
+```
+
+
+```python
+# Connect to Elasticsearch
+from haystack.document_stores import ElasticsearchDocumentStore
+
+# Connect to Elasticsearch
+document_store = ElasticsearchDocumentStore(
+ host="localhost",
+ username="",
+ password="",
+ index=doc_index,
+ label_index=label_index,
+ embedding_field="emb",
+ embedding_dim=768,
+ excluded_meta_data=["emb"],
+)
+```
+
+
+```python
+from haystack.nodes import PreProcessor
+
+# Add evaluation data to Elasticsearch Document Store
+# We first delete the custom tutorial indices to not have duplicate elements
+# and also split our documents into shorter passages using the PreProcessor
+preprocessor = PreProcessor(
+ split_length=200,
+ split_overlap=0,
+ split_respect_sentence_boundary=False,
+ clean_empty_lines=False,
+ clean_whitespace=False,
+)
+document_store.delete_documents(index=doc_index)
+document_store.delete_documents(index=label_index)
+
+# The add_eval_data() method converts the given dataset in json format into Haystack document and label objects. Those objects are then indexed in their respective document and label index in the document store. The method can be used with any dataset in SQuAD format.
+document_store.add_eval_data(
+ filename="../data/nq/nq_dev_subset_v2.json", doc_index=doc_index, label_index=label_index, preprocessor=preprocessor
+)
+```
+
+## Initialize the Two Components of an ExtractiveQAPipeline: Retriever and Reader
+
+
+```python
+# Initialize Retriever
+from haystack.nodes import ElasticsearchRetriever
+
+retriever = ElasticsearchRetriever(document_store=document_store)
+
+# Alternative: Evaluate dense retrievers (EmbeddingRetriever or DensePassageRetriever)
+# The EmbeddingRetriever uses a single transformer based encoder model for query and document.
+# In contrast, DensePassageRetriever uses two separate encoders for both.
+
+# Please make sure the "embedding_dim" parameter in the DocumentStore above matches the output dimension of your models!
+# Please also take care that the PreProcessor splits your files into chunks that can be completely converted with
+# the max_seq_len limitations of Transformers
+# The SentenceTransformer model "sentence-transformers/multi-qa-mpnet-base-dot-v1" generally works well with the EmbeddingRetriever on any kind of English text.
+# For more information and suggestions on different models check out the documentation at: https://www.sbert.net/docs/pretrained_models.html
+
+# from haystack.retriever import EmbeddingRetriever, DensePassageRetriever
+# retriever = EmbeddingRetriever(document_store=document_store, model_format="sentence_transformers",
+# embedding_model="sentence-transformers/multi-qa-mpnet-base-dot-v1")
+# retriever = DensePassageRetriever(document_store=document_store,
+# query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
+# passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
+# use_gpu=True,
+# max_seq_len_passage=256,
+# embed_title=True)
+# document_store.update_embeddings(retriever, index=doc_index)
+```
+
+
+```python
+# Initialize Reader
+from haystack.nodes import FARMReader
+
+reader = FARMReader("deepset/roberta-base-squad2", top_k=4, return_no_answer=True)
+
+# Define a pipeline consisting of the initialized retriever and reader
+from haystack.pipelines import ExtractiveQAPipeline
+
+pipeline = ExtractiveQAPipeline(reader=reader, retriever=retriever)
+
+# The evaluation also works with any other pipeline.
+# For example you could use a DocumentSearchPipeline as an alternative:
+
+# from haystack.pipelines import DocumentSearchPipeline
+# pipeline = DocumentSearchPipeline(retriever=retriever)
+```
+
+## Evaluation of an ExtractiveQAPipeline
+Here we evaluate retriever and reader in open domain fashion on the full corpus of documents i.e. a document is considered
+correctly retrieved if it contains the gold answer string within it. The reader is evaluated based purely on the
+predicted answer string, regardless of which document this came from and the position of the extracted span.
+
+The generation of predictions is seperated from the calculation of metrics. This allows you to run the computation-heavy model predictions only once and then iterate flexibly on the metrics or reports you want to generate.
+
+
+
+```python
+from haystack.schema import EvaluationResult, MultiLabel
+
+# We can load evaluation labels from the document store
+# We are also opting to filter out no_answer samples
+eval_labels = document_store.get_all_labels_aggregated(drop_negative_labels=True, drop_no_answers=False)
+eval_labels = [label for label in eval_labels if not label.no_answer] # filter out no_answer cases
+
+## Alternative: Define queries and labels directly
+
+# eval_labels = [
+# MultiLabel(
+# labels=[
+# Label(
+# query="who is written in the book of life",
+# answer=Answer(
+# answer="every person who is destined for Heaven or the World to Come",
+# offsets_in_context=[Span(374, 434)]
+# ),
+# document=Document(
+# id='1b090aec7dbd1af6739c4c80f8995877-0',
+# content_type="text",
+# content='Book of Life - wikipedia Book of Life Jump to: navigation, search This article is
+# about the book mentioned in Christian and Jewish religious teachings...'
+# ),
+# is_correct_answer=True,
+# is_correct_document=True,
+# origin="gold-label"
+# )
+# ]
+# )
+# ]
+
+# Similar to pipeline.run() we can execute pipeline.eval()
+eval_result = pipeline.eval(labels=eval_labels, params={"Retriever": {"top_k": 5}})
+```
+
+
+```python
+# The EvaluationResult contains a pandas dataframe for each pipeline node.
+# That's why there are two dataframes in the EvaluationResult of an ExtractiveQAPipeline.
+
+retriever_result = eval_result["Retriever"]
+retriever_result.head()
+```
+
+
+```python
+reader_result = eval_result["Reader"]
+reader_result.head()
+```
+
+
+```python
+# We can filter for all documents retrieved for a given query
+query = "who is written in the book of life"
+retriever_book_of_life = retriever_result[retriever_result["query"] == query]
+```
+
+
+```python
+# We can also filter for all answers predicted for a given query
+reader_book_of_life = reader_result[reader_result["query"] == query]
+```
+
+
+```python
+# Save the evaluation result so that we can reload it later and calculate evaluation metrics without running the pipeline again.
+eval_result.save("../")
+```
+
+## Calculating Evaluation Metrics
+Load an EvaluationResult to quickly calculate standard evaluation metrics for all predictions,
+such as F1-score of each individual prediction of the Reader node or recall of the retriever.
+To learn more about the metrics, see [Evaluation Metrics](https://haystack.deepset.ai/guides/evaluation#metrics-retrieval)
+
+
+```python
+saved_eval_result = EvaluationResult.load("../")
+metrics = saved_eval_result.calculate_metrics()
+print(f'Retriever - Recall (single relevant document): {metrics["Retriever"]["recall_single_hit"]}')
+print(f'Retriever - Recall (multiple relevant documents): {metrics["Retriever"]["recall_multi_hit"]}')
+print(f'Retriever - Mean Reciprocal Rank: {metrics["Retriever"]["mrr"]}')
+print(f'Retriever - Precision: {metrics["Retriever"]["precision"]}')
+print(f'Retriever - Mean Average Precision: {metrics["Retriever"]["map"]}')
+
+print(f'Reader - F1-Score: {metrics["Reader"]["f1"]}')
+print(f'Reader - Exact Match: {metrics["Reader"]["exact_match"]}')
+```
+
+## Generating an Evaluation Report
+A summary of the evaluation results can be printed to get a quick overview. It includes some aggregated metrics and also shows a few wrongly predicted examples.
+
+
+```python
+pipeline.print_eval_report(saved_eval_result)
+```
+
+## Advanced Evaluation Metrics
+As an advanced evaluation metric, semantic answer similarity (SAS) can be calculated. This metric takes into account whether the meaning of a predicted answer is similar to the annotated gold answer rather than just doing string comparison.
+To this end SAS relies on pre-trained models. For English, we recommend "cross-encoder/stsb-roberta-large", whereas for German we recommend "deepset/gbert-large-sts". A good multilingual model is "sentence-transformers/paraphrase-multilingual-mpnet-base-v2".
+More info on this metric can be found in our [paper](https://arxiv.org/abs/2108.06130) or in our [blog post](https://www.deepset.ai/blog/semantic-answer-similarity-to-evaluate-qa).
+
+
+```python
+advanced_eval_result = pipeline.eval(
+ labels=eval_labels, params={"Retriever": {"top_k": 1}}, sas_model_name_or_path="cross-encoder/stsb-roberta-large"
+)
+
+metrics = advanced_eval_result.calculate_metrics()
+print(metrics["Reader"]["sas"])
+```
+
+## Isolated Evaluation Mode
+The isolated node evaluation uses labels as input to the Reader node instead of the output of the preceeding Retriever node.
+Thereby, we can additionally calculate the upper bounds of the evaluation metrics of the Reader. Note that even with isolated evaluation enabled, integrated evaluation will still be running.
+
+
+
+```python
+eval_result_with_upper_bounds = pipeline.eval(
+ labels=eval_labels, params={"Retriever": {"top_k": 5}, "Reader": {"top_k": 5}}, add_isolated_node_eval=True
+)
+```
+
+
+```python
+pipeline.print_eval_report(eval_result_with_upper_bounds)
+```
+
+## Evaluation of Individual Components: Retriever
+Sometimes you might want to evaluate individual components, for example, if you don't have a pipeline but only a retriever or a reader with a model that you trained yourself.
+Here we evaluate only the retriever, based on whether the gold_label document is retrieved.
+
+
+```python
+## Evaluate Retriever on its own
+# Note that no_answer samples are omitted when evaluation is performed with this method
+retriever_eval_results = retriever.eval(top_k=5, label_index=label_index, doc_index=doc_index)
+# Retriever Recall is the proportion of questions for which the correct document containing the answer is
+# among the correct documents
+print("Retriever Recall:", retriever_eval_results["recall"])
+# Retriever Mean Avg Precision rewards retrievers that give relevant documents a higher rank
+print("Retriever Mean Avg Precision:", retriever_eval_results["map"])
+```
+
+Just as a sanity check, we can compare the recall from `retriever.eval()` with the multi hit recall from `pipeline.eval(add_isolated_node_eval=True)`.
+These two recall metrics are only comparable since we chose to filter out no_answer samples when generating eval_labels.
+
+
+
+```python
+metrics = eval_result_with_upper_bounds.calculate_metrics()
+print(metrics["Retriever"]["recall_multi_hit"])
+```
+
+## Evaluation of Individual Components: Reader
+Here we evaluate only the reader in a closed domain fashion i.e. the reader is given one query
+and its corresponding relevant document and metrics are calculated on whether the right position in this text is selected by
+the model as the answer span (i.e. SQuAD style)
+
+
+```python
+# Evaluate Reader on its own
+reader_eval_results = reader.eval(document_store=document_store, label_index=label_index, doc_index=doc_index)
+# Evaluation of Reader can also be done directly on a SQuAD-formatted file without passing the data to Elasticsearch
+# reader_eval_results = reader.eval_on_file("../data/nq", "nq_dev_subset_v2.json", device=device)
+
+# Reader Top-N-Accuracy is the proportion of predicted answers that match with their corresponding correct answer
+print("Reader Top-N-Accuracy:", reader_eval_results["top_n_accuracy"])
+# Reader Exact Match is the proportion of questions where the predicted answer is exactly the same as the correct answer
+print("Reader Exact Match:", reader_eval_results["EM"])
+# Reader F1-Score is the average overlap between the predicted answers and the correct answers
+print("Reader F1-Score:", reader_eval_results["f1"])
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/6.md b/docs/v1.3.0/_src/tutorials/tutorials/6.md
new file mode 100644
index 0000000000..7aaeea2797
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/6.md
@@ -0,0 +1,248 @@
+
+
+# Better Retrieval via "Dense Passage Retrieval"
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial6_Better_Retrieval_via_DPR.ipynb)
+
+### Importance of Retrievers
+
+The Retriever has a huge impact on the performance of our overall search pipeline.
+
+
+### Different types of Retrievers
+#### Sparse
+Family of algorithms based on counting the occurrences of words (bag-of-words) resulting in very sparse vectors with length = vocab size.
+
+**Examples**: BM25, TF-IDF
+
+**Pros**: Simple, fast, well explainable
+
+**Cons**: Relies on exact keyword matches between query and text
+
+
+#### Dense
+These retrievers use neural network models to create "dense" embedding vectors. Within this family there are two different approaches:
+
+a) Single encoder: Use a **single model** to embed both query and passage.
+b) Dual-encoder: Use **two models**, one to embed the query and one to embed the passage
+
+Recent work suggests that dual encoders work better, likely because they can deal better with the different nature of query and passage (length, style, syntax ...).
+
+**Examples**: REALM, DPR, Sentence-Transformers
+
+**Pros**: Captures semantinc similarity instead of "word matches" (e.g. synonyms, related topics ...)
+
+**Cons**: Computationally more heavy, initial training of model
+
+
+### "Dense Passage Retrieval"
+
+In this Tutorial, we want to highlight one "Dense Dual-Encoder" called Dense Passage Retriever.
+It was introdoced by Karpukhin et al. (2020, https://arxiv.org/abs/2004.04906.
+
+Original Abstract:
+
+_"Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks."_
+
+Paper: https://arxiv.org/abs/2004.04906
+Original Code: https://fburl.com/qa-dpr
+
+
+*Use this* [link](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial6_Better_Retrieval_via_DPR.ipynb) *to open the notebook in Google Colab.*
+
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,faiss]
+```
+
+
+```python
+from haystack.utils import clean_wiki_text, convert_files_to_dicts, fetch_archive_from_http, print_answers
+from haystack.nodes import FARMReader, TransformersReader
+```
+
+### Document Store
+
+#### Option 1: FAISS
+
+FAISS is a library for efficient similarity search on a cluster of dense vectors.
+The `FAISSDocumentStore` uses a SQL(SQLite in-memory be default) database under-the-hood
+to store the document text and other meta data. The vector embeddings of the text are
+indexed on a FAISS Index that later is queried for searching answers.
+The default flavour of FAISSDocumentStore is "Flat" but can also be set to "HNSW" for
+faster search at the expense of some accuracy. Just set the faiss_index_factor_str argument in the constructor.
+For more info on which suits your use case: https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
+
+
+```python
+from haystack.document_stores import FAISSDocumentStore
+
+document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")
+```
+
+#### Option 2: Milvus
+
+Milvus is an open source database library that is also optimized for vector similarity searches like FAISS.
+Like FAISS it has both a "Flat" and "HNSW" mode but it outperforms FAISS when it comes to dynamic data management.
+It does require a little more setup, however, as it is run through Docker and requires the setup of some config files.
+See [their docs](https://milvus.io/docs/v1.0.0/milvus_docker-cpu.md) for more details.
+
+
+```python
+# Milvus cannot be run on COlab, so this cell is commented out.
+# To run Milvus you need Docker (versions below 2.0.0) or a docker-compose (versions >= 2.0.0), neither of which is available on Colab.
+# See Milvus' documentation for more details: https://milvus.io/docs/install_standalone-docker.md
+
+# !pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[milvus]
+
+# from haystack.utils import launch_milvus
+# from haystack.document_stores import MilvusDocumentStore
+
+# launch_milvus()
+# document_store = MilvusDocumentStore()
+```
+
+### Cleaning & indexing documents
+
+Similarly to the previous tutorials, we download, convert and index some Game of Thrones articles to our DocumentStore
+
+
+```python
+# Let's first get some files that we want to use
+doc_dir = "data/tutorial6"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt6.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Convert files to dicts
+dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
+
+# Now, let's write the dicts containing documents to our DB.
+document_store.write_documents(dicts)
+```
+
+### Initalize Retriever, Reader & Pipeline
+
+#### Retriever
+
+**Here:** We use a `DensePassageRetriever`
+
+**Alternatives:**
+
+- The `ElasticsearchRetriever`with custom queries (e.g. boosting) and filters
+- Use `EmbeddingRetriever` to find candidate documents based on the similarity of embeddings (e.g. created via Sentence-BERT)
+- Use `TfidfRetriever` in combination with a SQL or InMemory Document store for simple prototyping and debugging
+
+
+```python
+from haystack.nodes import DensePassageRetriever
+
+retriever = DensePassageRetriever(
+ document_store=document_store,
+ query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
+ passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
+ max_seq_len_query=64,
+ max_seq_len_passage=256,
+ batch_size=16,
+ use_gpu=True,
+ embed_title=True,
+ use_fast_tokenizers=True,
+)
+# Important:
+# Now that after we have the DPR initialized, we need to call update_embeddings() to iterate over all
+# previously indexed documents and update their embedding representation.
+# While this can be a time consuming operation (depending on corpus size), it only needs to be done once.
+# At query time, we only need to embed the query and compare it the existing doc embeddings which is very fast.
+document_store.update_embeddings(retriever)
+```
+
+#### Reader
+
+Similar to previous Tutorials we now initalize our reader.
+
+Here we use a FARMReader with the *deepset/roberta-base-squad2* model (see: https://huggingface.co/deepset/roberta-base-squad2)
+
+
+
+##### FARMReader
+
+
+```python
+# Load a local model or any of the QA models on
+# Hugging Face's model hub (https://huggingface.co/models)
+
+reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)
+```
+
+### Pipeline
+
+With a Haystack `Pipeline` you can stick together your building blocks to a search pipeline.
+Under the hood, `Pipelines` are Directed Acyclic Graphs (DAGs) that you can easily customize for your own use cases.
+To speed things up, Haystack also comes with a few predefined Pipelines. One of them is the `ExtractiveQAPipeline` that combines a retriever and a reader to answer our questions.
+You can learn more about `Pipelines` in the [docs](https://haystack.deepset.ai/docs/latest/pipelinesmd).
+
+
+```python
+from haystack.pipelines import ExtractiveQAPipeline
+
+pipe = ExtractiveQAPipeline(reader, retriever)
+```
+
+## Voilà! Ask a question!
+
+
+```python
+# You can configure how many candidates the reader and retriever shall return
+# The higher top_k for retriever, the better (but also the slower) your answers.
+prediction = pipe.run(
+ query="Who created the Dothraki vocabulary?", params={"Retriever": {"top_k": 10}, "Reader": {"top_k": 5}}
+)
+```
+
+
+```python
+print_answers(prediction, details="minimum")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/7.md b/docs/v1.3.0/_src/tutorials/tutorials/7.md
new file mode 100644
index 0000000000..dfa9a91c35
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/7.md
@@ -0,0 +1,189 @@
+
+
+# Generative QA with "Retrieval-Augmented Generation"
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial7_RAG_Generator.ipynb)
+
+While extractive QA highlights the span of text that answers a query,
+generative QA can return a novel text answer that it has composed.
+In this tutorial, you will learn how to set up a generative system using the
+[RAG model](https://arxiv.org/abs/2005.11401) which conditions the
+answer generator on a set of retrieved documents.
+
+### Prepare environment
+
+#### Colab: Enable the GPU runtime
+Make sure you enable the GPU runtime to experience decent speed in this tutorial.
+**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**
+
+
+
+
+```python
+# Make sure you have a GPU running
+!nvidia-smi
+```
+
+Here are the packages and imports that we'll need:
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,faiss]
+```
+
+
+```python
+from typing import List
+import requests
+import pandas as pd
+from haystack import Document
+from haystack.document_stores import FAISSDocumentStore
+from haystack.nodes import RAGenerator, DensePassageRetriever
+from haystack.utils import fetch_archive_from_http
+```
+
+Let's download a csv containing some sample text and preprocess the data.
+
+
+
+```python
+# Download sample
+doc_dir = "data/tutorial7/"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/small_generator_dataset.csv.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Create dataframe with columns "title" and "text"
+df = pd.read_csv("small_generator_dataset.csv", sep=",")
+# Minimal cleaning
+df.fillna(value="", inplace=True)
+
+print(df.head())
+```
+
+We can cast our data into Haystack Document objects.
+Alternatively, we can also just use dictionaries with "text" and "meta" fields
+
+
+```python
+# Use data to initialize Document objects
+titles = list(df["title"].values)
+texts = list(df["text"].values)
+documents: List[Document] = []
+for title, text in zip(titles, texts):
+ documents.append(Document(content=text, meta={"name": title or ""}))
+```
+
+Here we initialize the FAISSDocumentStore, DensePassageRetriever and RAGenerator.
+FAISS is chosen here since it is optimized vector storage.
+
+
+```python
+# Initialize FAISS document store.
+# Set `return_embedding` to `True`, so generator doesn't have to perform re-embedding
+document_store = FAISSDocumentStore(faiss_index_factory_str="Flat", return_embedding=True)
+
+# Initialize DPR Retriever to encode documents, encode question and query documents
+retriever = DensePassageRetriever(
+ document_store=document_store,
+ query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
+ passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
+ use_gpu=True,
+ embed_title=True,
+)
+
+# Initialize RAG Generator
+generator = RAGenerator(
+ model_name_or_path="facebook/rag-token-nq",
+ use_gpu=True,
+ top_k=1,
+ max_length=200,
+ min_length=2,
+ embed_title=True,
+ num_beams=2,
+)
+```
+
+We write documents to the DocumentStore, first by deleting any remaining documents then calling `write_documents()`.
+The `update_embeddings()` method uses the retriever to create an embedding for each document.
+
+
+
+```python
+# Delete existing documents in documents store
+document_store.delete_documents()
+
+# Write documents to document store
+document_store.write_documents(documents)
+
+# Add documents embeddings to index
+document_store.update_embeddings(retriever=retriever)
+```
+
+Here are our questions:
+
+
+```python
+QUESTIONS = [
+ "who got the first nobel prize in physics",
+ "when is the next deadpool movie being released",
+ "which mode is used for short wave broadcast service",
+ "who is the owner of reading football club",
+ "when is the next scandal episode coming out",
+ "when is the last time the philadelphia won the superbowl",
+ "what is the most current adobe flash player version",
+ "how many episodes are there in dragon ball z",
+ "what is the first step in the evolution of the eye",
+ "where is gall bladder situated in human body",
+ "what is the main mineral in lithium batteries",
+ "who is the president of usa right now",
+ "where do the greasers live in the outsiders",
+ "panda is a national animal of which country",
+ "what is the name of manchester united stadium",
+]
+```
+
+Now let's run our system!
+The retriever will pick out a small subset of documents that it finds relevant.
+These are used to condition the generator as it generates the answer.
+What it should return then are novel text spans that form and answer to your question!
+
+
+```python
+# Or alternatively use the Pipeline class
+from haystack.pipelines import GenerativeQAPipeline
+from haystack.utils import print_answers
+
+pipe = GenerativeQAPipeline(generator=generator, retriever=retriever)
+for question in QUESTIONS:
+ res = pipe.run(query=question, params={"Generator": {"top_k": 1}, "Retriever": {"top_k": 5}})
+ print_answers(res, details="minimum")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/8.md b/docs/v1.3.0/_src/tutorials/tutorials/8.md
new file mode 100644
index 0000000000..ffbd75de36
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/8.md
@@ -0,0 +1,207 @@
+
+
+# Preprocessing
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial8_Preprocessing.ipynb)
+
+Haystack includes a suite of tools to extract text from different file types, normalize white space
+and split text into smaller pieces to optimize retrieval.
+These data preprocessing steps can have a big impact on the systems performance and effective handling of data is key to getting the most out of Haystack.
+
+Ultimately, Haystack expects data to be provided as a list documents in the following dictionary format:
+``` python
+docs = [
+ {
+ 'content': DOCUMENT_TEXT_HERE,
+ 'meta': {'name': DOCUMENT_NAME, ...}
+ }, ...
+]
+```
+
+This tutorial will show you all the tools that Haystack provides to help you cast your data into this format.
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+from pathlib import Path
+
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,ocr]
+
+!wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.03.tar.gz
+!tar -xvf xpdf-tools-linux-4.03.tar.gz && sudo cp xpdf-tools-linux-4.03/bin64/pdftotext /usr/local/bin
+```
+
+
+```python
+# Here are the imports we need
+from haystack.nodes import TextConverter, PDFToTextConverter, DocxToTextConverter, PreProcessor
+from haystack.utils import convert_files_to_dicts, fetch_archive_from_http
+```
+
+
+```python
+# This fetches some sample files to work with
+
+doc_dir = "data/tutorial8"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/preprocessing_tutorial8.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+```
+
+## Converters
+
+Haystack's converter classes are designed to help you turn files on your computer into the documents
+that can be processed by the Haystack pipeline.
+There are file converters for txt, pdf, docx files as well as a converter that is powered by Apache Tika.
+The parameter `valid_langugages` does not convert files to the target language, but checks if the conversion worked as expected.
+For converting PDFs, try changing the encoding to UTF-8 if the conversion isn't great.
+
+
+```python
+# Here are some examples of how you would use file converters
+
+converter = TextConverter(remove_numeric_tables=True, valid_languages=["en"])
+doc_txt = converter.convert(file_path=Path(f"{doc_dir}/classics.txt"), meta=None)[0]
+
+converter = PDFToTextConverter(remove_numeric_tables=True, valid_languages=["en"])
+doc_pdf = converter.convert(file_path=Path(f"{doc_dir}/bert.pdf"), meta=None)[0]
+
+converter = DocxToTextConverter(remove_numeric_tables=False, valid_languages=["en"])
+doc_docx = converter.convert(file_path=Path(f"{doc_dir}/heavy_metal.docx"), meta=None)[0]
+```
+
+
+```python
+# Haystack also has a convenience function that will automatically apply the right converter to each file in a directory.
+
+all_docs = convert_files_to_dicts(dir_path=doc_dir)
+```
+
+## PreProcessor
+
+The PreProcessor class is designed to help you clean text and split text into sensible units.
+File splitting can have a very significant impact on the system's performance and is absolutely mandatory for Dense Passage Retrieval models.
+In general, we recommend you split the text from your files into small documents of around 100 words for dense retrieval methods
+and no more than 10,000 words for sparse methods.
+Have a look at the [Preprocessing](https://haystack.deepset.ai/docs/latest/preprocessingmd)
+and [Optimization](https://haystack.deepset.ai/docs/latest/optimizationmd) pages on our website for more details.
+
+
+```python
+# This is a default usage of the PreProcessor.
+# Here, it performs cleaning of consecutive whitespaces
+# and splits a single large document into smaller documents.
+# Each document is up to 1000 words long and document breaks cannot fall in the middle of sentences
+# Note how the single document passed into the document gets split into 5 smaller documents
+
+preprocessor = PreProcessor(
+ clean_empty_lines=True,
+ clean_whitespace=True,
+ clean_header_footer=False,
+ split_by="word",
+ split_length=100,
+ split_respect_sentence_boundary=True,
+)
+docs_default = preprocessor.process(doc_txt)
+print(f"n_docs_input: 1\nn_docs_output: {len(docs_default)}")
+```
+
+## Cleaning
+
+- `clean_empty_lines` will normalize 3 or more consecutive empty lines to be just a two empty lines
+- `clean_whitespace` will remove any whitespace at the beginning or end of each line in the text
+- `clean_header_footer` will remove any long header or footer texts that are repeated on each page
+
+## Splitting
+By default, the PreProcessor will respect sentence boundaries, meaning that documents will not start or end
+midway through a sentence.
+This will help reduce the possibility of answer phrases being split between two documents.
+This feature can be turned off by setting `split_respect_sentence_boundary=False`.
+
+
+```python
+# Not respecting sentence boundary vs respecting sentence boundary
+
+preprocessor_nrsb = PreProcessor(split_respect_sentence_boundary=False)
+docs_nrsb = preprocessor_nrsb.process(doc_txt)
+
+print("RESPECTING SENTENCE BOUNDARY")
+end_text = docs_default[0]["content"][-50:]
+print('End of document: "...' + end_text + '"')
+print()
+print("NOT RESPECTING SENTENCE BOUNDARY")
+end_text_nrsb = docs_nrsb[0]["content"][-50:]
+print('End of document: "...' + end_text_nrsb + '"')
+```
+
+A commonly used strategy to split long documents, especially in the field of Question Answering,
+is the sliding window approach. If `split_length=10` and `split_overlap=3`, your documents will look like this:
+
+- doc1 = words[0:10]
+- doc2 = words[7:17]
+- doc3 = words[14:24]
+- ...
+
+You can use this strategy by following the code below.
+
+
+```python
+# Sliding window approach
+
+preprocessor_sliding_window = PreProcessor(split_overlap=3, split_length=10, split_respect_sentence_boundary=False)
+docs_sliding_window = preprocessor_sliding_window.process(doc_txt)
+
+doc1 = docs_sliding_window[0]["content"][:200]
+doc2 = docs_sliding_window[1]["content"][:100]
+doc3 = docs_sliding_window[2]["content"][:100]
+
+print('Document 1: "' + doc1 + '..."')
+print('Document 2: "' + doc2 + '..."')
+print('Document 3: "' + doc3 + '..."')
+```
+
+## Bringing it all together
+
+
+```python
+all_docs = convert_files_to_dicts(dir_path="data/preprocessing_tutorial")
+preprocessor = PreProcessor(
+ clean_empty_lines=True,
+ clean_whitespace=True,
+ clean_header_footer=False,
+ split_by="word",
+ split_length=100,
+ split_respect_sentence_boundary=True,
+)
+docs = preprocessor.process(all_docs)
+
+print(f"n_files_input: {len(all_docs)}\nn_docs_output: {len(docs)}")
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
+
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/9.md b/docs/v1.3.0/_src/tutorials/tutorials/9.md
new file mode 100644
index 0000000000..e0b1b34dfd
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/9.md
@@ -0,0 +1,250 @@
+
+
+# Training Your Own "Dense Passage Retrieval" Model
+
+[](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial9_DPR_training.ipynb)
+
+Haystack contains all the tools needed to train your own Dense Passage Retrieval model.
+This tutorial will guide you through the steps required to create a retriever that is specifically tailored to your domain.
+
+
+```python
+# Install the latest release of Haystack in your own environment
+#! pip install farm-haystack
+
+# Install the latest master of Haystack
+!pip install --upgrade pip
+!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab]
+```
+
+
+```python
+# Here are some imports that we'll need
+
+from haystack.nodes import DensePassageRetriever
+from haystack.utils import fetch_archive_from_http
+from haystack.document_stores import InMemoryDocumentStore
+```
+
+## Training Data
+
+DPR training performed using Information Retrieval data.
+More specifically, you want to feed in pairs of queries and relevant documents.
+
+To train a model, we will need a dataset that has the same format as the original DPR training data.
+Each data point in the dataset should have the following dictionary structure.
+
+``` python
+ {
+ "dataset": str,
+ "question": str,
+ "answers": list of str
+ "positive_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
+ "negative_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
+ "hard_negative_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
+ }
+```
+
+`positive_ctxs` are context passages which are relevant to the query.
+In some datasets, queries might have more than one positive context
+in which case you can set the `num_positives` parameter to be higher than the default 1.
+Note that `num_positives` needs to be lower or equal to the minimum number of `positive_ctxs` for queries in your data.
+If you have an unequal number of positive contexts per example,
+you might want to generate some soft labels by retrieving similar contexts which contain the answer.
+
+DPR is standardly trained using a method known as in-batch negatives.
+This means that positive contexts for a given query are treated as negative contexts for the other queries in the batch.
+Doing so allows for a high degree of computational efficiency, thus allowing the model to be trained on large amounts of data.
+
+`negative_ctxs` is not actually used in Haystack's DPR training so we recommend you set it to an empty list.
+They were used by the original DPR authors in an experiment to compare it against the in-batch negatives method.
+
+`hard_negative_ctxs` are passages that are not relevant to the query.
+In the original DPR paper, these are fetched using a retriever to find the most relevant passages to the query.
+Passages which contain the answer text are filtered out.
+
+If you'd like to convert your SQuAD format data into something that can train a DPR model,
+check out the utility script at [`haystack/retriever/squad_to_dpr.py`](https://github.com/deepset-ai/haystack/blob/master/haystack/retriever/squad_to_dpr.py)
+
+## Using Question Answering Data
+
+Question Answering datasets can sometimes be used as training data.
+Google's Natural Questions dataset, is sufficiently large
+and contains enough unique passages, that it can be converted into a DPR training set.
+This is done simply by considering answer containing passages as relevant documents to the query.
+
+The SQuAD dataset, however, is not as suited to this use case since its question and answer pairs
+are created on only a very small slice of wikipedia documents.
+
+## Download Original DPR Training Data
+
+WARNING: These files are large! The train set is 7.4GB and the dev set is 800MB
+
+We can download the original DPR training data with the following cell.
+Note that this data is probably only useful if you are trying to train from scratch.
+
+
+```python
+# Download original DPR data
+# WARNING: the train set is 7.4GB and the dev set is 800MB
+
+doc_dir = "data/tutorial9"
+
+s3_url_train = "https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-train.json.gz"
+s3_url_dev = "https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-dev.json.gz"
+
+fetch_archive_from_http(s3_url_train, output_dir=doc_dir + "train/")
+fetch_archive_from_http(s3_url_dev, output_dir=doc_dir + "dev/")
+```
+
+## Option 1: Training DPR from Scratch
+
+The default variables that we provide below are chosen to train a DPR model from scratch.
+Here, both passage and query embedding models are initialized using BERT base
+and the model is trained using Google's Natural Questions dataset (in a format specialised for DPR).
+
+If you are working in a language other than English,
+you will want to initialize the passage and query embedding models with a language model that supports your language
+and also provide a dataset in your language.
+
+
+```python
+# Here are the variables to specify our training data, the models that we use to initialize DPR
+# and the directory where we'll be saving the model
+
+doc_dir = "data/dpr_training/"
+
+train_filename = "train/biencoder-nq-train.json"
+dev_filename = "dev/biencoder-nq-dev.json"
+
+query_model = "bert-base-uncased"
+passage_model = "bert-base-uncased"
+
+save_dir = "../saved_models/dpr"
+```
+
+## Option 2: Finetuning DPR
+
+If you have your own domain specific question answering or information retrieval dataset,
+you might instead be interested in finetuning a pretrained DPR model.
+In this case, you would initialize both query and passage models using the original pretrained model.
+You will want to load something like this set of variables instead of the ones above
+
+
+```python
+# Here are the variables you might want to use instead of the set above
+# in order to perform pretraining
+
+doc_dir = "PATH_TO_YOUR_DATA_DIR"
+train_filename = "TRAIN_FILENAME"
+dev_filename = "DEV_FILENAME"
+
+query_model = "facebook/dpr-question_encoder-single-nq-base"
+passage_model = "facebook/dpr-ctx_encoder-single-nq-base"
+
+save_dir = "../saved_models/dpr"
+```
+
+## Initialization
+
+Here we want to initialize our model either with plain language model weights for training from scratch
+or else with pretrained DPR weights for finetuning.
+We follow the [original DPR parameters](https://github.com/facebookresearch/DPR#best-hyperparameter-settings)
+for their max passage length but set max query length to 64 since queries are very rarely longer.
+
+
+```python
+## Initialize DPR model
+
+retriever = DensePassageRetriever(
+ document_store=InMemoryDocumentStore(),
+ query_embedding_model=query_model,
+ passage_embedding_model=passage_model,
+ max_seq_len_query=64,
+ max_seq_len_passage=256,
+)
+```
+
+## Training
+
+Let's start training and save our trained model!
+
+On a V100 GPU, you can fit up to batch size 16 so we set gradient accumulation steps to 8 in order
+to simulate the batch size 128 of the original DPR experiment.
+
+When `embed_title=True`, the document title is prepended to the input text sequence with a `[SEP]` token
+between it and document text.
+
+When training from scratch with the above variables, 1 epoch takes around an hour and we reached the following performance:
+
+```
+loss: 0.046580662854042276
+task_name: text_similarity
+acc: 0.992524064068483
+f1: 0.8804297774366846
+acc_and_f1: 0.9364769207525838
+average_rank: 0.19631619339984652
+report:
+ precision recall f1-score support
+
+hard_negative 0.9961 0.9961 0.9961 201887
+ positive 0.8804 0.8804 0.8804 6515
+
+ accuracy 0.9925 208402
+ macro avg 0.9383 0.9383 0.9383 208402
+ weighted avg 0.9925 0.9925 0.9925 208402
+
+```
+
+
+```python
+# Start training our model and save it when it is finished
+
+retriever.train(
+ data_dir=doc_dir,
+ train_filename=train_filename,
+ dev_filename=dev_filename,
+ test_filename=dev_filename,
+ n_epochs=1,
+ batch_size=16,
+ grad_acc_steps=8,
+ save_dir=save_dir,
+ evaluate_every=3000,
+ embed_title=True,
+ num_positives=1,
+ num_hard_negatives=1,
+)
+```
+
+## Loading
+
+Loading our newly trained model is simple!
+
+
+```python
+reloaded_retriever = DensePassageRetriever.load(load_dir=save_dir, document_store=None)
+```
+
+## About us
+
+This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany
+
+We bring NLP to the industry via open source!
+Our focus: Industry specific language models & large scale QA systems.
+
+Some of our other work:
+- [German BERT](https://deepset.ai/german-bert)
+- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
+- [FARM](https://github.com/deepset-ai/FARM)
+
+Get in touch:
+[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
+
+By the way: [we're hiring!](https://www.deepset.ai/jobs)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/convert_ipynb.py b/docs/v1.3.0/_src/tutorials/tutorials/convert_ipynb.py
new file mode 100644
index 0000000000..3df5c1eec1
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/convert_ipynb.py
@@ -0,0 +1,31 @@
+import re
+
+from nbconvert import MarkdownExporter
+import os
+from pathlib import Path
+from headers import headers
+
+
+def atoi(text):
+ return int(text) if text.isdigit() else text
+
+
+def natural_keys(text):
+ test = [atoi(c) for c in re.split("(\d+)", text)]
+ return test
+
+
+dir = Path("../../../../tutorials")
+
+notebooks = [x for x in os.listdir(dir) if x[-6:] == ".ipynb"]
+# sort notebooks based on numbers within name of notebook
+notebooks = sorted(notebooks, key=lambda x: natural_keys(x))
+
+
+e = MarkdownExporter(exclude_output=True)
+for i, nb in enumerate(notebooks):
+ body, resources = e.from_filename(dir / nb)
+ print(f"Processing {dir}/{nb}")
+ with open(str(i + 1) + ".md", "w", encoding="utf-8") as f:
+ f.write(headers[i + 1] + "\n\n")
+ f.write(body)
diff --git a/docs/v1.3.0/_src/tutorials/tutorials/headers.py b/docs/v1.3.0/_src/tutorials/tutorials/headers.py
new file mode 100644
index 0000000000..b6ae875802
--- /dev/null
+++ b/docs/v1.3.0/_src/tutorials/tutorials/headers.py
@@ -0,0 +1,130 @@
+headers = {
+ 1: """""",
+ 2: """""",
+ 3: """""",
+ 4: """""",
+ 5: """""",
+ 6: """""",
+ 7: """""",
+ 8: """""",
+ 9: """""",
+ 10: """""",
+ 11: """""",
+ 12: """""",
+ 13: """""",
+ 14: """""",
+ 15: """""",
+ 16: """""",
+}
diff --git a/docs/v1.3.0/_src/usage/Makefile b/docs/v1.3.0/_src/usage/Makefile
new file mode 100644
index 0000000000..d4bb2cbb9e
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS ?=
+SPHINXBUILD ?= sphinx-build
+SOURCEDIR = .
+BUILDDIR = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+ @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+ @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/v1.3.0/_src/usage/conf.py b/docs/v1.3.0/_src/usage/conf.py
new file mode 100644
index 0000000000..12ac810874
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/conf.py
@@ -0,0 +1,51 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "Usage"
+copyright = "2020, deepset"
+author = "deepset"
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = ["sphinx_tabs.tabs", "sphinx_copybutton", "nbsphinx", "sphinx.ext.autosectionlabel"]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "**.ipynb_checkpoints"]
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+#
+html_theme = "alabaster"
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]
diff --git a/docs/v1.3.0/_src/usage/index.rst b/docs/v1.3.0/_src/usage/index.rst
new file mode 100644
index 0000000000..1a615bf974
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/index.rst
@@ -0,0 +1,23 @@
+.. Haystack documentation master file, created by
+ sphinx-quickstart on Tue Jul 28 16:18:42 2020.
+ You can adapt this file completely to your liking, but it should at least
+ contain the root `toctree` directive.
+
+Haystack is all you need to scale the latest advances in Question Answering technologies to large collections of documents.
+Whether you're building a search system, asking standard questions to an incoming stream of documents or
+extending your chatbot, Haystack has you covered.
+
+This section covers the core concepts and practical usage of Haystack. See our tutorials page for end to end examples
+or our API page for comprehensive descriptions of individual functions.
+
+.. toctree::
+ :maxdepth: 4
+ :caption: Contents:
+
+ usage/intro
+ usage/get_started
+ usage/database
+ usage/retriever
+ usage/reader
+ usage/domain_adaptation
+ usage/terms
diff --git a/docs/v1.3.0/_src/usage/make.bat b/docs/v1.3.0/_src/usage/make.bat
new file mode 100644
index 0000000000..2119f51099
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/make.bat
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+ set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+ echo.
+ echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+ echo.installed, then set the SPHINXBUILD environment variable to point
+ echo.to the full path of the 'sphinx-build' executable. Alternatively you
+ echo.may add the Sphinx directory to PATH.
+ echo.
+ echo.If you don't have Sphinx installed, grab it from
+ echo.http://sphinx-doc.org/
+ exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/docs/v1.3.0/_src/usage/usage/annotation.md b/docs/v1.3.0/_src/usage/usage/annotation.md
new file mode 100644
index 0000000000..880b4108f4
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/annotation.md
@@ -0,0 +1,69 @@
+# Annotation Tool
+
+- Create labels with different techniques: Come up with questions (+ answers) while reading passages (SQuAD style) or have a set of predefined questions and look for answers in the document (~ Natural Questions).
+- Structure your work via organizations, projects, users
+- Upload your documents or import labels from an existing SQuAD-style dataset
+- Export your labels in SQuAD Format
+
+
+
+# Hosted version
+ Signup here: [Haystack Annotation Tool](https://annotate.deepset.ai/login)
+
+# Local version (Docker)
+
+1. Configure credentials & database in the [`docker-compose.yml`](https://github.com/deepset-ai/haystack/blob/master/annotation_tool/docker-compose.yml):
+
+The credentials should match in database image and application configuration.
+
+ DEFAULT_ADMIN_EMAIL: "example@example.com"
+ DEFAULT_ADMIN_PASSWORD: "DEMO-PASSWORD"
+
+ PROD_DB_NAME: "databasename"
+ PROD_DB_USERNAME: "somesafeuser"
+ PROD_DB_PASSWORD: "somesafepassword"
+
+
+ POSTGRES_USER: "somesafeuser"
+ POSTGRES_PASSWORD: "somesafepassword"
+ POSTGRES_DB: "databasename"
+
+
+2. Run docker-compose by executing `docker-compose up`.
+
+
+3. The UI should be available at `localhost:7001`.
+
+# Usage
+The manual (of a slightly earlier version) can be found [here](https://drive.google.com/file/d/1Wv3OIC0Z7ibHIzOm9Xw_r0gjTFmpl-33/view). While it doesn't include all latest features, the basic workflow and tips for label quality are still the same.
+
+# Annotation FAQ
+
+1. What is a good question?
+ - A good question is a fact-seeking question that can be answered with an entity (person, organisation, location, etc.) or explanation. A bad question is ambiguous, incomprehensible, dependent on clear false presuppositions, opinion seeking, or not clearly a request for factual information.
+ - The question should ask about information present in the text passage given. It should not be answerable only with additional knowledge or your interpretation.
+ - Do not copy paste answer text into the question. Good questions do not contain the exact same words as the answer or the context around the answer. The question should be a reformulation with synonyms and in different order as the context of the answer.
+ - Questions should be very precise natural questions you would ask when you want information from another person.
+2. How many questions should you ask per text passage?
+ - Maximally ask 20 questions per passage
+ - Some text passages are not suited for 20 questions. Do not make up very constructed and complicated questions just to fill up the 20 - move on to the next text.
+ - Try to ask questions covering the whole passage and focus on questions covering important information. Do not only ask questions about a single sentence in that passage.
+3. What is a good answer span?
+ - Always mark whole words. Do not start or end the answer within a word.
+ - For short answers: The answer should be as short and as close to a spoken human answer as possible. Do not include punctuation.
+ - For long answers: Please mark whole sentences with punctuation. The sentences can also pick up parts of the question, or mark even whole text passages. Mark passages only if they are not too large (e.g. not more than 8-10 sentences).
+4. How do I differentiate long vs short answers?
+ - If there is a short answer possible you should always select short answer over long answer.
+ - Short precise answers like numbers or a few words are short answers.
+ - Long answers include lists of possibilities or multiple sentences are needed to answer the question correctly.
+5. How to handle multiple possible answers to a single question?
+ - As of now there is no functionality to mark multiple answers per single question.
+ - Workaround: You can add a question with the same text but different answer selection by using the button below the question list (Button reads “custom question”)
+6. What to do with grammatically wrong or incorrectly spelled questions?
+ - Include them. When users use the tool and ask questions they will likely contain grammar and spelling errors, too.
+ - Exception: The question needs to be understandable without reading and interpretation of the corresponding text passage. If you do not understand the question, please mark the question as “I don’t understand the question”.
+7. What to do with text passages that are not properly converted or contain (in part) information that cannot be labelled (e.g. just lists or garbage text)?
+ - Please do not annotate this text
+ - You can write down what is missing, or the cause why you cannot label the text + the text number and title.
+8. Which browser to use?
+ - Please use the Chrome browser. The tool is not tested for other browsers.
diff --git a/docs/v1.3.0/_src/usage/usage/document_store.md b/docs/v1.3.0/_src/usage/usage/document_store.md
new file mode 100644
index 0000000000..bca65eac2d
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/document_store.md
@@ -0,0 +1,362 @@
+
+
+
+# DocumentStores
+
+You can think of the DocumentStore as a "database" that:
+- stores your texts and meta data
+- provides them to the retriever at query time
+
+There are different DocumentStores in Haystack to fit different use cases and tech stacks.
+
+## Initialisation
+
+Initialising a new DocumentStore within Haystack is straight forward.
+
+
+
+
+
+Each DocumentStore constructor allows for arguments specifying how to connect to existing databases and the names of indexes.
+See API documentation for more info.
+
+## Input Format
+
+DocumentStores expect Documents in dictionary form, like that below.
+They are loaded using the `DocumentStore.write_documents()` method.
+See [Preprocessing](/docs/latest/preprocessingmd) for more information on the cleaning and splitting steps that will help you maximize Haystack's performance.
+
+[//]: # (Add link to preprocessing section)
+
+```python
+from haystack.document_store import ElasticsearchDocumentStore
+
+document_store = ElasticsearchDocumentStore()
+dicts = [
+ {
+ 'text': DOCUMENT_TEXT_HERE,
+ 'meta': {'name': DOCUMENT_NAME, ...}
+ }, ...
+]
+document_store.write_documents(dicts)
+```
+
+## Writing Documents (Sparse Retrievers)
+
+Haystack allows for you to write store documents in an optimised fashion so that query times can be kept low.
+For **sparse**, keyword based retrievers such as BM25 and TF-IDF,
+you simply have to call `DocumentStore.write_documents()`.
+The creation of the inverted index which optimises querying speed is handled automatically.
+
+```python
+document_store.write_documents(dicts)
+```
+
+## Writing Documents (Dense Retrievers)
+
+For **dense** neural network based retrievers like Dense Passage Retrieval, or Embedding Retrieval,
+indexing involves computing the Document embeddings which will be compared against the Query embedding.
+
+The storing of the text is handled by `DocumentStore.write_documents()` and the computation of the
+embeddings is started by `DocumentStore.update_embeddings()`.
+
+```python
+document_store.write_documents(dicts)
+document_store.update_embeddings(retriever)
+```
+
+This step is computationally intensive since it will engage the transformer based encoders.
+Having GPU acceleration will significantly speed this up.
+
+
+
+## Choosing the Right Document Store
+
+The Document Stores have different characteristics. You should choose one depending on the maturity of your project, the use case and technical environment:
+
+
+
+
+
+
+
+
+[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html)
+Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html)
+an instance.
+
+If you have Docker set up, we recommend pulling the Docker image and running it.
+```bash
+docker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.2
+docker run -d -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.9.2
+```
+
+Note that we also have a utility function `haystack.utils.launch_es` that can start up an Elasticsearch instance.
+
+Next you can initialize the Haystack object that will connect to this instance.
+
+```python
+from haystack.document_store import ElasticsearchDocumentStore
+
+document_store = ElasticsearchDocumentStore()
+```
+
+Note that we also support [OpenSearch](https://opensearch.org/).
+Follow [their documentation](https://opensearch.org/docs/)
+to run it and connect to it using Haystack's `OpenSearchDocumentStore` class.
+
+We further support [AWS Elastic Search Service](https://aws.amazon.com/elasticsearch-service/) with [signed Requests](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html):
+Use e.g. [aws-requests-auth](https://github.com/davidmuller/aws-requests-auth) to create an auth object and pass it as `aws4auth` to the `ElasticsearchDocumentStore` constructor.
+
+
+
+
+
+
+
+
+
+Follow the [official documentation](https://www.milvus.io/docs/v1.0.0/milvus_docker-cpu.md) to start a Milvus instance via Docker.
+Note that we also have a utility function `haystack.utils.launch_milvus` that can start up a Milvus instance.
+
+You can initialize the Haystack object that will connect to this instance as follows:
+```python
+from haystack.document_store import MilvusDocumentStore
+
+document_store = MilvusDocumentStore()
+```
+
+
+
+
+
+
+
+
+
+The `FAISSDocumentStore` requires no external setup. Start it by simply using this line.
+```python
+from haystack.document_store import FAISSDocumentStore
+
+document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")
+```
+
+
+
+
+
+
+
+
+
+The `InMemoryDocumentStore()` requires no external setup. Start it by simply using this line.
+```python
+from haystack.document_store import InMemoryDocumentStore
+
+document_store = InMemoryDocumentStore()
+```
+
+
+
+
+
+
+
+
+
+The `SQLDocumentStore` requires SQLite, PostgresQL or MySQL to be installed and started.
+Note that SQLite already comes packaged with most operating systems.
+
+```python
+from haystack.document_store import SQLDocumentStore
+
+document_store = SQLDocumentStore()
+```
+
+
+
+
+
+
+
+
+
+The `WeaviateDocumentStore` requires a running Weaviate Server.
+You can start a basic instance like this (see the [Weaviate docs](https://www.semi.technology/developers/weaviate/current/) for details):
+```
+ docker run -d -p 8080:8080 --env AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED='true' --env PERSISTENCE_DATA_PATH='/var/lib/weaviate' semitechnologies/weaviate:1.7.2
+```
+
+Afterwards, you can use it in Haystack:
+```python
+from haystack.document_store import WeaviateDocumentStore
+
+document_store = WeaviateDocumentStore()
+```
+
+
+
+
+
+
+
+
+
+See the official [OpenSearch documentation](https://opensearch.org/docs/opensearch/install/docker/) on how to install and start an instance.
+
+If you have Docker set up, we recommend pulling the Docker image and running it.
+```bash
+docker pull opensearchproject/opensearch:1.0.0
+docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.0.0
+```
+
+Note that we also have a utility function `haystack.utils.launch_opensearch` that can start up an OpenSearch instance.
+
+Next you can initialize the Haystack object that will connect to this instance.
+
+```python
+from haystack.document_store import OpenSearchDocumentStore
+
+document_store = OpenSearchDocumentStore()
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Fast & accurate sparse retrieval with many tuning options
+- Basic support for dense retrieval
+- Production-ready
+- Support also for Open Distro
+
+**Cons:**
+- Slow for dense retrieval with more than ~ 1 Mio documents
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Scalable DocumentStore that excels at handling vectors (hence suited to dense retrieval methods like DPR)
+- Encapsulates multiple ANN libraries (e.g. FAISS and ANNOY) and provides added reliability
+- Runs as a separate service (e.g. a Docker container)
+- Allows dynamic data management
+
+**Cons:**
+- No efficient sparse retrieval
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Fast & accurate dense retrieval
+- Highly scalable due to approximate nearest neighbour algorithms (ANN)
+- Many options to tune dense retrieval via different index types (more info [here](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index))
+
+**Cons:**
+- No efficient sparse retrieval
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Simple
+- Exists already in many environments
+
+**Cons:**
+- Only compatible with minimal TF-IDF Retriever
+- Bad retrieval performance
+- Not recommended for production
+
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Simple & fast to test
+- No database requirements
+- Supports MySQL, PostgreSQL and SQLite
+
+**Cons:**
+- Not scalable
+- Not persisting your data on disk
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Simple vector search
+- Stores everything in one place: documents, meta data and vectors - so less network overhead when scaling this up
+- Allows combination of vector search and scalar filtering, i.e. you can filter for a certain tag and do dense retrieval on that subset
+
+**Cons:**
+- Less options for ANN algorithms than FAISS or Milvus
+- No BM25 / Tf-idf retrieval
+
+
+
+
+
+
+
+
+
+**Pros:**
+- Fully open source fork of Elasticsearch
+- Has support for Approximate Nearest Neighbours vector search
+
+**Cons:**
+- It's ANN algorithms seem a little less performant that FAISS or Milvus in our benchmarks
+
+
+
+
+#### Our Recommendations
+
+**Restricted environment:** Use the `InMemoryDocumentStore`, if you are just giving Haystack a quick try on a small sample and are working in a restricted environment that complicates running Elasticsearch or other databases
+
+**Allrounder:** Use the `ElasticSearchDocumentStore`, if you want to evaluate the performance of different retrieval options (dense vs. sparse) and are aiming for a smooth transition from PoC to production
+
+**Vector Specialist:** Use the `MilvusDocumentStore`, if you want to focus on dense retrieval and possibly deal with larger datasets
+
+
diff --git a/docs/v1.3.0/_src/usage/usage/domain_adaptation.md b/docs/v1.3.0/_src/usage/usage/domain_adaptation.md
new file mode 100644
index 0000000000..dede4d79cc
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/domain_adaptation.md
@@ -0,0 +1,157 @@
+
+
+# Domain Adaptation
+
+## Generalisation
+
+In our experience, language models trained on SQuAD show very strong general question answering capabilities.
+Though SQuAD is composed entirely of Wikipedia articles, these models are flexible enough to deal with many different styles of text.
+
+Before trying to adapt these models to your domain, we’d recommend trying one of the off the shelf models.
+We’ve found that these models are often flexible enough for a wide range of use cases.
+
+
+
+**Intuition**
+
+Most people probably don’t know what an HP Valve is.
+But you don’t always need to know what a HP Valve is to answer “What is connected to a HP Valve?”
+The answer might be there in plain language.
+In the same way, many QA models have a good enough grasp of language to answer questions about concepts in an unseen domain.
+
+
+
+
+## Finetuning
+
+Any model that can be loaded into Haystack can also be finetuned within Haystack.
+Simply provide the domain specific dataset and call `Reader.train()` on an initialised model.
+
+```
+reader.train(data_dir=train_data,
+ train_filename="dev-v2.0.json",
+ n_epochs=1,
+ save_dir="my_model")
+```
+
+At the end of training, the finetuned model will be saved in the specified `save_dir` and can be loaded as a `Reader`.
+
+
+
+**Recommendation**
+
+See Tutorial 2 for a runnable example of this process.
+If you’re interested in measuring how much your model has improved,
+please also check out Tutorial 5 which walks through the steps needed to perform evaluation.
+
+
+
+## Generating Labels
+
+Using our [Haystack Annotate tool](https://annotate.deepset.ai/login) (Beta),
+you can easily create a labelled dataset using your own documents featuring your own question/ answer pairs.
+
+
+
+
+
+Features include:
+
+
+* Structured workspaces via organisations, projects and users
+
+
+* Easy upload of your own documents and labels in a variety of formats (txt, pdf, SQuAD style)
+
+
+* Export of labels to be used directly in Haystack
+
+Annotate also supports two different workflows:
+
+
+* Think up questions and answers while reading passages (SQuAD style)
+
+
+* Have a set of predefined questions and look for answers in the document (~ Natural Questions style)
+
+## User Feedback
+
+A simpler and faster process to finetune models to your domain is to utilise user feedback.
+Dedicated annotation work can be costly and time consuming
+but user feedback is an efficient alternative since it allows for labels to be generated by users, for users,
+all while the system is already in production.
+We, for example, have used a simple thumbs up / down system in our demos to allow
+users to give feedback.
+
+
+
+Through the Rest API, users can annotate each Haystack result as being either:
+
+* Correct
+* Correct document but wrong text
+* Wrong document and wrong text
+
+To get started, follow these steps:
+
+* Start up the REST API
+ * The simplest way to do this is to call `docker-compose up` from the root directory of the Haystack repository
+ * Alternatively, run `gunicorn rest_api.application:app -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker -t 300`
+* Make a POST request to the `doc-qa-feedback` endpoint with the following payload:
+```
+{
+ "question": "string",
+ "is_correct_answer": true,
+ "document_id": "string",
+ "model_id": 0,
+ "is_correct_document": true,
+ "answer": "string",
+ "offset_start_in_doc": 0
+}
+```
+To fill out all these values, you can use the response from an executed search request on the `doc-qa` endpoint.
+
+* Export your labels in SQuAD format by making a GET request to the `export-doc-qa-feedback` endpoint
+``` python
+# SQUAD format
+{
+ "data": [
+ {
+ "title": "Normans",
+ "paragraphs": [
+ {
+ "context": "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the...",
+ "qas": [
+ {
+ "question": "In what country is Normandy located?",
+ "id": "56ddde6b9a695914005b9628",
+ "answers": [
+ {
+ "text": "France",
+ "answer_start": 159
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
+```
+
+User feedback labelling also works with FAQ style QA.
+Refer to the Swagger API documentation (http://127.0.0.1:8000/docs) for more details.
+
+This labelled data can then be used to
+augment the training data and enable `Reader` finetuning.
+Alternatively, they can also be used to form an evaluation set to
+measure the performance of trained models,
+or monitor how well the model is doing in a live environment.
+
diff --git a/docs/v1.3.0/_src/usage/usage/faq.md b/docs/v1.3.0/_src/usage/usage/faq.md
new file mode 100644
index 0000000000..37b09ff9b8
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/faq.md
@@ -0,0 +1,90 @@
+
+
+#Frequently Asked Questions
+
+##Why am I seeing duplicate answers being returned?
+
+The ElasticsearchDocumentStore and MilvusDocumentStore rely on Elasticsearch and Milvus backend services which
+persist after your Python script has finished running.
+If you rerun your script without deleting documents, you could end up with duplicate
+copies of your documents in your database.
+The easiest way to avoid this is to call `DocumentStore.delete_documents()` after initialization
+to ensure that you are working with an empty DocumentStore.
+
+DocumentStores also have a `duplicate_documents` argument in their `__init__()` and `write_documents` methods
+where you can define whether you'd like skip writing duplicates, overwrite existing duplicates or raise an error when there are duplicates.
+
+##How can I make sure that my GPU is being engaged when I use Haystack?
+
+You will want to ensure that a CUDA enabled GPU is being engaged when Haystack is running (you can check by running `nvidia-smi -l` on your command line).
+Components which can be sped up by GPU have a `use_gpu` argument in their constructor which you will want to set to `True`.
+
+##How do I speed up my predictions?
+
+There are many different ways to speed up the performance of your Haystack system.
+
+The Reader is usually the most computationally expensive component in a pipeline
+and you can often speed up your system by using a smaller model, like `deepset/minilm-uncased-squad2` (see [benchmarks](https://huggingface.co/deepset/minilm-uncased-squad2)). This usually comes with a small trade-off in accuracy.
+
+You can reduce the work load on the Reader by instructing the Retriever to pass on less documents.
+This is done by setting the `top_k_retriever` parameter to a lower value.
+
+Making sure that your documents are shorter can also increase the speed of your system. You can split
+your documents into smaller chunks by using the `PreProcessor` (see [tutorial](https://haystack.deepset.ai/docs/latest/tutorial11md)).
+
+For more optimization suggestions, have a look at our [optimization page](https://haystack.deepset.ai/docs/latest/optimizationmd)
+and also our [blogs](https://medium.com/deepset-ai)
+
+##How do I use Haystack for my language?
+
+The components in Haystack, such as the `Retriever` or the `Reader`, are designed in a language agnostic way. However you may
+have to set certain parameters or load models pretrained for your language in order to get good performance out of Haystack.
+See our [languages page](https://haystack.deepset.ai/docs/latest/languagesmd) for more details.
+
+##How can I add metadata to my documents so that I can apply filters?
+
+When providing your documents in the input format (see [here](https://haystack.deepset.ai/docs/latest/documentstoremd#Input-Format))
+you can provide metadata information as a dictionary under the `meta` key. At query time, you can provide a `filters` argument
+(most likely through `Pipelines.run()`) that specifies the accepted values for a certain metadata field
+(for an example of what a `filters` dictionary might look like, please refer to [this example](https://haystack.deepset.ai/docs/latest/apiretrievermd#__init__))
+
+##How can I see predictions during evaluation?
+
+To see predictions during evaluation, you want to initialize the `EvalDocuments` or `EvalAnswers` with `debug=True`.
+This causes their `EvalDocuments.log` or `EvalAnswers.log` to be populated with a record of each prediction made.
+
+##How can I serve my Haystack model?
+
+Haystack models can be wrapped in a REST API. For basic details on how to set this up, please refer to this section
+on our [Github page](https://github.com/deepset-ai/haystack/blob/master/README.md#7-rest-api).
+More comprehensive documentation coming soon!
+
+##How can I interpret the confidence scores being returned by the Reader?
+
+The confidence scores are in the range of 0 and 1 and reflect how confident the model is in each prediction that it makes.
+Having a confidence score is particularly useful in cases where you need Haystack to work with a certain accuracy threshold.
+Many of our users have built systems where predictions below a certain confidence value are routed on to a fallback system.
+
+For more information on model confidence and how to tune it, please refer to [this section](https://haystack.deepset.ai/docs/latest/readermd#Confidence-Scores).
+
+##My documents aren't showing up in my DocumentStore even though I've called `DocumentStore.write_documents()`
+
+When indexing, retrieving or querying for documents from a DocumentStore, you can specify an `index` on which to perform this action.
+This can be specified in almost all methods of `DocumentStore` as well as `Retriever.retrieve()`.
+Ensure that you are performing these operations on the one index!
+Note that this also applies at evaluation where labels are written into their own separate DocumentStore index.
+
+##What is the difference between the FARMReader and the TransformersReader?
+
+In short, the FARMReader using a QA pipeline implementation that comes from our own
+[FARM framework](https://github.com/deepset-ai/FARM) that we can more easily update and also optimize for performance.
+By contrast, the TransformersReader uses a QA pipeline implementation that comes from HuggingFace's [Transformers](https://github.com/huggingface/transformers).
+See [this section](https://haystack.deepset.ai/docs/latest/readermd#Deeper-Dive-FARM-vs-Transformers)
+for a more details about their differences!
\ No newline at end of file
diff --git a/docs/v1.3.0/_src/usage/usage/generator.md b/docs/v1.3.0/_src/usage/usage/generator.md
new file mode 100644
index 0000000000..2409961b80
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/generator.md
@@ -0,0 +1,36 @@
+
+
+# Generator
+
+While extractive QA highlights the span of text that answers a query,
+generative QA can return a novel text answer that it has composed.
+The best current approaches, such as [Retriever-Augmented Generation](https://arxiv.org/abs/2005.11401) and [LFQA](https://yjernite.github.io/lfqa.html),
+can draw upon both the knowledge it gained during language model pretraining (parametric memory)
+as well as passages provided to it with a retriever (non-parametric memory).
+With the advent of Transformer based retrieval methods such as [Dense Passage Retrieval](https://arxiv.org/abs/2004.04906),
+retriever and generator can be trained concurrently from the one loss signal.
+
+
+
+**Tutorial**
+
+Checkout our tutorial notebooks for a guide on how to build your own generative QA system with RAG ([here](/docs/latest/tutorial7md))
+or with LFQA ([here](/docs/latest/tutorial12md)).
+
+
+
+Pros
+* More appropriately phrased answers
+* Able to synthesize information from different texts
+* Can draw on latent knowledge stored in language model
+
+Cons
+* Not easy to track what piece of information the generator is basing its response off of
+
diff --git a/docs/v1.3.0/_src/usage/usage/get_started.md b/docs/v1.3.0/_src/usage/usage/get_started.md
new file mode 100644
index 0000000000..c727fdafbf
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/get_started.md
@@ -0,0 +1,166 @@
+
+
+# Get Started
+
+## Installation
+
+
+
+
+
+Note: On Windows add the arg `-f https://download.pytorch.org/whl/torch_stable.html` to install PyTorch correctly
+
+## The Building Blocks of Haystack
+
+Here’s a sample of some Haystack code showing a question answering system using a retriever and a reader.
+For a working code example, check out our [starter tutorial](/docs/latest/tutorial1md).
+
+```python
+# DocumentStore: holds all your data
+document_store = ElasticsearchDocumentStore()
+
+# Clean & load your documents into the DocumentStore
+dicts = convert_files_to_dicts(doc_dir, clean_func=clean_wiki_text)
+document_store.write_documents(dicts)
+
+# Retriever: A Fast and simple algo to indentify the most promising candidate documents
+retriever = ElasticsearchRetriever(document_store)
+
+# Reader: Powerful but slower neural network trained for QA
+model_name = "deepset/roberta-base-squad2"
+reader = FARMReader(model_name)
+
+# Pipeline: Combines all the components
+pipe = ExtractiveQAPipeline(reader, retriever)
+
+# Voilà! Ask a question!
+question = "Who is the father of Sansa Stark?"
+prediction = pipe.run(query=question)
+print_answers(prediction)
+```
+
+## Loading Documents into the DocumentStore
+
+In Haystack, DocumentStores expect Documents in a dictionary format. They are loaded as follows:
+
+```python
+document_store = ElasticsearchDocumentStore()
+dicts = [
+ {
+ 'text': DOCUMENT_TEXT_HERE,
+ 'meta': {'name': DOCUMENT_NAME, ...}
+ }, ...
+]
+document_store.write_documents(dicts)
+```
+
+When we talk about Documents in Haystack, we are referring specifically to the individual blocks of text that are being held in the DocumentStore.
+You might want to use all the text in one file as a Document, or split it into multiple Documents.
+This splitting can have a big impact on speed and performance.
+
+
+
+
+
+
+
+The most straightforward way to install Haystack is through pip.
+ +```python +$ pip install farm-haystack +``` + +
++ +```python +$ pip install farm-haystack +``` + +
+
+
+
+
+
+If you’d like to run a specific, unreleased version of Haystack, or make edits to the way Haystack runs,
+you’ll want to install it using `git` and `pip --editable`.
+This clones a copy of the repo to a local directory and runs Haystack from there.
+ +```python +$ git clone https://github.com/deepset-ai/haystack.git +$ cd haystack +$ pip install --editable . +``` + +By default, this will give you the latest version of the master branch. Use regular git commands to switch between different branches and commits. +
++ +```python +$ git clone https://github.com/deepset-ai/haystack.git +$ cd haystack +$ pip install --editable . +``` + +By default, this will give you the latest version of the master branch. Use regular git commands to switch between different branches and commits. +
+
+**Tip:** If Haystack is running very slowly, you might want to try splitting your text into smaller Documents.
+If you want an improvement to performance, you might want to try concatenating text to make larger Documents.
+See [Optimization](/docs/latest/optimizationmd) for more details.
+
+
+
+
+## Running Search Queries
+
+There are many different flavours of search that can be created using Haystack.
+But to give just one example of what can be achieved, let's look more closely at
+an Open Domain Question Answering (ODQA) Pipeline.
+
+
+**Querying** in an ODQA system involves searching for an answer to a given question within the full document store.
+This process will:
+
+
+* make the Retriever filter for a small set of relevant candidate documents
+
+
+* get the Reader to process this set of candidate documents
+
+
+* return potential answers to the given question
+
+Usually, there are tight time constraints on querying and so it needs to be a lightweight operation.
+When documents are loaded, Haystack will precompute any of the results that might be useful at query time.
+
+In Haystack, querying is performed with a `Pipeline` object which connects the reader to the retriever.
+
+```python
+# Pipeline: Combines all the components
+pipe = ExtractiveQAPipeline(reader, retriever)
+
+# Voilà! Ask a question!
+question = "Who is the father of Sansa Stark?"
+prediction = pipe.run(query=question)
+print_answers(prediction)
+```
+
+When the query is complete, you can expect to see results that look something like this:
+
+```python
+[
+ { 'answer': 'Eddard',
+ 'context': 's Nymeria after a legendary warrior queen. She travels '
+ "with her father, Eddard, to King's Landing when he is made "
+ 'Hand of the King. Before she leaves,'
+ }, ...
+]
+```
+
+##Custom Search Pipelines
+
+Haystack providers many different building blocks for you to mix and match.
+They include:
+- Readers
+- Retrievers (sparse and dense)
+- DocumentStores
+- Summarizers
+- Generators
+- Translators
+
+These can all be combined in the configuration that you want.
+Have a look at our [Pipelines page](/docs/latest/pipelinesmd) to see what's possible!
\ No newline at end of file
diff --git a/docs/v1.3.0/_src/usage/usage/intro.md b/docs/v1.3.0/_src/usage/usage/intro.md
new file mode 100644
index 0000000000..71037d23c9
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/intro.md
@@ -0,0 +1,67 @@
+
+
+# What is Haystack?
+
+Haystack is an **open-source framework** for building **search systems** that work intelligently over large document collections.
+Recent advances in NLP have enabled the application of question answering, retrieval and summarization to real world settings
+and Haystack is designed to be the bridge between research and industry.
+
+
+* **Latest NLP models**: Utilize all transformer based models (BERT, RoBERTa, MiniLM, DPR ...) and smoothly switch when new ones get published
+
+* **Flexible databases**: Load data into and query from a range of databases such as Elasticsearch, Milvus, FAISS, SQL and more
+
+* **Scalability**: Production-ready deployments that scale to millions of documents
+
+* **End-to-End**: All tooling you need to implement, evaluate, improve and run a search system
+
+* **Domain adaptation**: Fine-tune models to your own domain & improve them continuously via user feedback
+
+## Retriever-Reader
+
+The most common system built with Haystack is the Retriever-Reader Pipeline which is designed to optimize for both
+speed and performance on the task of Open Domain Question Answering.
+In practice, this is a great back bone for creating a search system that can handle detailed full sentence queries.
+
+
+
+
+**Readers**, also known as Open-Domain QA systems in Machine Learning speak,
+are powerful models that do close analysis of documents and perform the core task of question answering.
+The Readers in Haystack are trained from the latest transformer based language models and can be significantly sped up using GPU acceleration.
+However, it is not currently feasible to use the Reader directly on large collection of documents.
+
+
+
+The **Retriever** assists the Reader by acting as a lightweight filter that reduces the number of documents that the Reader has to process.
+It does this by:
+
+
+* Scanning through all documents in the database
+
+
+* Quickly identifying the relevant and dismissing the irrelevant
+
+
+* Passing on only a small candidate set of documents to the Reader
+
+Current methods fall into one of the two categories:
+
+
+* sparse
+ * keyword based
+ * fast indexing and querying
+ * e.g. BM25
+
+
+* dense
+ * neural network based
+ * computationally heavy indexing but fast querying
+ * e.g. Dense Passage Retrieval
diff --git a/docs/v1.3.0/_src/usage/usage/knowledge_graph.md b/docs/v1.3.0/_src/usage/usage/knowledge_graph.md
new file mode 100644
index 0000000000..b3469f0f69
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/knowledge_graph.md
@@ -0,0 +1,108 @@
+
+
+# Question Answering on a Knowledge Graph
+
+Haystack allows loading and querying knowledge graphs. In particular, Haystack can:
+
+* Load an existing knowledge graph given as a .ttl file
+* Execute SPARQL queries on a knowledge graph
+* Execute text queries on the knowledge graph by translating them to SPARQL queries with the help of a pre-trained seq2seq model
+
+Haystack's knowledge graph functionalities are still in a very early stage. Thus, don't expect our [exemplary tutorial](https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial10_Knowledge_Graph.py) to work on your custom dataset out-of-the-box.
+Two classes implement the functionalities: GraphDBKnowledgeGraph and Text2SparqlRetriever.
+
+## GraphDBKnowledgeGraph
+
+GraphDBKnowledgeGraph is a triple store similar to Haystack's document stores. Currently, it is the only implementation of the BaseKnowledgeGraph class.
+GraphDBKnowledgeGraph runs on GraphDB. The licensing of GraphDB is rather complicated and it's more than unfortunate that GraphDB cannot be used right away in colab notebooks.
+On your local machine, you can start a GraphDB instance by running:
+
+```docker run -d -p 7200:7200 --name graphdb-instance-tutorial docker-registry.ontotext.com/graphdb-free:9.4.1-adoptopenjdk11```
+
+By default, GraphDBKnowledgeGraph connects to a GraphDB instance running on localhost at port 7200.
+Similar to Haystack's ElasticsearchDocumentStore, the only additional setting needed is an index name.
+(Note that GraphDB internally calls these indices repositories.)
+
+```kg = GraphDBKnowledgeGraph(index="tutorial_10_index")```
+
+Indices can be deleted and created with ```GraphDBKnowledgeGraph.delete_index()``` and ```GraphDBKnowledgeGraph.create_index(config_path)```.
+```config_path``` needs to point to a .ttl file that contains configuration settings (see [GraphDB documentation](https://graphdb.ontotext.com/documentation/free/configuring-a-repository.html#configure-a-repository-programmatically) for details or use the file from our [tutorial](https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial10_Knowledge_Graph.py)). It starts with something like:
+
+```
+#
+# Sesame configuration template for a GraphDB Free repository
+#
+@prefix rdfs:
+
+**Note**
+
+This feature will be implemented by [this PR.](https://github.com/deepset-ai/haystack/pull/1160)
+
+
+
+The PreProcessor's sentence tokenization is language specific.
+If you are using the PreProcessor on a language other than English,
+make sure to set the `language` argument when initializing it.
+
+##Retrievers
+
+The sparse retriever methods themselves(BM25, TF-IDF) are language agnostic.
+Their only requirement is that the text be split into words.
+The ElasticsearchDocumentStore relies on an analyzer to impose word boundaries,
+but also to handle punctuation, casing and stop words.
+
+The default analyzer is an English analyzer.
+While it can still work decently for a large range of languages,
+you will want to set it to your language's analyzer for optimal performance.
+In some cases, such as with Thai, the default analyzer is completely incompatible.
+See [this page](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html)
+for the full list of language specific analyzers.
+
+```python
+from haystack.document_store import ElasticsearchDocumentStore
+
+document_store = ElasticsearchDocumentStore(analyzer="thai")
+```
+
+The models used in dense retrievers are language specific.
+Be sure to check language of the model used in your EmbeddingRetriever.
+The default model that is loaded in the DensePassageRetriever is for English.
+
+We have created a [German DensePassageRetriever model](https://deepset.ai/germanquad) and know other teams who work on further languages.
+If you have a language model and a question answering dataset in your own language, you can also train a DPR model using Haystack!
+Below is a simplified example.
+See [our tutorial](/docs/latest/tutorial9md) and also the [API reference](/docs/latest/apiretrievermd#train) for `DensePassageRetriever.train()` for more details.
+
+```python
+from haystack.retriever import DensePassageRetriever
+
+dense_passage_retriever = DensePassageRetriever(document_store)
+dense_passage_retriever.train(self,
+ data_dir: str,
+ train_filename: str,
+ dev_filename: str = None,
+ test_filename: str = None,
+ batch_size: int = 16,
+ embed_title: bool = True,
+ num_hard_negatives: int = 1,
+ n_epochs: int = 3)
+```
+
+##Readers
+
+While models are comparatively more performant on English,
+thanks to a wealth of available English training data,
+there are a couple QA models that are directly usable in Haystack.
+
+
+
+
+
+We are the creators of the **German** model and you can find out more about it [here](https://deepset.ai/germanquad)
+
+The **French**, **Italian**, **Spanish**, **Portuguese** and **Chinese** models are monolingual language models trained on versions of the SQuAD dataset in their respective languages
+and their authors report decent results in their model cards
+(e.g. [here](https://huggingface.co/illuin/camembert-base-fquad) and [here](https://huggingface.co/mrm8488/bert-italian-finedtuned-squadv1-it-alfa)).
+There also exist Korean QA models on the model hub but their performance is not reported.
+
+The **zero-shot model** that is shown above is a **multilingual XLM-RoBERTa Large** that is trained on English SQuAD.
+It is clear, from our [evaluations](https://huggingface.co/deepset/xlm-roberta-large-squad2#model_card),
+that the model has been able to transfer some of its English QA capabilities to other languages,
+but still its performance lags behind that of the monolingual models.
+Nonetheless, if there is not yet a monolingual model for your language and it is one of the 100 supported by XLM-RoBERTa,
+this zero-shot model may serve as a decent first baseline.
+
diff --git a/docs/v1.3.0/_src/usage/usage/optimization.md b/docs/v1.3.0/_src/usage/usage/optimization.md
new file mode 100644
index 0000000000..816d560a80
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/optimization.md
@@ -0,0 +1,107 @@
+
+
+# Optimization
+
+## Speeding up Reader
+
+In most pipelines, the Reader will be the most computationally expensive component.
+If this is a step that you would like to speed up, you can opt for a smaller Reader model
+that can process more passages in the same amount of time.
+
+On our [benchmarks page](https://haystack.deepset.ai/bm/benchmarks), you will find a comparison of
+many of the common model architectures. While our default recommendation is RoBERTa,
+MiniLM offers much faster processing for only a minimal drop in accuracy.
+You can find the models that we've trained on [the HuggingFace Model Hub](https://huggingface.co/deepset)
+
+## GPU acceleration
+
+The transformer based models used in Haystack are designed to be run on a GPU enabled machine.
+The design of these models means that they greatly benefit from the parallel processing capabilities of graphics cards.
+If Haystack has successfully detected a graphics card, you should see these lines in your console output.
+
+```
+INFO - farm.utils - Using device: CUDA
+INFO - farm.utils - Number of GPUs: 1
+```
+
+You can track the work load on your CUDA enabled Nvidia GPU by tracking the output of `nvidia-smi -l` on the command line
+while your Haystack program is running.
+
+## Document Length
+
+Document length has a very direct impact on the speed of the Reader
+which is why we recommend using the `PreProcessor` class to clean and split your documents.
+**If you halve the length of your documents, you will halve the workload placed onto your Reader.**
+
+For **sparse retrievers**, very long documents pose a challenge since the signal of the relevant section of text
+can get washed out by the rest of the document.
+To get a good balance between Reader speed and Retriever performance, we split documents to a maximum of 500 words.
+If there is no Reader in the pipeline following the Retriever, we recommend that **documents be no longer than 10,000 words**.
+
+**Dense retrievers** are limited in the length of text that they can read in one pass.
+As such, it is important that documents are not longer than the dense retriever's maximum input length.
+By default, Haystack's DensePassageRetriever model has a maximum length of 256 tokens.
+As such, we recommend that documents contain significantly less words.
+We have found decent performance with **documents around 100 words long**.
+
+## Respecting Sentence Boundaries
+
+When splitting documents, it is generally not a good idea to let document boundaries fall in the middle of sentences.
+Doing so means that each document will contain incomplete sentence fragments
+which maybe be hard for both retriever and reader to interpret.
+It is therefore recommended to set `split_respect_sentence_boundary=True` when initializing your `PreProcessor`.
+
+## Choosing the Right top-k Values
+
+The `top-k` parameter in both the `Retriever` and `Reader` determine how many results they return.
+More specifically, `Retriever` `top-k` dictates how many retrieved documents are passed on to the next stage,
+while `Reader` `top-k` determines how many answer candidates to show.
+
+In our experiments, we have found that **`Retriever` `top_k=10`
+gives decent overall performance** and so we have set this as the default in Haystack.
+
+The choice of `Retriever` `top-k` is a trade-off between speed and accuracy,
+especially when there is a `Reader` in the pipeline.
+Setting it higher means passing more documents to the `Reader`,
+thus reducing the chance that the answer-containing passage is missed.
+However, passing more documents to the `Reader` will create a larger workload for the component.
+
+These parameters can easily be tweaked as follows if using a `Finder`:
+``` python
+answers = pipeline.run(query="What did Einstein work on?", top_k_retriever=10, top_k_reader=5)
+```
+or like this if directly calling the `Retriever`:
+``` python
+retrieved_docs = retriever.retrieve(top_k=10)
+```
+
+## Metadata Filtering
+
+Metadata can be attached to the documents which you index into your DocumentStore (see the input data format [here](/docs/latest/retrievermd)).
+At query time, you can apply filters based on this metadata to limit the scope of your search and ensure your answers
+come from a specific slice of your data.
+
+For example, if you have a set of annual reports from various companies,
+you may want to perform a search on just a specific year, or on a small selection of companies.
+This can reduce the work load of the retriever and also ensure that you get more relevant results.
+
+Filters are applied via the `filters` argument of the `Retriever` class. In practice, this argument will probably
+be passed into the `Pipeline.run()` call, which will then route it on to the `Retriever` class
+(see our the Arguments on the [Pipelines page](/docs/latest/pipelinesmd) for an explanation).
+
+```python
+pipeline.run(
+ query="Why did the revenue increase?",
+ filters={
+ "years": ["2019"],
+ "companies": ["BMW", "Mercedes"]
+ }
+)
+```
diff --git a/docs/v1.3.0/_src/usage/usage/pipelines.md b/docs/v1.3.0/_src/usage/usage/pipelines.md
new file mode 100644
index 0000000000..e0aa93b4e4
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/pipelines.md
@@ -0,0 +1,188 @@
+
+
+# Pipelines
+
+### Flexibility powered by DAGs
+In order to build modern search pipelines, you need two things: powerful building blocks and an easy way to stick them together.
+The `Pipeline` class is exactly built for this purpose and enables many search scenarios beyond QA.
+The core idea is that you can build a Directed Acyclic Graph (DAG) where each node is one building block (Reader, Retriever, Generator ...).
+Here's a simple example for a standard Open-Domain QA Pipeline:
+
+```python
+from haystack import Pipeline
+
+p = Pipeline()
+p.add_node(component=retriever, name="ESRetriever1", inputs=["Query"])
+p.add_node(component=reader, name="QAReader", inputs=["ESRetriever1"])
+res = p.run(query="What did Einstein work on?")
+```
+
+You can **draw the DAG** to better inspect what you are building:
+```python
+p.draw(path="custom_pipe.png")
+```
+
+
+### Arguments
+
+Each node in a Pipeline defines the arguments the run() method accepts. The Pipeline class takes care of passing relevant
+arguments to the node. In addition to mandatory inputs like `query`, the `run()` accepts optional node parameters like
+`top_k` with the `params` argument. For instance, `params={"top_k": 5}` will set the `top_k` of all nodes as 5. To
+target params to a specific node, the node name can be explicitly specifie as `params={"Retriever": {"top_k": 5}}`.
+
+
+```python
+res = pipeline.run(
+ query="What did Einstein work on?",
+ params={"Retriever": {"top_k": 5}, "Reader": {"top_k": 3}}
+)
+```
+
+### YAML File Definitions
+
+For your convenience, there is also the option of defining and loading pipelines in YAML files.
+Having your pipeline available in a YAML is particularly useful when
+you move between experimentation and production environments.
+Just export the YAML from your notebook / IDE and import it into your production environment.
+It also helps with version control of pipelines, allows you to share your pipeline easily with colleagues,
+and simplifies the configuration of pipeline parameters in production.
+
+For example, you can define and save a simple Retriever Reader pipeline by saving the following to a file:
+
+```yaml
+version: '0.7'
+
+components: # define all the building-blocks for Pipeline
+- name: MyReader # custom-name for the component; helpful for visualization & debugging
+ type: FARMReader # Haystack Class name for the component
+ params:
+ no_ans_boost: -10
+ model_name_or_path: deepset/roberta-base-squad2
+- name: MyESRetriever
+ type: ElasticsearchRetriever
+ params:
+ document_store: MyDocumentStore # params can reference other components defined in the YAML
+ custom_query: null
+- name: MyDocumentStore
+ type: ElasticsearchDocumentStore
+ params:
+ index: haystack_test
+
+pipelines: # multiple Pipelines can be defined using the components from above
+- name: my_query_pipeline # a simple extractive-qa Pipeline
+ nodes:
+ - name: MyESRetriever
+ inputs: [Query]
+ - name: MyReader
+ inputs: [MyESRetriever]
+```
+
+To load, simply call:
+
+```python
+pipeline.load_from_yaml(Path("sample.yaml"))
+```
+
+For another example YAML config, check out [this file](https://github.com/deepset-ai/haystack/blob/master/rest_api/pipeline/pipelines.yaml).
+
+### Multiple retrievers
+You can now also use multiple Retrievers and join their results:
+```python
+p = Pipeline()
+p.add_node(component=es_retriever, name="ESRetriever", inputs=["Query"])
+p.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["Query"])
+p.add_node(component=JoinDocuments(join_mode="concatenate"), name="JoinResults", inputs=["ESRetriever", "DPRRetriever"])
+p.add_node(component=reader, name="QAReader", inputs=["JoinResults"])
+res = p.run(query="What did Einstein work on?", params={"ESRetriever": {"top_k": 1}, "DPRRetriever": {"top_k": 3}})
+```
+
+
+### Custom nodes
+It is easy to build custom nodes. Just respect the following requirements:
+
+1. Create a Class that inherits from `BaseComponent`.
+2. Add a `run()` method to your class with any parameters it needs to process the input. Ensure that the parameters are either passed with `params` to the pipeline or are returned by the preceding nodes.
+3. Do whatever you want within `run()` (e.g., reformatting the query).
+4. Return a tuple that contains your output data (for the next node) and the name of the outgoing edge `output_dict, "output_1`.
+5. Add a class attribute `outgoing_edges = 1` that defines your node's number of output options. You only need a higher number here if you have a decision node (see below).
+
+### Decision nodes
+Or you can add decision nodes where only one "branch" is executed afterwards. This allows, for example, to classify an incoming query and depending on the result routing it to different modules:
+
+```python
+ from haystack import BaseComponent, Pipeline
+
+ class QueryClassifier(BaseComponent):
+ outgoing_edges = 2
+
+ def run(self, query):
+ if "?" in query:
+ return {}, "output_1"
+
+ else:
+ return {}, "output_2"
+
+ pipe = Pipeline()
+ pipe.add_node(component=QueryClassifier(), name="QueryClassifier", inputs=["Query"])
+ pipe.add_node(component=es_retriever, name="ESRetriever", inputs=["QueryClassifier.output_1"])
+ pipe.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["QueryClassifier.output_2"])
+ pipe.add_node(component=JoinDocuments(join_mode="concatenate"), name="JoinResults",
+ inputs=["ESRetriever", "DPRRetriever"])
+ pipe.add_node(component=reader, name="QAReader", inputs=["JoinResults"])
+ res = p.run(query="What did Einstein work on?", params={"ESRetriever": {"top_k": 1}, "DPRRetriever": {"top_k": 3}})
+```
+
+### Evaluation nodes
+
+There are nodes in Haystack that are used to evaluate the performance of readers, retrievers and combine systems.
+To get hands on with this kind of node, have a look at the [evaluation tutorial](/docs/latest/tutorial5md).
+
+### Default Pipelines (replacing the "Finder")
+Last but not least, we added some "Default Pipelines" that allow you to run standard patterns with very few lines of code.
+This is replacing the `Finder` class which was deprecated with Haystack 0.6.0 .
+
+```
+from haystack.pipeline import DocumentSearchPipeline, ExtractiveQAPipeline, Pipeline, JoinDocuments
+
+# Extractive QA
+qa_pipe = ExtractiveQAPipeline(reader=reader, retriever=retriever)
+res = qa_pipe.run(query="When was Kant born?", params={"Retriever": {"top_k": 3}, "Reader": {"top_k": 5}})
+
+# Document Search
+doc_pipe = DocumentSearchPipeline(retriever=retriever)
+res = doc_pipe.run(query="Physics Einstein", params={"Retriever": {"top_k": 3}})
+
+# Generative QA
+doc_pipe = GenerativeQAPipeline(generator=rag_generator, retriever=retriever)
+res = doc_pipe.run(query="Physics Einstein", params={"Retriever": {"top_k": 3}})
+
+# FAQ based QA
+doc_pipe = FAQPipeline(retriever=retriever)
+res = doc_pipe.run(query="How can I change my address?", params={"Retriever": {"top_k": 3}})
+
+```
+So to migrate your QA system from the deprecated `Finder` to `ExtractiveQAPipeline` you'd need to:
+```
+# 1. Change import
+from haystack.pipeline import ExtractiveQAPipeline
+
+# 2. Replace the Finder
+qa_pipe = ExtractiveQAPipeline(reader=reader, retriever=retriever)
+
+# 3. Replace get_answers() with run()
+res = qa_pipe.run(query="When was Kant born?", top_k_retriever=3, top_k_reader=5)
+
+# 4. Access your results from ["documents"] rather than ["answers"]
+print(res["documents"])
+```
+See also the [Pipelines API documentation](/docs/latest/apipipelinesmd) for more details.
+
+We plan many more features around the new pipelines incl. parallelized execution, distributed execution, dry runs - so stay tuned ...
+
diff --git a/docs/v1.3.0/_src/usage/usage/preprocessing.md b/docs/v1.3.0/_src/usage/usage/preprocessing.md
new file mode 100644
index 0000000000..a285cc44ce
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/preprocessing.md
@@ -0,0 +1,144 @@
+
+
+# Preprocessing
+
+Haystack includes a suite of tools to:
+
+* extract text from different file types,
+* normalize white space
+* split text into smaller pieces to optimize retrieval
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("deepset/gelectra-large-germanquad")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("illuin/camembert-base-fquad")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("mrm8488/bert-italian-finedtuned-squadv1-it-alfa")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("uer/roberta-base-chinese-extractive-qa")
+# or
+reader = FARMReader("wptoux/albert-chinese-large-qa")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es")
+# or
+reader = FARMReader("mrm8488/bert-base-spanish-wwm-cased-finetuned-spa-squad2-es")
+```
+
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("pierreguillou/bert-base-cased-squad-v1.1-portuguese")
+# or
+reader = FARMReader("pucpr/bioBERTpt-squad-v1.1-portuguese")
+
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("deepset/xlm-roberta-large-squad2")
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("deepset/gelectra-large-germanquad")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("illuin/camembert-base-fquad")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("mrm8488/bert-italian-finedtuned-squadv1-it-alfa")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("uer/roberta-base-chinese-extractive-qa")
+# or
+reader = TransformersReader("wptoux/albert-chinese-large-qa")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es")
+# or
+reader = TransformersReader("mrm8488/bert-base-spanish-wwm-cased-finetuned-spa-squad2-es")
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("pierreguillou/bert-base-cased-squad-v1.1-portuguese")
+# or
+reader = TransformersReader("pucpr/bioBERTpt-squad-v1.1-portuguese")
+
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("deepset/xlm-roberta-large-squad2")
+```
+
+
+
+
+Check out our [preprocessing tutorial](/docs/latest/tutorial8md) if you'd like to start working with code examples already!
+
+
+
+These data preprocessing steps can have a big impact on the systems performance
+and effective handling of data is key to getting the most out of Haystack.
+
+The Document Store expects its inputs to come in the following format.
+The sections below will show you all the tools you'll need to ready your data for storing.
+
+```python
+docs = [
+ {
+ 'content': DOCUMENT_TEXT_HERE,
+ 'meta': {'name': DOCUMENT_NAME, ...}
+ }, ...
+]
+```
+
+## File Conversion
+
+There are a range of different file converters in Haystack that
+can extract text from files and cast them into the unified dictionary format shown above.
+Haystack features support for txt, pdf and docx files and there is even a converter that leverages Apache Tika.
+Please refer to [the API docs](/docs/latest/file_convertersmd) to see which converter best suits you.
+
+
+
+
+
+## Web Crawler
+
+In Haystack, you will find a web crawler that will help you scrape text from websites and save it to file.
+See the [API documentation](https://haystack.deepset.ai/docs/latest/apicrawlermd) for more details.
+
+```python
+from haystack.connector import Crawler
+
+crawler = Crawler()
+docs = crawler.crawl(urls=["https://haystack.deepset.ai/docs/latest/get_startedmd"],
+ output_dir="crawled_files",
+ filter_urls= ["haystack\.deepset\.ai\/docs\/"])
+```
+
+## PreProcessor
+
+While each of the above conversion methods produce documents that are already in the format expected by the Document Store,
+it is recommended that they are further processed in order to ensure optimal Retriever and Reader performance.
+The `PreProcessor` takes one of the documents created by the converter as input,
+performs various cleaning steps and splits them into multiple smaller documents.
+
+For suggestions on how best to split your documents, see [Optimization](/docs/latest/optimizationmd)
+
+```python
+from haystack.preprocessor import PreProcessor
+
+doc = converter.convert(file_path=file, meta=None)
+processor = PreProcessor(
+ clean_empty_lines=True,
+ clean_whitespace=True,
+ clean_header_footer=True,
+ split_by="word",
+ split_length=200,
+ split_respect_sentence_boundary=True,
+ split_overlap=0
+)
+docs = processor.process(doc)
+```
+
+* `clean_empty_lines` will normalize 3 or more consecutive empty lines to be just a two empty lines
+* `clean_whitespace` will remove any whitespace at the beginning or end of each line in the text
+* `clean_header_footer` will remove any long header or footer texts that are repeated on each page
+* `split_by` determines what unit the document is split by: `'word'`, `'sentence'` or `'passage'`
+* `split_length` sets a maximum number of `'word'`, `'sentence'` or `'passage'` units per output document
+* `split_respect_sentence_boundary` ensures that document boundaries do not fall in the middle of sentences
+* `split_overlap` sets the amount of overlap between two adjacent documents after a split. Setting this to a positive number essentially enables the sliding window approach.
+
diff --git a/docs/v1.3.0/_src/usage/usage/query_classifier.md b/docs/v1.3.0/_src/usage/usage/query_classifier.md
new file mode 100644
index 0000000000..4673e914d4
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/query_classifier.md
@@ -0,0 +1,188 @@
+
+
+# Query Classifier
+
+Queries come in all shapes and forms. A keyword-based search differs from a question posed in natural language. In Haystack, we can account for these differences by integrating a special node into our QA pipeline: the query classifier.
+
+A query classifier puts each incoming query into one of two predefined classes, and routes it to the appropriate section of the pipeline.
+Haystack comes with classifiers to distinguish between the three most common query types (Keywords, Question, Statement) and allows two different types of models (SKlearn and Transformer).
+
+Using a query classifier can potentially yield the following benefits:
+
+* Getting better search results (e.g. by routing only proper questions to DPR / QA branches and not keyword queries)
+* Less GPU costs (e.g. if 50% of your traffic is only keyword queries you could just use elastic here and save the GPU resources for the other 50% of traffic with semantic queries)
+
+
+### Common Query types
+
+#### 1. Keyword Queries:
+Such queries don't have semantic meaning, merely consist of keywords and the order of words does not matter:
+* arya stark father
+* jon snow country
+* arya stark younger brothers
+
+#### 2. Questions (Interrogative Queries):
+In such queries users ask a question in a complete, "natural" sentence. Regardless of the presence of "?" in the query the goal here is to detect the intent of the user whether any question is asked or not in the query:
+
+* who is the father of arya stark?
+* which country was jon snow filmed in
+* who are the younger brothers of arya stark?
+
+#### 3. Statements (Declarative Queries):
+Such queries consist also of a regular, natural sentence with semantic relations between the words. However, they are rather a statement than a question:
+
+* Arya stark was a daughter of a lord.
+* Show countries that Jon snow was filmed in.
+* List all brothers of Arya.
+
+### Usage standalone: Try a Query Classifier
+To test how a query classifier works before integrating it into a pipeline, you can run it just as an individual component:
+
+```python
+from haystack.pipeline import TransformersQueryClassifier
+
+queries = ["Arya Stark father","Jon Snow UK",
+ "who is the father of arya stark?","Which country was jon snow filmed in?"]
+
+question_classifier = TransformersQueryClassifier(model_name_or_path="shahrukhx01/bert-mini-finetune-question-detection")
+# Or Sklearn based:
+
+for query in queries:
+ result = question_classifier.run(query=query)
+ if result[1] == "output_1":
+ category = "question"
+ else:
+ category = "keywords"
+
+ print(f"Query: {query}, raw_output: {result}, class: {category}")
+
+# Returns:
+# Query: Arya Stark father, raw_output: ({'query': 'Arya Stark father'}, 'output_2'), class: keywords
+# Query: Jon Snow UK, raw_output: ({'query': 'Jon Snow UK'}, 'output_2'), class: keywords
+# Query: who is the father of arya stark?, raw_output: ({'query': 'who is the father of arya stark?'}, 'output_1'), class: question
+# Query: Which country was jon snow filmed in?, raw_output: ({'query': 'Which country was jon snow filmed in?'}, 'output_1'), class: question
+
+```
+Note how the node returns two objects: the query (e.g.'Arya Stark father') and the name of the output edge (e.g. "output_2"). This information can be leveraged in a pipeline for routing the query to the next node.
+
+### Usage in a pipeline: Use different retrievers depending on the query type
+
+You can use a Query Classifier within a pipeline as a "decision node". Depending on the output of the classifier other parts of the pipeline will be executed. For example, we can route keyword queries to an ElasticsearchRetriever and semantic queries (questions/statements) to DPR.
+
+
+
+Below, we define a pipeline with a `TransformersQueryClassifier` that routes questions/statements to the node's `output_1` and keyword queries to `output_2`. We leverage this structure in the pipeline by connecting the DPRRetriever to `QueryClassifier.output_1` and the ESRetriever to `QueryClassifier.output_2`.
+
+```python
+from haystack.pipeline import TransformersQueryClassifier, Pipeline
+from haystack.utils import print_answers
+
+query_classifier = TransformersQueryClassifier(model_name_or_path="shahrukhx01/bert-mini-finetune-question-detection")
+
+pipe = Pipeline()
+pipe.add_node(component=query_classifier, name="QueryClassifier", inputs=["Query"])
+pipe.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["QueryClassifier.output_1"])
+pipe.add_node(component=es_retriever, name="ESRetriever", inputs=["QueryClassifier.output_2"])
+
+# Pass a question -> run DPR
+res_1 = pipe.run(
+ query="Who is the father of Arya Stark?",
+ top_k_retriever=10
+)
+
+# Pass keywords -> run the ElasticsearchRetriever
+res_2 = pipe.run(
+ query="arya stark father",
+ top_k_retriever=10
+)
+
+```
+### Usage in a pipeline: Run QA only on proper questions
+
+If you add QA to an existing search system, it can make sense to only use it for real questions that come in and keep a basic document search with elasticsearch for the remaining keyword queries. You can use a Query Classifier to build such a hybrid pipeline:
+
+```python
+haystack.pipeline import TransformersQueryClassifier, Pipeline
+from haystack.utils import print_answers
+
+query_classifier = TransformersQueryClassifier(model_name_or_path="shahrukhx01/question-vs-statement-classifier")
+
+pipe = Pipeline()
+pipe.add_node(component=query_classifier, name="QueryClassifier", inputs=["Query"])
+pipe.add_node(component=dpr_retriever, name="DPRRetriever", inputs=["QueryClassifier.output_1"])
+pipe.add_node(component=es_retriever, name="ESRetriever", inputs=["QueryClassifier.output_2"])
+pipe.add_node(component=reader, name="QAReader", inputs=["DPRRetriever"])
+
+# Pass a question -> run DPR + QA -> return answers
+res_1 = pipe.run(
+ query="Who is the father of Arya Stark?",
+ top_k_retriever=10
+)
+
+# Pass keywords -> run only ElasticsearchRetriever -> return docs
+res_2 = pipe.run(
+ query="arya stark father",
+ top_k_retriever=10
+)
+
+```
+
+
+### Which models are available?
+The transformer classifier is more accurate than the SkLearn classifier as it can use the context and order of words. However, it requires more memory and most probably GPU for faster inference. You can mitigate those down sides by choosing a very small transformer model. The default models we trained are using a mini BERT architecture which is only about `50 MB` in size and allows relatively fast inference on CPU.
+
+#### Transformers
+Pass your own `Transformer` binary classification model from file/huggingface or use one of the following pretrained ones hosted on Huggingface:
+1) Keywords vs. Questions/Statements (Default)
+
+ ```python
+ TransformersQueryClassifier(model_name_or_path="shahrukhx01/bert-mini-finetune-question-detection")
+ # output_1 => question/statement
+ # output_2 => keyword query
+ ```
+
+ [Readme](https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier/readme.txt)
+
+
+2) Questions vs. Statements
+ ```python
+ TransformersQueryClassifier(model_name_or_path="shahrukhx01/question-vs-statement-classifier")
+ # output_1 => question
+ # output_2 => statement
+ ```
+
+ [Readme](https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier_statements/readme.txt)
+
+
+#### Sklearn
+Pass your own `Sklearn` binary classification model or use one of the following pretrained Gradient boosting models:
+
+1) Keywords vs. Questions/Statements (Default)
+
+ ```python
+ SklearnQueryClassifier(query_classifier = "https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier/model.pickle",
+ query_vectorizer = "https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier/vectorizer.pickle")
+
+ # output_1 => question/statement
+ # output_2 => keyword query
+ ```
+ [Readme](https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier/readme.txt)
+
+
+2) Questions vs. Statements
+
+ ```python
+ SklearnQueryClassifier(query_classifier = "https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier_statements/model.pickle",
+ query_vectorizer = "https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier_statements/vectorizer.pickle")
+
+ output_1 => question
+ output_2 => statement
+ ```
+ [Readme](https://ext-models-haystack.s3.eu-central-1.amazonaws.com/gradboost_query_classifier_statements/readme.txt)
diff --git a/docs/v1.3.0/_src/usage/usage/question_generator.md b/docs/v1.3.0/_src/usage/usage/question_generator.md
new file mode 100644
index 0000000000..81ae731261
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/question_generator.md
@@ -0,0 +1,45 @@
+
+
+# Question Generator
+
+
+
+
+
+
+
+
+```python
+from haystack.file_converter import PDFToTextConverter
+
+converter = PDFToTextConverter(remove_numeric_tables=True, valid_languages=["de","en"])
+doc = converter.convert(file_path=file, meta=None)
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.file_converter import DocxToTextConverter
+
+converter = DocxToTextConverter(remove_numeric_tables=True, valid_languages=["de","en"])
+doc = converter.convert(file_path=file, meta=None)
+```
+
+
+
+
+
+
+
+
+
+
+Haystack also has a `convert_files_to_dicts()` utility function that will convert
+all txt or pdf files in a given folder into this dictionary format.
+
+```python
+from haystack.preprocessor.utils import convert_files_to_dicts
+
+docs = convert_files_to_dicts(dir_path=doc_dir)
+```
+
+
+
+
+**Running examples**
+
+Have a look at our [tutorial notebook](/docs/latest/tutorial13md))) if you'd like to start trying out Question Generation straight away!
+
+
+
+The Question Generation module is used to generate SQuAD style questions on a given document.
+
+This module is useful when it comes to labelling in a new domain. It can be used to generate questions quickly for an
+annotator to answer. If used in conjunction with a trained Reader model, you can automatically generate question answer
+pairs. High impact annotations can then be created if a human annotator looks over these pairs and corrects the incorrect predictions.
+
+Question generation is also a good way to make large documents more navigable. Generated questions can
+quickly give the user a sense of what information is contained within the document, thus acting as a kind of summarization.
+
+To initialize a question generator, simply call:
+
+```python
+from haystack.question_generator import QuestionGenerator
+
+question_generator = QuestionGenerator()
+```
+
+This loads the [`valhalla/t5-base-e2e-qg`](https://huggingface.co/valhalla/t5-base-e2e-qg) model by default which is a T5 model trained on SQuAD for question generation.
+
+To run the node in isolation, simply use the `generate()` method:
+
+```python
+result = question_generator.generate(text="Nirvana was an American rock band formed in Aberdeen, Washington in 1987.")
+```
+
+Otherwise, the node can be used in a pipeline where its `run()` method will called.
diff --git a/docs/v1.3.0/_src/usage/usage/ranker.md b/docs/v1.3.0/_src/usage/usage/ranker.md
new file mode 100644
index 0000000000..e829dcfbfd
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/ranker.md
@@ -0,0 +1,55 @@
+
+
+# Ranker
+
+There are pure "semantic document search" use cases that do not need question answering functionality but only document ranking.
+While the [Retriever](/docs/latest/retrievermd) is a perfect fit for document retrieval, we can further improve its results with the Ranker.
+For example, BM25 (sparse retriever) does not take into account semantics of the documents and the query but only their keywords.
+The Ranker can re-rank the results of the retriever step by taking semantics into account.
+Similar to the Reader, it is based on the latest language models.
+Instead of returning answers, it returns documents in re-ranked order.
+
+Without a Ranker and its re-ranking step, the querying process is faster but the query results might be of lower quality.
+If you want to do "semantic document search" instead of a question answering, try first with a Retriever only.
+In case the semantic similarity of the query and the resulting documents is low, add a Ranker.
+
+Note that a Ranker needs to be initialised with a model trained on a text pair classification task.
+You can train the model also with the train() method of the Ranker.
+Alternatively, [this example](https://github.com/deepset-ai/FARM/blob/master/examples/text_pair_classification.py) shows how to train a text pair classification model in FARM.
+
+
+## FARMRanker
+
+### Description
+
+The FARMRanker consists of a Transformer-based model for document re-ranking using the TextPairClassifier of [FARM](https://github.com/deepset-ai/FARM).
+Given a text pair of query and passage, the TextPairClassifier either predicts label "1" if the pair is similar or label "0" if they are dissimilar (accompanied with a probability).
+While the underlying model can vary (BERT, Roberta, DistilBERT, ...), the interface remains the same.
+With a FARMRanker, you can:
+* Directly get predictions (re-ranked version of the supplied list of Document) via predict() if supplying a pre-trained model
+* Take a plain language model (e.g. `bert-base-cased`) and train it for TextPairClassification via train()
+
+### Initialisation
+
+```python
+from haystack.document_store import ElasticsearchDocumentStore
+from haystack.retriever import ElasticsearchRetriever
+from haystack.ranker import FARMRanker
+from haystack import Pipeline
+
+document_store = ElasticsearchDocumentStore()
+...
+retriever = ElasticsearchRetriever(document_store)
+ranker = FARMRanker(model_name_or_path="saved_models/roberta-base-asnq-binary")
+...
+p = Pipeline()
+p.add_node(component=retriever, name="ESRetriever", inputs=["Query"])
+p.add_node(component=ranker, name="Ranker", inputs=["ESRetriever"])
+```
diff --git a/docs/v1.3.0/_src/usage/usage/reader.md b/docs/v1.3.0/_src/usage/usage/reader.md
new file mode 100644
index 0000000000..2cd26d352a
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/reader.md
@@ -0,0 +1,386 @@
+
+
+# Reader
+
+The Reader, also known as Open-Domain QA systems in Machine Learning speak,
+is the core component that enables Haystack to find the answers that you need.
+Haystack’s Readers are:
+
+
+* built on the latest transformer based language models
+
+
+* strong in their grasp of semantics
+
+
+* sensitive to syntactic structure
+
+
+* state-of-the-art in QA tasks like SQuAD and Natural Questions
+
+
+
+
+
+While these models can work on CPU, it is recommended that they are run using GPUs to keep query times low.
+
+## Choosing the Right Model
+
+In Haystack, you can start using pretrained QA models simply by providing its HuggingFace Model Hub name to the Reader.
+The loading of model weights is handled by Haystack,
+and you have the option of using the QA pipeline from deepset FARM or HuggingFace Transformers (see FARM vs Transformers for details).
+
+Currently, there are a lot of different models out there and it can be rather overwhelming trying to pick the one that fits your use case.
+To get you started, we have a few recommendations for you to try out.
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+model = "deepset/roberta-base-squad2"
+reader = FARMReader(model, use_gpu=True)
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+model = "deepset/roberta-base-squad2"
+reader = TransformersReader(model, use_gpu=1)
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+**An optimised variant of BERT and a great starting point.**
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("deepset/roberta-base-squad2")
+```
+
+* **Pro**: Strong all round model
+
+* **Con**: There are other models that are either faster or more accurate
+
+
+
+
+
+
+
+
+
+**A cleverly distilled model that sacrifices a little accuracy for speed.**
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("deepset/minilm-uncased-squad2")
+```
+
+* **Pro**: Inference speed up to 50% faster than BERT base
+
+* **Con**: Still doesn’t match the best base sized models in accuracy
+
+
+
+
+
+
+
+
+
+**Large, powerful, SotA model.**
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader("ahotrod/albert_xxlargev1_squad2_512")
+```
+
+* **Pro**: Better accuracy than any other open source model in QA
+
+* **Con**: The computational power needed make it impractical for most use cases
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+**An optimised variant of BERT and a great starting point.**
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("deepset/roberta-base-squad2")
+```
+
+* **Pro**: Strong all round model
+
+* **Con**: There are other models that are either faster or more accurate
+
+
+
+
+
+
+
+
+
+**A cleverly distilled model that sacrifices a little accuracy for speed.**
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("deepset/minilm-uncased-squad2")
+```
+
+* **Pro**: Inference speed up to 50% faster than BERT base
+
+* **Con**: Still doesn’t match the best base sized models in accuracy
+
+
+
+
+
+
+
+
+
+**Large, powerful, SotA model.**
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader("ahotrod/albert_xxlargev1_squad2_512")
+```
+
+* **Pro**: Better accuracy than any other open source model in QA
+
+* **Con**: The computational power needed make it impractical for most use cases
+
+
+
+
+**Recommendations:**
+
+**All-rounder**: In the class of base sized models trained on SQuAD, **RoBERTa** has shown better performance than BERT
+and can be capably handled by any machine equipped with a single NVidia V100 GPU.
+We recommend this as the starting point for anyone wanting to create a performant and computationally reasonable instance of Haystack.
+
+**Built for Speed**: If speed and GPU memory are more of a priority to you than accuracy,
+you should try the MiniLM model.
+It is a smaller model that is trained to mimic larger models through the distillation process,
+and it outperforms the BERT base on SQuAD even though it is about 40% smaller.
+
+
+**State of the Art Accuracy**: For most, **ALBERT XXL** will be too large to feasibly work with.
+But if performance is your sole concern, and you have the computational resources,
+you might like to try ALBERT XXL which has set SoTA performance on SQuAD 2.0.
+
+
+
+
+
+
+## Confidence Scores
+
+When printing the full results of a Reader,
+you will see that each prediction is accompanied
+by a value in the range of 0 to 1 reflecting the model's confidence in that prediction
+
+In the output of `print_answers()`, you will find the model confidence in dictionary key called `confidence`.
+
+```python
+from haystack.utils import print_answers
+
+print_answers(prediction, details="all")
+```
+
+```python
+{
+ 'answers': [
+ { 'answer': 'Eddard',
+ 'context': 's Nymeria after a legendary warrior queen. '
+ 'She travels with her father, Eddard, to '
+ "King's Landing when he is made Hand of the "
+ 'King. Before she leaves,',
+ 'confidence': 0.9899835586547852,
+ ...
+ },
+ ]
+}
+```
+
+In order to align this probability score with the model's accuracy, finetuning needs to be performed
+on a specific dataset.
+To this end, the reader has a method `calibrate_confidence_scores(document_store, device, label_index, doc_index, label_origin)`.
+The parameters of this method are the same as for the `eval()` method because the calibration of confidence scores is performed on a dataset that comes with gold labels.
+The calibration calls the `eval()` method internally and therefore needs a DocumentStore containing labeled questions and evaluation documents.
+
+Have a look at this [FARM tutorial](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering_confidence.py)
+to see how to compare calibrated confidence scores with uncalibrated confidence scores within FARM.
+Note that a finetuned confidence score is specific to the domain that it is finetuned on.
+There is no guarantee that this performance can transfer to a new domain.
+
+Having a confidence score is particularly useful in cases where you need Haystack to work with a certain accuracy threshold.
+Many of our users have built systems where predictions below a certain confidence value are routed
+on to a fallback system.
+
+
+
+## Deeper Dive: FARM vs Transformers
+
+Apart from the **model weights**, Haystack Readers contain all the components found in end-to-end open domain QA systems.
+This includes **tokenization**, **embedding computation**, **span prediction** and **candidate aggregation**.
+While the handling of model weights is the same between the FARM and Transformers libraries, their QA pipelines differ in some ways.
+The major points are:
+
+
+* The **TransformersReader** will sometimes predict the same span twice while duplicates are removed in the **FARMReader**
+
+
+* The **FARMReader** currently uses the tokenizers from the HuggingFace Transformers library while the **TransformersReader** uses the tokenizers from the HuggingFace Tokenizers library
+
+
+* Start and end logits are normalized per passage and multiplied in the **TransformersReader** while they are summed and not normalised in the **FARMReader**
+
+If you’re interested in the finer details of these points, have a look at [this](https://github.com/deepset-ai/haystack/issues/248#issuecomment-661977237) GitHub comment.
+
+We see value in maintaining both kinds of Readers since Transformers is a very familiar library to many of Haystack’s users
+but we at deepset can more easily update and optimise the FARM pipeline for speed and performance.
+
+
+Haystack also has a close integration with FARM which means that you can further fine-tune your Readers on labelled data using a FARMReader.
+See our tutorials for an end-to-end example or below for a shortened example.
+
+```python
+from haystack.reader import FARMReader
+
+# Initialise Reader
+model = "deepset/roberta-base-squad2"
+reader = FARMReader(model)
+
+# Perform finetuning
+train_data = "PATH/TO_YOUR/TRAIN_DATA"
+train_filename = "train.json"
+save_dir = "finetuned_model"
+reader.train(train_data, train_filename, save_dir=save_dir)
+
+# Load
+finetuned_reader = FARMReader(save_dir)
+```
+
+## Deeper Dive: From Language Model to Haystack Reader
+
+Language models form the core of most modern NLP systems and that includes the Readers in Haystack.
+They build a general understanding of language when performing training tasks such as Masked Language Modeling or Replaced Token Detection
+on large amounts of text.
+Well trained language models capture the word distribution in one or more languages
+but more importantly, convert input text into a set of word vectors that capture elements of syntax and semantics.
+
+In order to convert a language model into a Reader model, it needs first to be trained on a Question Answering dataset.
+To do so requires the addition of a question answering prediction head on top of the language model.
+The task can be thought of as a token classification task where every input token is assigned a probability of being
+either the start or end token of the correct answer.
+In cases where the answer is not contained within the passage, the prediction head is also expected to return a `no_answer` prediction.
+
+
+Since language models are limited in the number of tokens which they can process in a single forward pass,
+a sliding window mechanism is implemented to handle variable length documents.
+This functions by slicing the document into overlapping passages of (approximately) `max_seq_length`
+that are each offset by `doc_stride` number of tokens.
+These can be set when the Reader is initialized.
+
+
+
+
+
+Predictions are made on each individual passage and the process of aggregation picks the best candidates across all passages.
+If you’d like to learn more about what is happening behind the scenes, have a look at [this](https://medium.com/deepset-ai/modern-question-answering-systems-explained-4d0913744097) article.
diff --git a/docs/v1.3.0/_src/usage/usage/retriever.md b/docs/v1.3.0/_src/usage/usage/retriever.md
new file mode 100644
index 0000000000..94bc1401fe
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/retriever.md
@@ -0,0 +1,281 @@
+
+
+# Retriever
+
+The Retriever is a lightweight filter that can quickly go through the full document store and pass on a set of candidate documents that are relevant to the query.
+When used in combination with a Reader, it is a tool for sifting out the obvious negative cases, saving the Reader from doing more work than it needs to and speeding up the querying process.
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import FARMReader
+
+reader = FARMReader(... max_seq_len=384, doc_stride=128 ...)
+```
+
+
+
+
+
+
+
+
+
+```python
+from haystack.reader import TransformersReader
+
+reader = TransformersReader(... max_seq_len=384, doc_stride=128 ...)
+```
+
+
+
+
+**Recommendations**
+
+* BM25 (sparse)
+
+* Dense Passage Retrieval (dense)
+
+
+
+
+
+Note that not all Retrievers can be paired with every DocumentStore.
+Here are the combinations which are supported:
+
+| | Memory | Elasticsearch | SQL | FAISS | Milvus |
+| --- | --- | --- | ---- | ---- | ---- |
+| BM25 | N | Y | N | N | N |
+| TF-IDF | Y | Y | Y | N | N |
+| Embedding | Y | Y | N | Y | Y |
+| DPR | Y | Y | N | Y | Y |
+
+See [Optimization](/docs/latest/optimizationmd) for suggestions on how to choose top-k values.
+
+
+## TF-IDF
+
+### Description
+
+TF-IDF is a commonly used baseline for information retrieval that exploits two key intuitions:
+
+
+* documents that have more lexical overlap with the query are more likely to be relevant
+
+* words that occur in fewer documents are more significant than words that occur in many documents
+
+Given a query, a tf-idf score is computed for each document as follows:
+
+```python
+score = tf * idf
+```
+
+Where:
+
+
+* `tf` is how many times words in the query occur in that document.
+
+
+* `idf` is the inverse of the fraction of documents containing the word.
+
+In practice, both terms are usually log normalised.
+
+### Initialisation
+
+```python
+from haystack.document_store import InMemoryDocumentStore
+from haystack.retriever.sparse import TfidfRetriever
+from haystack.pipeline import ExtractiveQAPipeline
+
+document_store = InMemoryDocumentStore()
+...
+retriever = TfidfRetriever(document_store)
+...
+p = ExtractiveQAPipeline(reader, retriever)
+```
+
+## BM25 (Recommended)
+
+### Description
+
+BM25 is a variant of TF-IDF that we recommend you use if you are looking for a retrieval method that does not need a neural network for indexing.
+It improves upon its predecessor in two main aspects:
+
+
+* It saturates `tf` after a set number of occurrences of the given term in the document
+
+
+* It normalises by document length so that short documents are favoured over long documents if they have the same amount of word overlap with the query
+
+### Initialisation
+
+```python
+from haystack.document_store import ElasticsearchDocumentStore
+from haystack.retriever import ElasticsearchRetriever
+from haystack.pipeline import ExtractiveQAPipeline
+
+document_store = ElasticsearchDocumentStore()
+...
+retriever = ElasticsearchRetriever(document_store)
+...
+p = ExtractiveQAPipeline(reader, retriever)
+```
+
+See [this](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) blog post for more details about the algorithm.
+
+
+## Dense Passage Retrieval (Recommended)
+
+### Description
+
+[Dense Passage Retrieval](https://arxiv.org/abs/2004.04906) is a highly performant retrieval method that calculates relevance using dense representations.
+Key features:
+
+
+* One BERT base model to encode documents
+
+
+* One BERT base model to encode queries
+
+
+* Ranking of documents done by dot product similarity between query and document embeddings
+
+
+Indexing using DPR is comparatively expensive in terms of required computation since all documents in the database need to be processed through the transformer.
+The embeddings that are created in this step can be stored in FAISS, a database optimized for vector similarity.
+DPR can also work with the ElasticsearchDocumentStore or the InMemoryDocumentStore.
+
+There are two design decisions that have made DPR particularly performant.
+
+
+* Separate encoders for document and query helps since queries are much shorter than documents
+
+
+* Training with ‘In-batch negatives’ (gold labels are treated as negative examples for other samples in same batch) is highly efficient
+
+In Haystack, you can simply download the pretrained encoders needed to start using DPR.
+If you’d like to learn how to set up a DPR based system, have a look at the [tutorial](/docs/latest/tutorial6md)!
+
+### Initialisation
+
+
+
+**Tip**
+
+When using DPR, it is recommended that you use the dot product similarity function since that is how it is trained.
+To do so, simply provide `similarity="dot_product"` when initializing the DocumentStore
+as is done in the code example below.
+
+
+
+```python
+from haystack.document_store import FAISSDocumentStore
+from haystack.retriever import DensePassageRetriever
+from haystack.pipeline import ExtractiveQAPipeline
+
+document_store = FAISSDocumentStore(similarity="dot_product")
+...
+retriever = DensePassageRetriever(
+ document_store=document_store,
+ query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
+ passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base"
+)
+...
+finder = ExtractiveQAPipeline(reader, retriever)
+```
+
+
+
+**Training DPR:** Haystack supports training of your own DPR model! Check out the [tutorial](/docs/latest/tutorial9md) to see how this is done!
+
+
+
+
+
+## Embedding Retrieval
+
+### Description
+
+In Haystack, you also have the option of using a single transformer model to encode document and query.
+One style of model that is suited to this kind of retrieval is that of [Sentence Transformers](https://github.com/UKPLab/sentence-transformers).
+These models are trained in Siamese Networks and use triplet loss such that they learn to embed similar sentences near to each other in a shared embedding space.
+
+They are particular suited to cases where your query input is similar in style to that of the documents in your database
+i.e. when you are searching for most similar documents.
+This is not inherently suited to query based search where the length, language and format of the query usually significantly differs from the searched for text.
+
+
+
+**Tip**
+
+When using Sentence Transformer models, we recommend that you use a cosine similarity function.
+To do so, simply provide `similarity="cosine"` when initializing the DocumentStore
+as is done in the code example below.
+
+
+
+### Initialisation
+
+```python
+from haystack.document_store import ElasticsearchDocumentStore
+from haystack.retriever import EmbeddingRetriever
+from haystack.pipeline import ExtractiveQAPipeline
+
+document_store = ElasticsearchDocumentStore(similarity="cosine")
+...
+retriever = EmbeddingRetriever(document_store=document_store,
+ embedding_model="deepset/sentence_bert")
+...
+p = ExtractiveQAPipeline(reader, retriever)
+```
+
+## Deeper Dive: Dense vs Sparse
+
+Broadly speaking, retrieval methods can be split into two categories: **dense** and **sparse**.
+
+**Sparse** methods, like TF-IDF and BM25, operate by looking for shared keywords between the document and query.
+They are:
+
+
+* simple but effective
+
+
+* don’t need to be trained
+
+
+* work on any language
+
+More recently, **dense** approaches such as Dense Passage Retrieval (DPR) have shown even better performance than their sparse counter parts.
+These methods embed both document and query into a shared embedding space using deep neural networks
+and the top candidates are the nearest neighbour documents to the query.
+They are:
+
+
+* powerful but computationally more expensive especially during indexing
+
+
+* trained using labelled datasets
+
+
+* language specific
+
+### Qualitative Differences
+
+Between these two types there are also some qualitative differences too.
+For example, sparse methods treat text as a bag-of-words meaning that they **do not take word order and syntax into account**,
+while the latest generation of dense methods use transformer based encoders
+which are designed to be **sensitive** to these factors.
+
+Also dense methods are very capable of building strong semantic representations of text,
+but they **struggle when encountering out-of-vocabulary** words such as new names.
+By contrast, sparse methods don’t need to learn representations of words,
+they only care about whether they are present or absent in the text.
+As such, **they handle out-of-vocabulary words with no problem**.
+
+
+### Indexing
+
+Dense methods perform indexing by processing all the documents through a neural network and storing the resulting vectors.
+This is a much more expensive operation than the creation of the inverted-index in sparse methods
+and will require significant computational power and time.
+
+
+### Terminology
+
+
+
+The terms **dense** and **sparse** refer to the representations that the algorithms build for each document and query.
+**Sparse** methods characterise texts using vectors with one dimension corresponding to each word in the vocabulary.
+Dimensions will be zero if the word is absent and non-zero if it is present.
+Since most documents contain only a small subset of the full vocabulary,
+these vectors are considered sparse since non-zero values are few and far between.
+
+**Dense** methods, by contrast, pass text as input into neural network encoders
+and represent text in a vector of a manually defined size (usually 768).
+Though individual dimensions are not mapped to any corresponding vocabulary or linguistic feature,
+each dimension encodes some information about the text.
+There are rarely 0s in these vectors hence their relative density.
diff --git a/docs/v1.3.0/_src/usage/usage/roadmap.md b/docs/v1.3.0/_src/usage/usage/roadmap.md
new file mode 100644
index 0000000000..47835bf4b3
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/roadmap.md
@@ -0,0 +1,45 @@
+
+
+# Open roadmap
+
+We believe open-source is more than open source code. It's a lot about people, collaboration, transparency and trust.
+Therefore, we decided to be as open as possible with our roadmap and sprint planning.
+In fact, you can see all of it in real-time on GitHub.
+We hope this helps to clarify the direction of the open-source project and inspires discussions in the community.
+
+## How to access it
+
+### Add Zenhub Plugin
+We decided for Zenhub, as it allows a close integration with GitHub and real-time sharing of roadmaps and sprints.
+Once you have installed the browser plugin below, you will see additional tabs and infos on the Haystack GitHub page.
+
+Zenhub Plugin: https://www.zenhub.com/extension
+
+### Roadmap
+_Zenhub Tab -> Roadmap (left menu)_
+
+Here you can find our most recent roadmap with the **high-level projects** that are planned for the upcoming quarters.
+We update it regularly and refine the projects as they come closer in time.
+
+
+
+### Board
+_Zenhub Tab -> Board (left menu)_
+
+If you are interested in the **operational tasks** and their status, you can find our agile board here.
+
+
+### Additional issue details & Releases
+_Right panel in regular Github issues_
+
+With Zenhub you can also see some additional tags in every GitHub issue.
+For those of you who wonder about the next release date: We aim for releases every ~ 4 weeks and will tag the issues that will need to be finished before a bit in advance.
+
+
diff --git a/docs/v1.3.0/_src/usage/usage/summarizer.md b/docs/v1.3.0/_src/usage/usage/summarizer.md
new file mode 100644
index 0000000000..93bde9cdaf
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/summarizer.md
@@ -0,0 +1,56 @@
+
+
+# Summarizer
+
+Retrievers are excellent at returning a set of candidate documents,
+but you might not have the time to read through them all.
+Haystack's Summmarizer is here to help you make sense of the documents at a glance.
+
+There is a full integration with Huggingface Transformers and using any of their summarization
+models is as simple as providing the model name.
+See the up-to-date list of available models [here](https://huggingface.co/models?filter=summarization).
+By default, the Google [Pegasus](https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html) model is loaded.
+
+```python
+from haystack.summarizer import TransformersSummarizer
+from haystack.schema import Document
+
+docs = [Document("PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions.\
+ The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by\
+ the shutoffs which were expected to last through at least midday tomorrow.")]
+
+summarizer = TransformersSummarizer(model_name_or_path="google/pegasus-xsum")
+summary = summarizer.predict(documents=docs, generate_single_summary=True)
+```
+
+The contents of summary should contain both the summarization and also the original document text.
+
+```python
+[
+ {
+ "text": "California's largest electricity provider has turned off power to hundreds of thousands of customers.",
+ "meta": {
+ "context": "PGE stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions."
+ },
+ ...
+ }
+]
+```
+
+The summarizer can also functions as a node in a pipeline.
+
+```python
+from haystack.pipeline import Pipeline
+
+p = Pipeline()
+p.add_node(component=retriever, name="ESRetriever1", inputs=["Query"])
+p.add_node(component=summarizer, name="Summarizer", inputs=["ESRetriever1"])
+res = p.run(query="What did Einstein work on?", top_k_retriever=10)
+```
diff --git a/docs/v1.3.0/_src/usage/usage/terms.md b/docs/v1.3.0/_src/usage/usage/terms.md
new file mode 100644
index 0000000000..fb77485096
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/terms.md
@@ -0,0 +1,65 @@
+
+
+# Glossary
+
+**BERT** - A popular, transformer based language model which has been improved upon but is still considered a common benchmark.
+
+**Dense** - Vectors that contain many non-zero values are considered dense.
+Retrieval methods can also be called dense if they create dense vector representations of documents.
+
+**Document** - A Document in Haystack refers to the individual pieces of text that are stored in the DocumentStore.
+Multiple Documents might originally come from the one file.
+It is ultimately up to you how to divide up your corpus into Documents.
+
+**Document Store** - The component in Haystack that stores the text documents and their metadata.
+Can have a variety of backends such as Elasticsearch, SQL or FAISS.
+
+**FARM** - An open-source transfer learning [framework](https://github.com/deepset-ai/FARM) by deepset.
+FARM’s question answering models are used in Haystack’s Readers.
+
+**Indexing** - To store data in a database in a way that optimises retrieval time.
+The exact steps involved in indexing depend on what kind of retrieval method is chosen.
+
+**Language Model** - The component in an NLP model that stores general language understanding, but no task specific knowledge.
+
+**Model Hub** - The [repository](https://huggingface.co/models) set up by HuggingFace where trained models can be saved to and loaded from.
+With Haystack, you can directly load and use any question answering model found on the model hub.
+
+**Neural Network** - A machine learning architecture composed of artificial neurons that learn a task when exposed to labelled training data.
+
+**Prediction Head** - The modelling component that adapts the general knowledge of the language model for a specific task.
+In question answering models (and hence in Haystack Readers), this is usually a single layer neural network.
+
+**Querying** - The task of returning relevant documents from a database.
+
+**Question Answering (QA)** - A popular task in the world of NLP where systems have to find answers to questions.
+The term is generally used to refer to extractive question answering,
+where a system has to find the minimal text span in a given document that contains the answer to the question.
+Note however, that it may also refer to abstractive question answering or FAQ matching.
+
+**Reader** - The component in Haystack that does the closest reading of a document to extract
+the exact text which answers a question.
+It is, at its core, a trained Question Answering model.
+
+**Retriever** - A lightweight filter that selects only the most relevant documents for the Reader to further process.
+
+**Semantic Search** - A style of search that relies not on the matching of exact string forms
+but on the similarity of meaning between a query and a piece of text.
+
+**Sparse** - Vectors that are composed primarily of zeros are called sparse.
+Retrieval methods are also considered sparse if they build sparse vector representations of documents.
+
+**SQuAD** - The [Stanford Question Answering Dataset](https://rajpurkar.github.io/SQuAD-explorer/) is the defacto standard QA dataset.
+The documents are paragraphs from Wikipedia and the question / answer pairs are created by human annotators.
+
+**Transformers** - Originally refers to the deep learning architecture that is composed of stacked self-attention layers
+(first conceptualised [here](https://arxiv.org/pdf/1706.03762.pdf)).
+Can also refer to HuggingFace’s [repository](https://github.com/huggingface/transformers)
+which contains implementations of popular model architectures.
diff --git a/docs/v1.3.0/_src/usage/usage/translator.md b/docs/v1.3.0/_src/usage/usage/translator.md
new file mode 100644
index 0000000000..409e30148d
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/translator.md
@@ -0,0 +1,61 @@
+
+
+# Translator
+
+Texts come in different languages. This is not different for search and there are plenty of options to deal with it.
+One of them is actually to translate the incoming query, the documents or the search results.
+
+Let's imagine you have an English corpus of technical docs, but the mother tongue of many of your users is French.
+You can use a Translator node in your pipeline to
+1. Translate the incoming query from French to English
+2. Search in your English corpus for the right document / answer
+3. Translate the results back from English to French
+
+
+
+**Example (Stand-alone Translator)**
+
+You can use the Translator component directly to translate your query or document(s):
+```python
+from haystack.schema import Document
+from haystack.translator import TransformersTranslator
+
+DOCS = [
+ Document(
+ text="""Heinz von Foerster was an Austrian American scientist
+ combining physics and philosophy, and widely attributed
+ as the originator of Second-order cybernetics."""
+ )
+ ]
+translator = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-en-fr")
+res = translator.translate(documents=DOCS, query=None)
+```
+
+**Example (Wrapping another Pipeline)**
+
+You can also wrap one of your existing pipelines and "add" the translation nodes at the beginning and at the end of your pipeline.
+For example, lets translate the incoming query to from French to English, then do our document retrieval and then translate the results back from English to French:
+
+```python
+from haystack.pipeline import TranslationWrapperPipeline, DocumentSearchPipeline
+from haystack.translator import TransformersTranslator
+
+pipeline = DocumentSearchPipeline(retriever=my_dpr_retriever)
+
+in_translator = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-fr-en")
+out_translator = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-en-fr")
+
+pipeline_with_translation = TranslationWrapperPipeline(input_translator=in_translator,
+ output_translator=out_translator,
+ pipeline=pipeline)
+```
+
+
+
diff --git a/docs/v1.3.0/_src/usage/usage/use_cases.md b/docs/v1.3.0/_src/usage/usage/use_cases.md
new file mode 100644
index 0000000000..49311328c7
--- /dev/null
+++ b/docs/v1.3.0/_src/usage/usage/use_cases.md
@@ -0,0 +1,69 @@
+
+
+
+# Use cases
+
+## Semantic Search System
+
+Take the leap from using keyword search on your own documents to semantic search with Haystack.
+
+
+* Store your documents in the database of your choice (Elasticsearch, SQL, in memory, FAISS)
+
+
+* Perform question driven queries.
+
+Expect to see results that highlight the very sentence that contains the answer to your question.
+Thanks to the power of Transformer based language models, results are chosen based on compatibility in meaning
+rather than lexical overlap.
+
+
+
+
+
+## Information Extractor
+
+Automate the extraction of relevant information from a set of documents that pertain to the same topics but for different entities.
+
+Haystack can:
+
+
+* Apply a set of standard questions to each document in a store
+
+
+* Return a NO_ANSWER if a given document does not contain the answer to a question
+
+Say you have the financial reports for different companies over different years.
+You can gather a set of standard questions which are applicable to each financial report,
+like *what is the revenue forecast for 2020?* or *what are the main sources of income?*.
+Haystack will try to find an answer for each question within each document!
+
+We’ve seen this style of application be particularly effective in the sphere of finance and patent law
+but we see a lot of potential in using this to gain a better overview of academic papers and internal business documents.
+
+
+## FAQ Style Question Answering
+
+Leverage existing FAQ documents and semantic similarity search to answer new incoming questions.
+The workflow is as follows:
+
+
+* Store a set of FAQ documents in Haystack
+
+
+* The user presents a new question
+
+
+* Haystack will find the closest match to the new question in the FAQ documents
+
+
+* The user will be presented with the most similar Question Answer pair
+
+Haystack’s flexibility allows you to give new users more dynamic access to your existing documentation.
diff --git a/haystack/document_stores/base.py b/haystack/document_stores/base.py
index f66e9c3791..6f44d024f1 100644
--- a/haystack/document_stores/base.py
+++ b/haystack/document_stores/base.py
@@ -24,7 +24,7 @@
try:
from numba import njit # pylint: disable=import-error
except (ImportError, ModuleNotFoundError):
- logger.info("Numba not found, replacing njit() with no-op implementation. " "Enable it with 'pip install numba'.")
+ logger.info("Numba not found, replacing njit() with no-op implementation. Enable it with 'pip install numba'.")
def njit(f):
return f
@@ -47,6 +47,7 @@ def run(self, sparql_query: str, index: Optional[str] = None, headers: Optional[
output = {"sparql_result": result}
return output, "output_1"
+ @abstractmethod
def query(self, sparql_query: str, index: Optional[str] = None, headers: Optional[Dict[str, str]] = None):
raise NotImplementedError
diff --git a/haystack/json-schemas/haystack-pipeline-1.2.1rc0.schema.json b/haystack/json-schemas/haystack-pipeline-1.2.1rc0.schema.json
index 064b817925..921e8a1ea0 100644
--- a/haystack/json-schemas/haystack-pipeline-1.2.1rc0.schema.json
+++ b/haystack/json-schemas/haystack-pipeline-1.2.1rc0.schema.json
@@ -1,6 +1,6 @@
{
"$schema": "http://json-schema.org/draft-07/schema",
- "$id": "https://haystack.deepset.ai/json-schemas/haystack-pipeline-1.2.1rc0.schema.json",
+ "$id": "https://haystack.deepset.ai/haystack/json-schemas/haystack-pipeline-1.2.1rc0.schema.json",
"title": "Haystack Pipeline",
"description": "Haystack Pipeline YAML file describing the nodes of the pipelines. For more info read the docs at: https://haystack.deepset.ai/components/pipelines#yaml-file-definitions",
"type": "object",
@@ -12,6 +12,9 @@
"oneOf": [
{
"const": "1.2.1rc0"
+ },
+ {
+ "const": "1.3.0"
}
]
},
@@ -45,6 +48,9 @@
{
"$ref": "#/definitions/OpenSearchDocumentStoreComponent"
},
+ {
+ "$ref": "#/definitions/PineconeDocumentStoreComponent"
+ },
{
"$ref": "#/definitions/SQLDocumentStoreComponent"
},
@@ -901,10 +907,6 @@
"title": "Parameters",
"type": "object",
"properties": {
- "host": {
- "title": "Host",
- "default": "https://admin:admin@localhost:9200/"
- },
"similarity": {
"title": "Similarity",
"default": "cosine"
@@ -969,6 +971,102 @@
],
"additionalProperties": false
},
+ "PineconeDocumentStoreComponent": {
+ "type": "object",
+ "properties": {
+ "name": {
+ "title": "Name",
+ "description": "Custom name for the component. Helpful for visualization and debugging.",
+ "type": "string"
+ },
+ "type": {
+ "title": "Type",
+ "description": "Haystack Class name for the component.",
+ "type": "string",
+ "const": "PineconeDocumentStore"
+ },
+ "params": {
+ "title": "Parameters",
+ "type": "object",
+ "properties": {
+ "api_key": {
+ "title": "Api Key",
+ "type": "string"
+ },
+ "environment": {
+ "title": "Environment",
+ "default": "us-west1-gcp",
+ "type": "string"
+ },
+ "sql_url": {
+ "title": "Sql Url",
+ "default": "sqlite:///pinecone_document_store.db",
+ "type": "string"
+ },
+ "pinecone_index": {
+ "title": "Pinecone Index",
+ "type": "string",
+ "default": null
+ },
+ "embedding_dim": {
+ "title": "Embedding Dim",
+ "default": 768,
+ "type": "integer"
+ },
+ "return_embedding": {
+ "title": "Return Embedding",
+ "default": false,
+ "type": "boolean"
+ },
+ "index": {
+ "title": "Index",
+ "default": "document",
+ "type": "string"
+ },
+ "similarity": {
+ "title": "Similarity",
+ "default": "cosine",
+ "type": "string"
+ },
+ "replicas": {
+ "title": "Replicas",
+ "default": 1,
+ "type": "integer"
+ },
+ "shards": {
+ "title": "Shards",
+ "default": 1,
+ "type": "integer"
+ },
+ "embedding_field": {
+ "title": "Embedding Field",
+ "default": "embedding",
+ "type": "string"
+ },
+ "progress_bar": {
+ "title": "Progress Bar",
+ "default": true,
+ "type": "boolean"
+ },
+ "duplicate_documents": {
+ "title": "Duplicate Documents",
+ "default": "overwrite",
+ "type": "string"
+ }
+ },
+ "required": [
+ "api_key"
+ ],
+ "additionalProperties": false,
+ "description": "Each parameter can reference other components defined in the same YAML file."
+ }
+ },
+ "required": [
+ "type",
+ "name"
+ ],
+ "additionalProperties": false
+ },
"SQLDocumentStoreComponent": {
"type": "object",
"properties": {
@@ -1399,9 +1497,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
@@ -1670,9 +1765,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
@@ -2752,9 +2844,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
@@ -3097,9 +3186,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
diff --git a/haystack/json-schemas/haystack-pipeline-unstable.schema.json b/haystack/json-schemas/haystack-pipeline-unstable.schema.json
index 4f49f36310..2f3a958ffb 100644
--- a/haystack/json-schemas/haystack-pipeline-unstable.schema.json
+++ b/haystack/json-schemas/haystack-pipeline-unstable.schema.json
@@ -1,6 +1,6 @@
{
"$schema": "http://json-schema.org/draft-07/schema",
- "$id": "https://haystack.deepset.ai/json-schemas/haystack-pipeline-unstable.schema.json",
+ "$id": "https://haystack.deepset.ai/haystack/json-schemas/haystack-pipeline-unstable.schema.json",
"title": "Haystack Pipeline",
"description": "Haystack Pipeline YAML file describing the nodes of the pipelines. For more info read the docs at: https://haystack.deepset.ai/components/pipelines#yaml-file-definitions",
"type": "object",
@@ -15,6 +15,9 @@
},
{
"const": "1.2.1rc0"
+ },
+ {
+ "const": "1.3.0"
}
]
},
@@ -48,6 +51,9 @@
{
"$ref": "#/definitions/OpenSearchDocumentStoreComponent"
},
+ {
+ "$ref": "#/definitions/PineconeDocumentStoreComponent"
+ },
{
"$ref": "#/definitions/SQLDocumentStoreComponent"
},
@@ -904,10 +910,6 @@
"title": "Parameters",
"type": "object",
"properties": {
- "host": {
- "title": "Host",
- "default": "https://admin:admin@localhost:9200/"
- },
"similarity": {
"title": "Similarity",
"default": "cosine"
@@ -972,6 +974,102 @@
],
"additionalProperties": false
},
+ "PineconeDocumentStoreComponent": {
+ "type": "object",
+ "properties": {
+ "name": {
+ "title": "Name",
+ "description": "Custom name for the component. Helpful for visualization and debugging.",
+ "type": "string"
+ },
+ "type": {
+ "title": "Type",
+ "description": "Haystack Class name for the component.",
+ "type": "string",
+ "const": "PineconeDocumentStore"
+ },
+ "params": {
+ "title": "Parameters",
+ "type": "object",
+ "properties": {
+ "api_key": {
+ "title": "Api Key",
+ "type": "string"
+ },
+ "environment": {
+ "title": "Environment",
+ "default": "us-west1-gcp",
+ "type": "string"
+ },
+ "sql_url": {
+ "title": "Sql Url",
+ "default": "sqlite:///pinecone_document_store.db",
+ "type": "string"
+ },
+ "pinecone_index": {
+ "title": "Pinecone Index",
+ "type": "string",
+ "default": null
+ },
+ "embedding_dim": {
+ "title": "Embedding Dim",
+ "default": 768,
+ "type": "integer"
+ },
+ "return_embedding": {
+ "title": "Return Embedding",
+ "default": false,
+ "type": "boolean"
+ },
+ "index": {
+ "title": "Index",
+ "default": "document",
+ "type": "string"
+ },
+ "similarity": {
+ "title": "Similarity",
+ "default": "cosine",
+ "type": "string"
+ },
+ "replicas": {
+ "title": "Replicas",
+ "default": 1,
+ "type": "integer"
+ },
+ "shards": {
+ "title": "Shards",
+ "default": 1,
+ "type": "integer"
+ },
+ "embedding_field": {
+ "title": "Embedding Field",
+ "default": "embedding",
+ "type": "string"
+ },
+ "progress_bar": {
+ "title": "Progress Bar",
+ "default": true,
+ "type": "boolean"
+ },
+ "duplicate_documents": {
+ "title": "Duplicate Documents",
+ "default": "overwrite",
+ "type": "string"
+ }
+ },
+ "required": [
+ "api_key"
+ ],
+ "additionalProperties": false,
+ "description": "Each parameter can reference other components defined in the same YAML file."
+ }
+ },
+ "required": [
+ "type",
+ "name"
+ ],
+ "additionalProperties": false
+ },
"SQLDocumentStoreComponent": {
"type": "object",
"properties": {
@@ -1402,9 +1500,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
@@ -1673,9 +1768,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
@@ -2755,9 +2847,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
@@ -3100,9 +3189,6 @@
"type": "array",
"items": {
"anyOf": [
- {
- "type": "integer"
- },
{
"type": "string"
},
diff --git a/haystack/json-schemas/haystack-pipeline.schema.json b/haystack/json-schemas/haystack-pipeline.schema.json
index bc868e97ee..f8e15aecb4 100644
--- a/haystack/json-schemas/haystack-pipeline.schema.json
+++ b/haystack/json-schemas/haystack-pipeline.schema.json
@@ -55,6 +55,9 @@
"oneOf": [
{
"const": "1.2.1rc0"
+ },
+ {
+ "const": "1.3.0"
}
]
}
diff --git a/haystack/nodes/_json_schema.py b/haystack/nodes/_json_schema.py
index e9a880476a..f02e5862f1 100644
--- a/haystack/nodes/_json_schema.py
+++ b/haystack/nodes/_json_schema.py
@@ -360,9 +360,7 @@ def new_version_entry(version):
def update_json_schema(
- update_index: bool,
- destination_path: Path = JSON_SCHEMAS_PATH,
- index_path: Path = JSON_SCHEMAS_PATH / "haystack-pipeline.schema.json",
+ update_index: bool, destination_path: Path = JSON_SCHEMAS_PATH, index_name: Path = "haystack-pipeline.schema.json"
):
# Locate the latest schema's path
latest_schema_path = destination_path / Path(
@@ -426,23 +424,24 @@ def update_json_schema(
# Update the JSON schema index too
if update_index:
- index = load(index_path)
+ index = load(destination_path / index_name)
index["oneOf"][-1]["allOf"][0]["properties"]["version"]["oneOf"] = supported_versions_block
- dump(index, index_path)
+ dump(index, destination_path / index_name)
# Dump the new schema file
new_schema["$id"] = f"{SCHEMA_URL}{filename}"
unstable_versions_block = [{"const": haystack_version}]
new_schema["properties"]["version"]["oneOf"] = [{"const": haystack_version}]
dump(new_schema, destination_path / filename)
+ logger.info(f"Schema saved in {destination_path / filename}")
# Update schema index with a whole new entry
if update_index:
- index = load(index_path)
+ index = load(destination_path / index_name)
new_entry = new_version_entry(haystack_version)
if all(new_entry != entry for entry in index["oneOf"]):
index["oneOf"].append(new_version_entry(haystack_version))
- dump(index, index_path)
+ dump(index, destination_path / index_name)
# If the two schemas are compatible, no need to write a new one:
# Just add the new version to the list of versions supported by
@@ -479,12 +478,13 @@ def update_json_schema(
unstable_versions_block = supported_versions_block
latest_schema["properties"]["version"]["oneOf"] = supported_versions_block
dump(latest_schema, latest_schema_path)
+ logger.info(f"Schema updated in {destination_path / latest_schema_path}")
# Update the JSON schema index too
if update_index:
- index = load(index_path)
+ index = load(destination_path / index_name)
index["oneOf"][-1]["allOf"][0]["properties"]["version"]["oneOf"] = supported_versions_block
- dump(index, index_path)
+ dump(index, destination_path / index_name)
# Update the unstable schema (for tests and internal use).
unstable_filename = "haystack-pipeline-unstable.schema.json"
@@ -492,3 +492,4 @@ def update_json_schema(
unstable_schema["$id"] = f"{SCHEMA_URL}{unstable_filename}"
unstable_schema["properties"]["version"]["oneOf"] = [{"const": "unstable"}] + unstable_versions_block
dump(unstable_schema, destination_path / unstable_filename)
+ logger.info(f"Unstable schema saved in {destination_path / unstable_filename}")