Skip to content

v1.2.0
Compare
Choose a tag to compare
@julian-risch julian-risch released this 23 Feb 16:02
v1.2.0
d21b6a5
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

⭐ Highlights

Brownfield Support of Existing Elasticsearch Indices

You have an existing Elasticsearch index from other projects and now want to try out Haystack? The newly added method es_index_to_document_store provides brownfield support of existing Elasticsearch indices by converting each of the records in the provided index to Haystack Document objects and writing them to the specified DocumentStore.

document_store = es_index_to_document_store(
    document_store=InMemoryDocumentStore(), #or any other Haystack DocumentStore
    original_index_name="existing_index",
    original_content_field="content",
    original_name_field="name",
    included_metadata_fields=["date_field"],
    index="new_index",
)

It can even be used on a regular basis in order to add new records of the Elasticsearch index to the DocumentStore! #2229

Tapas Reader With Scores

The new model class TapasForScoredQA introduced in #1997 supports Tapas Reader models that return confidence scores. When you load a Tapas Reader model, Haystack automatically infers whether the model supports confidence scores and chooses the correct model class under the hood. The returned answers are sorted first by a general table score and then by answer span scores. To try it out, just use one of the new TableReader models:

reader = TableReader(model_name_or_path="deepset/tapas-large-nq-reader", max_seq_len=512) #or
reader = TableReader(model_name_or_path="deepset/tapas-large-nq-hn-reader", max_seq_len=512)

Extended Meta Data Filtering

We extended the filter capabilities of all(*) document stores to support more complex filter expressions than previously. Besides simple selections on multiple fields you can now use more complex comparison expressions and connect these using boolean operators. For people having used mongodb the new syntax should look familiar. Filters are defined as nested dictionaries. The keys of the dictionaries can be a logical operator ("$and", "$or", "$not"), a comparison operator ("$eq", "$in", "$gt", "$gte", "$lt", "$lte") or a metadata field name.

Logical operator keys take a dictionary of metadata field names and/or logical operators as value. Metadata field names take a dictionary of comparison operators as value. Comparison operator keys take a single value or (in case of "$in") a list of values as value.

If no logical operator is provided, "$and" is used as default operation.
If no comparison operator is provided, "$eq" (or "$in" if the comparison value is a list) is used as default operation.

Therefore, we don't have any breaking changes and you can keep on using your existing filter expressions.

Example:

filters = {
    "$and": {
        "type": {"$eq": "article"},
        "date": {"$gte": "2015-01-01", "$lt": "2021-01-01"},
        "rating": {"$gte": 3},
        "$or": {
            "genre": {"$in": ["economy", "politics"]},
            "publisher": {"$eq": "nytimes"}
        }
    }
}
# or simpler using default operators
filters = {
    "type": "article",
    "date": {"$gte": "2015-01-01", "$lt": "2021-01-01"},
    "rating": {"$gte": 3},
    "$or": {
        "genre": ["economy", "politics"],
        "publisher": "nytimes"
    }
}

(*) FAISSDocumentStore and MilvusDocumentStore currently do not support filters during search.

Code Style and Linting

In addition to mypy we already had for static type checking, we now use pylint for linting and the Haystack code base does now comply with Black formatting standards. As a result, the code is formatted in a consistent way and easier to read. When you would like to contribute to Haystack you don't need to worry about that though - our CI will automatically format your code changes correctly. Our contributor guidelines give more details in case you would like to run the checks locally. #2115 #2130

Installation with fewer dependencies

Installing Haystack has become easier and faster thanks to optional dependencies. From now on, there is no need to install all dependencies if you don't need them. For example, pip3 install farm-haystack will install the latest release together with only a small subset of packages required for basic Pipelines with an ElasticsearchDocumentStore. As another example, if you are experimenting with FAISSDocumentStore in a colab notebook, you can install Haystack from the master branch together with FAISS dependency by running: !pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,faiss]. The installation guide reflects these updates and the full list of subsets of dependencies can be found here. Keep in mind, though, that this system works best with pip versions above 22 #1994

⚠️ Known Issues

Installing haystack with all dependencies results in heavy pip backtracking that might never finish.
This is due to a dependency conflict that was introduced by a new release of one of our sub dependencies.
To circumvent this problem install haystack like this:

pip install farm-haystack[all] "azure-core<1.23"

This might also be needed for other non-default dependencies (e.g. farm-haystack[dev] "azure-core<1.23").
See #2280 for more information.

⚠️ Breaking Changes

  • Improve dependency management by @ZanSara in #1994
  • Make ui and rest proper packages by @ZanSara in #2098
  • Add aiorwlock to 'ray' extra & fix maximum version for some dependencies by @ZanSara in #2140

🤓 Detailed Changes

Pipeline

Models

DocumentStores

REST API

Docker

Documentation

CI

Other Changes

New Contributors

❤️ Big thanks to all contributors and the whole community!

Contributors

tiangolo, tholor, and 15 other contributors
Assets 2
Loading