Swirl Metasearch 2.5

[erikspears]

Team Swirl is announcing General Availability of Swirl 2.5!

The theme for this release was performance. Swirl 2.5 produces a median search time of 3 queries per second in our test setup, at a maximum rate of 15 queries/second on a Standard F16s v2 (16 vcpus, 32 GiB memory) server.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

Performance Improvements

🔹 Swirl's asynchronous back-end was refactored to use the group method and replaced RabbitMQ with Redis

🔹 Relevancy processing was split into two stages to improve performance

• The revised CosineRelevancyPostResultProcessor must be in the Search.post_result_processors list.
• Please review the JSON in the SearchProviders/ directory and update existing configurations to match.

HubSpot SearchProviders

🔹 Search Company, Contact, and Deal records with new token-based HubSpot SearchProviders

• HubSpot requires creation of a "Private Apps" API token by the Super Admin with these scopes
• In result_mapptings, the url mapping must contain the HubSpot Org ID

Improvements

🔹 The DateFindingResultProcessor was added to the Google PSE SearchProvider JSON. It finds a date in a large percentage of results that otherwise wouldn't have one, and copies the date to the date_published field. Existing PSE SearchProvider configurations should be updated as follows:

"result_processors": [
            "MappingResultProcessor",
            "DateFinderResultProcessor",
            "CosineRelevancyResultProcessor"
        ],

🔹 Swirl's control script (python swirl.py) was updated

• A default key was added to the services list, including debug, in swirl/services.py
• The celery-beats service was set to default: False
• swirl.py was modified to load only services with default: True

To use the subscribe or expiration features, celery-beats must be started manually:
python swirl.py start celery-beats

Celery-beats must also be stopped separately.

🔹 Swirl now outputs a single log entry with each request at the default log level INFO

2023-08-02 10:49:09,466 INFO     admin search 452 FULL_RESULTS_READY 32 2.2

Detailed logging is available in Debug mode: restart Swirl with the --debug flag to enable

🔹 The install.sh script was refactored to better handle spaCy upgrades

🔹 Swirl's Docker image was updated to use the latest, stable release of Python, version 3.11.4

🔹 Use of the unsafe eval() function was removed

Resolved Issues

🔹 Swirl's log files now persist and append new entries across restarts

🔹 The SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages were updated

🔹 A seesion-related issue in the connectors for Sqlite3, PostgreSQL, and Elastic was fixed

🔹 Errors in the DedupeByFieldResultProcessor were addressed

🔹 The SearchProvider results_per_query value is handled correctly now

🔹 An issue where some searches didn't appear to reach the Swirl backend was fixed

🔹 Unnecessary WARNING messages were removed from the logs

🔹 Result URLs on Search objects are now constructed correctly using the Request instead of loading from django settings

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the MS Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Subscribed searches are not returning results for Microsoft sources; this will be addressed in an upcoming release.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.5 requires database migration. See Upgrading Swirl.

⚠️ Version 2.5 replaces RabbitMQ with Redis. Swirl's latest Docker image already contains this change. See the Quick Start - Local Installation section for details on how to install Redis with Swirl on MacOS or Linux.

⚠️ The revised CosineRelevancyResultProcessor must be added last to the result_processors block of all SearchProvider configurations except ChatGPT. For example:

"result_processors": [
            "MappingResultProcessor",
            "LenLimitingResultProcessor",
            "CosineRelevancyResultProcessor"
        ],

This following of Error message in the Swirl logs indicates that one or more SearchProviders have not been updated:

INFO     search.py: invoking processor: CosineRelevancyPostResultProcessor
2023-07-31 16:31:39,268 ERROR    CosineRelevancyPostResultProcessor_2051: Error: Dictionary of result lengths is empty. Was CosineRelevancyResultProcessor included in Search Providers Processor configuration?

⚠️ Exisitng SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages shoudl be updated:

• Elastic: Remove hosts= and http-auth= (just that text, not the values that follow)
• OpenSearch: Update the query_template syntax to the following:

"query_template": "{\"query\":{\"query_string\":{\"query\":\"{query_string}\",\"default_field\":\"{default_field}\",\"default_operator\":\"and\"}}}",

• Microsoft Outlook Messages: Update both result_grouping_field and result_processors to the following:

"result_grouping_field": "conversationId",
"result_processors": [
            "MappingResultProcessor",
            "DedupeByFieldResultProcessor",
	    "CosineRelevancyResultProcessor"
],

Documentation

🔹 Quick Start 🔹 User Guide 🔹 Admin Guide 🔹 M365 Guide 🔹 Developer Guide 🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!