Feature/ogc 508 replace elastic search by postgres text search

[Tschuppi81]

INITIAL FEEDBACK TO TEXT SEARCH IN POSTGRES - WORK IN PROGRESS

Feature/ogc 508 replace elastic search by postgres text search

core: Replace elastic search by postgres full text search

With this changes elastic search (es) is still in place and productive while we introduce the postgres full text search accessible using 'search_postgres' keyword in the url.
es url: http://localhost:8080/onegov_agency/bs/search?q=Arnold
psql url: http://localhost:8080/onegov_agency/bs/search_postgres?q=Arnold

hint: onegov-core upgrade

Main changes:

• Re-indexing
  ../onegov-cloud/src/onegov/search/cli.py
• Upgrade script (uses the same code as re-indexing)
  ../onegov-cloud/src/onegov/search/upgrade.py
• Extension of mixin class Searchable
  .. /onegov-cloud/src/onegov/search/mixins.py
• Adjusting all searchable models adding the fts index in postgres

Open Items:

• unit tests are not reworked yet
• language configuration
• improve ranking

Checklist

• I have performed a self-review of my code
• I considered adding a reviewer
• I have added an upgrade hint such as data migration commands to be run
• I made changes/features for both org and town6
• I have tested my code thoroughly by hand
  • I have tested database upgrades
• I have added tests for my changes/features

[Daverball]

This will probably break a lot of things (anything that uses directory_id in a query) and also does not do what you want it to do. The reason this previously failed for search indexing is simply because to_tsvector is not implemented for UUID so it needs to first be converted to TEXT, however, I'm not sure if the format is the same between Postgres and Python (i.e. if there will be dashes or not in the resulting string if you do a cast).

Type coercion is going to be tricky in general and doing it automatically might not be possible, although you may be able to do something with CASE and pg_typeof, instead of the the coalesce you've been doing to handle NULL.

to_tsvector definitely works for strings and JSON, although in the case of JSON i'm not sure what will end up in the search index, if it's just the values or also the keys. I don't remember if it works for array and hash, but you will need to figure out a way to make all the various cases work correctly.

Also removing all those hybrid_property is also not what you want to do to make this work, but you will need to figure out a way to generate an equivalent SQL expression for what the hybrid_property returns, so it can end up in the search index. In some cases this might not be possible and we may need a computed column to calculate the value in-app and store this value in the database.

Another route would be to forego the automatically updated index and instead generate and update it in-app by subscribing to the relevant orm events, as we have been doing previously for elastic search. This would mean that in-app we pull out all the string values that should end up in the index and then update the tsvector column based on those values.

[Tschuppi81]

@Daverball In the above commit I am fighting with more complex hybrid expressions. Most of them gather data from other tables and I could not figure out how to properly join the corresponding tables. Currently the 'reindexing' fails in CourseEvent.
Note: This commit to be reverted - just shows the join problem...

[Daverball]

@Tschuppi81 This is going to be tricky in general and probably a bad idea in terms of performance anyways, you are going to be better off adding caches for the joined attributes and using @observes methods to keep the caches up to date.

Or we could give up on calculating the search index online, since we're still quite a ways off and just generate a dictionary offline of all the search values which then can be turned into a TSVECTOR directly if interpreted as JSONB. While somewhat inefficient, this would still be quicker than what we had before with elasticsearch, since we save ourselves from having to talk with the elasticsearch server. It would also get rid of the issues related to polymorphism, since SQLAlchemy is aware of what polymorphic identity we're talking about, while in postgres we would have to write CASE statements to distinguish the various versions in the same table.