Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for case insensitive regexp + add support for REGEXP SQLite module #1737

Merged
merged 17 commits into from
Dec 30, 2024

Conversation

larsschwegmann
Copy link
Contributor

@larsschwegmann larsschwegmann commented Oct 14, 2024

This PR adds support for case insentive regex filtering (postgres and sqlite). It also adds support for the REGEXP sqlite module (needs to be installed in sqlite to work).

Description

Postgres and SQLite offer ways to query with regular expression while ignoring case, similiarly how we already have case insensitive contains, etc. This PR adds support for these filters.

This PR also adds support for SQLite (regexp() function needs to be supplied at runtime, e.g. via an extension). It is installed by default on most linux distributions of sqlite.

Motivation and Context

I want to query case insensitive regex patterns. Having sqlite support is good for completeness. Many people including me use sqlite for unit tests, so having feature parity across db backends is aa big plus.

How Has This Been Tested?

Ran the tests suite

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added the changelog accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@coveralls
Copy link

coveralls commented Oct 14, 2024

Pull Request Test Coverage Report for Build 12543888866

Details

  • 51 of 68 (75.0%) changed or added relevant lines in 11 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.1%) to 89.915%

Changes Missing Coverage Covered Lines Changed/Added Lines %
tortoise/backends/sqlite/init.py 4 5 80.0%
tortoise/filters.py 1 2 50.0%
tortoise/backends/sqlite/client.py 5 8 62.5%
tortoise/contrib/sqlite/regex.py 13 25 52.0%
Totals Coverage Status
Change from base Build 12541013179: -0.1%
Covered Lines: 6442
Relevant Lines: 7046

💛 - Coveralls

@abondar
Copy link
Member

abondar commented Nov 9, 2024

@larsschwegmann Hi!

Sorry I didn't pay attention to this PR, seems like I missed it

Could you please add some tests for that feature?

@larsschwegmann
Copy link
Contributor Author

Sorry, I also missed your response while i was on vacation. I'll add tests later today 👍

@larsschwegmann
Copy link
Contributor Author

@abondar Hi again! I added tests for the new functionality. Please take a look and let me know what you think. Feel free to merge if it looks okay to you 😄

@@ -81,6 +82,7 @@ async def create_connection(self, with_db: bool) -> None:
for pragma, val in self.pragmas.items():
cursor = await self._connection.execute(f"PRAGMA {pragma}={val}")
await cursor.close()
await install_regexp_function(self._connection)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems a bit strange that we are automatically installing new functions in user db just on connection

May be we should add some kind of flag to make it explicit and only in case user need it?
If we do so, also would be nice to check if it was activated inside function and propagate proper error for user to prompt him to switch that flag

Even though it sounds like unneeded extra steps - it seems reasonable to not modify user db without explicit requests to do so

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, you are probably right about that. although it does not modify the db, it could introduce bad behaviour in case a user has already overwritten the regexp or match functions. I like the approach with the flag you proposed, I'll try and implement that 😄

@larsschwegmann
Copy link
Contributor Author

larsschwegmann commented Dec 16, 2024

Hi @abondar !
I made the installation of the regexp user functions to the sqlite engine optional via a config credential. I added a new run group to the makefile (make test_sqlite_regexp) that runs the sqlite tests with TORTOISE_TEST_DB=sqlite://:memory:?install_regexp_functions=True. This is how users can also enable it in their own sqlite configs via the DB URL. I also added an entry to the Capabilities class, so that every db client can expose that info and the tests are excluded when using the vanilla sqlite://:memory: engine.
I hope you are fine with my solution. Please let me know if this works for you 😄

@abondar
Copy link
Member

abondar commented Dec 21, 2024

@larsschwegmann Hi!

Sorry, I missed your update and now branch is stale
Could you please rebase it, so I will be able to merge it?

@larsschwegmann
Copy link
Contributor Author

Hi @abondar , thanks for reaching out :)
I have rebased the branch 👍

@larsschwegmann
Copy link
Contributor Author

Hi @abondar , could you please check when you have time? I have rebased again

@abondar abondar merged commit 8ac9b28 into tortoise:develop Dec 30, 2024
8 checks passed
@larsschwegmann
Copy link
Contributor Author

Thank you for merging @abondar ! 😄
Do you happen to know if there are plans already when the next version will be released?

@abondar
Copy link
Member

abondar commented Jan 20, 2025

I think it should be soon enough
Afaik - @henadzit preparing new version now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants