Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add docker-compose #113

Open
wants to merge 186 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
186 commits
Select commit Hold shift + click to select a range
29eeccc
chore: language update for install from source.
billy-the-fish Jul 10, 2024
e354514
Excellent :-).
billy-the-fish Jul 10, 2024
9965e16
Merge pull request #46 from timescale/language-review
billy-the-fish Jul 10, 2024
73fdae4
Update README.md
avthars Jul 11, 2024
a3e2b35
Move asdf plpython3u instructions to plpython3u section
alejandrodnm Jul 22, 2024
237499e
Merge pull request #48 from timescale/adn/readme-fix
alejandrodnm Jul 23, 2024
de1bc62
Adapt build/install to deal with versioning
jgpruitt Jul 15, 2024
6fc188d
Add linter and format checks
jgpruitt Jul 25, 2024
94b1b82
Fixed silly logic bug
jgpruitt Jul 26, 2024
007d9cf
Add space in README
mfreed Aug 2, 2024
2c68539
Fix typo referencing OpenAI instead of Ollama
mfreed Aug 2, 2024
b6e1a53
Merge pull request #57 from timescale/mfreed-patch-1
mfreed Aug 2, 2024
c47a4e0
Merge pull request #58 from timescale/mfreed-patch-2
mfreed Aug 2, 2024
ea947b5
Add support for rank_fields parameter to cohere_rerank
jgpruitt Jul 26, 2024
ce1b6d4
Add support for base_url param to openai
jgpruitt Jul 26, 2024
a470e4a
Modify func volatility for performance
jgpruitt Aug 5, 2024
ee7efea
Convert ollama tests to pytest
jgpruitt Aug 7, 2024
80af00b
Convert anthropic tests to pytest
jgpruitt Aug 7, 2024
564acf4
Convert openai tests to pytest
jgpruitt Aug 7, 2024
fd8a908
Convert cohere tests to pytest
jgpruitt Aug 7, 2024
f00b047
Remove old tests
jgpruitt Aug 7, 2024
1f7fe5a
Update DEVELOPMENT doc
jgpruitt Aug 7, 2024
2d5059f
docs: update README.md
eltociear Aug 12, 2024
20c3853
1. create new branch rag_demo_v1 , add verified file and merged to m…
Wang-Yong2018 Aug 19, 2024
2458010
change the demo sql from doc folder to example folder.
Wang-Yong2018 Aug 19, 2024
8edee29
add blog url on top of the rag_report_demo_why_some_customer_donot_li…
Wang-Yong2018 Sep 7, 2024
630dd1a
add url on top of rag_report_demo_why_some_custom_donot_like_pizza.sql
Wang-Yong2018 Sep 7, 2024
26ccdc3
Add grant_ai_usage function
jgpruitt Aug 12, 2024
4533b97
thinking through privileges. no clear solution yet
jgpruitt Aug 12, 2024
f69bd12
Add vectorizer config functions
jgpruitt Aug 13, 2024
33a23b7
more progress on vectorizer
jgpruitt Aug 13, 2024
10bf2c6
now sending http request
jgpruitt Aug 13, 2024
7d00b99
Add retries with exponential backoff and logging
jgpruitt Aug 13, 2024
52e4502
in a working state
jgpruitt Aug 14, 2024
4e140f0
Improve make test-server so it works from host
jgpruitt Aug 14, 2024
ffb8290
Add source_pk to vectorizer config
jgpruitt Aug 14, 2024
0835f29
Add asynchronous and external columns to vectorizer
jgpruitt Aug 14, 2024
fc5d497
Move queue table, funcs, triggers to ai schema
jgpruitt Aug 14, 2024
1477682
Also check the source table
jgpruitt Aug 14, 2024
c71fab0
Add queue table/schema back to vectorizer table
jgpruitt Aug 14, 2024
26eb013
Allow overriding queue schema and table
jgpruitt Aug 14, 2024
26384ed
Remove version from vectorizer test
jgpruitt Aug 15, 2024
de78039
Schedule vectorizers with pg_cron or timescaledb jobs
jgpruitt Aug 19, 2024
8e94522
Add psql_cmd func to make test more readable
jgpruitt Aug 20, 2024
f6b12c9
Validate chunking and formatting configs
jgpruitt Aug 20, 2024
f27fc72
Fix issue with background jobs
jgpruitt Aug 20, 2024
c9559bd
Improve vectorizer test
jgpruitt Aug 20, 2024
73149ce
Improve vectorizer test
jgpruitt Aug 20, 2024
54fc809
Rework embedding config
jgpruitt Aug 21, 2024
41810ab
Rework scheduling configs
jgpruitt Aug 21, 2024
6a83d48
Rework formatting config
jgpruitt Aug 21, 2024
02b0d8e
Rework the chunking config
jgpruitt Aug 21, 2024
e2fae94
Only allow the owner of a table to vectorizer it
jgpruitt Aug 21, 2024
19f7eaf
Add vectorizer_queue_depth function
jgpruitt Aug 22, 2024
bd5bbed
Deal with vectorizers that have no queue
jgpruitt Aug 22, 2024
eec6da9
Make vectorizer_queue_depth stable
jgpruitt Aug 22, 2024
1ffe7ae
Add trigger_name to vectorizer
jgpruitt Aug 22, 2024
dfa2119
Add drop_vectorizer function
jgpruitt Aug 22, 2024
79abef4
Fix pgspot issue
jgpruitt Aug 22, 2024
1ea3c04
Allow list of roles to be granted privs
jgpruitt Aug 22, 2024
cb4df3d
Add GUC for vectorizer URL
jgpruitt Aug 22, 2024
0969761
Add default template for formatting_python_template
jgpruitt Aug 22, 2024
d83d05c
Use default python_template
jgpruitt Aug 22, 2024
129bdaf
Add leading underscore to default queue table name
jgpruitt Aug 22, 2024
12a40b5
Add indexing configs
jgpruitt Aug 22, 2024
72cdaed
Add tests for indexing configs
jgpruitt Aug 22, 2024
4dc287a
Change target suffix to _embedding_store
jgpruitt Aug 22, 2024
86f2c39
Change target pk to embedding_uuid
jgpruitt Aug 22, 2024
cd471c6
Remove target_column parameter
jgpruitt Aug 22, 2024
8f94232
Add view joining embeddings back to source
jgpruitt Aug 22, 2024
d26fbd2
Add privileges to the view
jgpruitt Aug 26, 2024
0465e15
Primary key columns must be not null
jgpruitt Aug 26, 2024
d4ebe87
Parse indexing config in test server
jgpruitt Aug 26, 2024
32d1b3e
Fully parse request in test server
jgpruitt Aug 26, 2024
7a7cd2e
Add vectorizer_status view
jgpruitt Aug 26, 2024
39363f2
Add a 'make run' target
jgpruitt Aug 26, 2024
6e78e1a
Remove leading underscores from param names
jgpruitt Aug 26, 2024
89f698c
Don't use reserved work for param
jgpruitt Aug 26, 2024
66d0f97
Fixes for pg_cron and tests of the same
jgpruitt Aug 27, 2024
455be82
Collection of minor fixes
jgpruitt Aug 28, 2024
29be812
Send http request to delete vectorizer
jgpruitt Aug 28, 2024
9b2812c
Minimal dump/restore test
jgpruitt Aug 28, 2024
d048f1e
Ensure that dump/restore works for non-superusers
jgpruitt Aug 29, 2024
2fe3d86
Use session_user instead of pg_database_owner
jgpruitt Sep 3, 2024
64f8a01
Rename execute_async_ext_vectorizer
jgpruitt Sep 3, 2024
d9f62cb
Make several funcs immutable
jgpruitt Sep 3, 2024
6e49fc8
Enable join removal optimization on view
jgpruitt Sep 3, 2024
7960777
Move trigger func to ai schema
jgpruitt Sep 3, 2024
608ffc9
Select 1 instead of *
jgpruitt Sep 3, 2024
c13a695
Simplify _vectorizer_schedule_async_ext_job
jgpruitt Sep 3, 2024
b811ffe
Remove asynchronous and external fields from vectorizer
jgpruitt Sep 3, 2024
54fd2b2
Enable/disable/delete scheduled jobs
jgpruitt Sep 3, 2024
995b482
Split large files into smaller ones
jgpruitt Sep 3, 2024
e6a00bf
Add config_type to vectorizer configs
jgpruitt Sep 3, 2024
efdf736
Fix drop_vectorizer comment
jgpruitt Sep 3, 2024
ffe0fbb
Add vectorizer_errors table
jgpruitt Sep 3, 2024
3cafcbe
Add ai.chunking_recursive_character_text_splitter
jgpruitt Sep 3, 2024
3655553
Add tests for _validate_embedding
jgpruitt Sep 3, 2024
6562f02
Refactor config validation
jgpruitt Sep 3, 2024
bdce11c
Check that configs are objects
jgpruitt Sep 4, 2024
d9f65c5
Don't send http request on delete vectorizer
jgpruitt Sep 4, 2024
4643a64
Move test server to vectorizer dir
jgpruitt Sep 4, 2024
9849d9d
Rename some functions
jgpruitt Sep 4, 2024
6ec9527
Move grants to internal functions
jgpruitt Sep 4, 2024
1c9be13
Ensure that idempotent and incremental files are strictly ordered
jgpruitt Sep 4, 2024
2fd8339
Run tests as non-superuser
jgpruitt Sep 4, 2024
8795f72
Minor cleanup to test fixtures
jgpruitt Sep 4, 2024
aa092c7
Add tests for privileges
jgpruitt Sep 4, 2024
22b456c
Ensure that vectorizer is usable after dump/restore
jgpruitt Sep 4, 2024
79e9735
Fix lint-sql
jgpruitt Sep 4, 2024
13ce785
Add default separators for recursive chunking
jgpruitt Sep 4, 2024
9cba57e
Add enqueue_existing flag to create_vectorizer
jgpruitt Sep 4, 2024
7bf4cf7
Make postgres version parameterized
jgpruitt Sep 4, 2024
80d4da4
Don't build sql on make install. Use committed.
jgpruitt Sep 4, 2024
b8da296
Add 0.4.0 sql file for integration tests
jgpruitt Sep 4, 2024
5e83fec
Fix make help messages
jgpruitt Sep 4, 2024
e1d6b9f
Fix python formatting
jgpruitt Sep 4, 2024
bc22fd3
Be more strict with paths and working dirs in build for pip
jgpruitt Sep 5, 2024
7e047a0
Need more verbose pip
jgpruitt Sep 5, 2024
cca8b51
Python install improvements
jgpruitt Sep 5, 2024
8b80a51
Use --no-build-isolation when installing python lib
jgpruitt Sep 5, 2024
4f7238b
Fixing pip install for 3.10
jgpruitt Sep 5, 2024
48d26c4
Use version from __init__.py
jgpruitt Sep 5, 2024
c4e7ff6
Test the contents of the extension
jgpruitt Sep 9, 2024
4122e6a
Adjust privileges. Revoke from public. Grant to owner
jgpruitt Sep 10, 2024
d1be1d9
Clean up docker/build
jgpruitt Sep 11, 2024
35de12d
Formatting build.py
jgpruitt Sep 11, 2024
ab0372d
Change vectorizer GUC names
jgpruitt Sep 12, 2024
508c003
pass secrets in event payload
alejandrodnm Sep 12, 2024
defec5b
GUC rename
jgpruitt Sep 12, 2024
adff758
Bug fix for secrets
jgpruitt Sep 12, 2024
4456867
Default grant_to to tsdbadmin
jgpruitt Sep 12, 2024
e01c702
Make sure schema usage is granted too
jgpruitt Sep 12, 2024
07958c7
Create vector indexes in background job
jgpruitt Sep 12, 2024
2720e9c
Add pgvectorscale to docker image for testing
jgpruitt Sep 13, 2024
9ab992c
Make trigger func security definer
jgpruitt Sep 13, 2024
5b860f2
Bug fixes for vector index creation
jgpruitt Sep 13, 2024
c378a7d
Explain security definer on trigger in comment
jgpruitt Sep 13, 2024
cf4a830
Pretty comments
jgpruitt Sep 13, 2024
7550c50
Remove pgcron scheduling implementation
jgpruitt Sep 13, 2024
dfd4176
Fix issue with docker ps hanging
jgpruitt Sep 13, 2024
a4ae9fb
Remove columns param from formatting_python_template
jgpruitt Sep 13, 2024
79e22f7
Test diskann index creation
jgpruitt Sep 13, 2024
29c57a9
Test hnsw index creation
jgpruitt Sep 13, 2024
511ae7d
Add vectorizer processing config
jgpruitt Sep 16, 2024
028a72c
Add concurrency setting to processing config
jgpruitt Sep 16, 2024
1637705
fix: create test users by a superuser
jgpruitt Sep 17, 2024
ef73101
Remove drop of pgcron in tests
jgpruitt Sep 13, 2024
f177de5
fix: correct the pg_setting name for url
jgpruitt Sep 19, 2024
e292153
fix: correct the guc name in vectorizer test
jgpruitt Sep 19, 2024
2ef1ff3
fix: minor bug fixes to build.py
jgpruitt Sep 20, 2024
022cd47
Add openai_chat_complete_simple (and tests)
akulkarni Sep 24, 2024
bfcf608
fix: Add support for recursive_character_text_splitter to test server
jgpruitt Sep 20, 2024
0a7a44e
fix: raise exception if no pk on source
jgpruitt Sep 30, 2024
7024e34
fix: ensure column to chunk it text-like type
jgpruitt Sep 30, 2024
39b63f5
fix: update test actuals for new function
jgpruitt Sep 30, 2024
c6d30c7
feat: execute multiple vectorizers for large queue
jgpruitt Sep 30, 2024
69c1851
fix: only count what we have to
jgpruitt Sep 30, 2024
2469bcc
fix: build sql
jgpruitt Sep 30, 2024
7daaa12
feat: documentation for version 0.4.0
jgpruitt Sep 17, 2024
8ae6a71
chore: add debug msg and null-safe check to vec job
jgpruitt Oct 1, 2024
4b4b386
Add ts-vector team to CODEOWNERS
adolsalamanca Oct 1, 2024
6d1dead
chore: move extension python to subdir
jgpruitt Oct 1, 2024
30e0533
chore: reorganize and improve handling of python deps
jgpruitt Oct 1, 2024
2b27a9d
feat: install/uninstall/clean make targets for vectorizer
jgpruitt Oct 2, 2024
7dfd6f2
feat: add dockerfile for vectorizer and related make targets
jgpruitt Oct 2, 2024
547d64d
chore: update make help
jgpruitt Oct 2, 2024
da81af8
feat: use advisory lock to protect against redundant index builds
jgpruitt Oct 3, 2024
c29b689
fix: index creation bug when options specified
jgpruitt Oct 3, 2024
a23bc41
feat: test index creation under concurrency
jgpruitt Oct 3, 2024
e8ca37b
fix: do not install vectorizer cli by default
jgpruitt Oct 4, 2024
c32e4db
fix: make setup.cfg compatible with python 3.10
jgpruitt Oct 4, 2024
8070011
fix: remove pip arg not supported on old versions
jgpruitt Oct 4, 2024
4f3f2c6
feat: add config option to only build vec idx when q empty
jgpruitt Oct 4, 2024
161b48e
feat: add vectorizer cli tool
jgpruitt Oct 7, 2024
a7180b0
fix: fix security issues with openai_chat_complete_simple
jgpruitt Oct 8, 2024
fd4a5a5
fix: small sql lint issues
jgpruitt Oct 8, 2024
09883c9
feat: make certain tests run by default
jgpruitt Oct 7, 2024
57df731
feat: add ci config
jgpruitt Oct 7, 2024
a6b616d
refactor: use monorepo structure (#101)
alejandrodnm Oct 10, 2024
eb75b36
feat: detect naming collisions and raise helpful errors
jgpruitt Oct 10, 2024
153bb90
chore: change scheduling default to 5 minutes
jgpruitt Oct 10, 2024
a1c3201
refactor: update test image path
jgpruitt Oct 10, 2024
108c9d4
chore: rename processing_cloud_functions to processing_default
jgpruitt Oct 10, 2024
d75af9d
feat: if scheduling is none, indexing must be none
jgpruitt Oct 10, 2024
0f8ef77
chore: add docker-compose
alejandrodnm Oct 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.env
.git
.github
.idea
.idea
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
* @jgpruitt @cevian @avthars
* @jgpruitt @cevian @avthars @timescale/ts-vector
/.github/ @jgpruitt @cevian @avthars
62 changes: 62 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: CI

on: [push, pull_request, workflow_dispatch]
permissions:
contents: read

jobs:
build-and-test:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
cache: "pip" # caching pip dependencies

- run: pip install -r requirements-dev.txt

- name: Verify Docker installation
run: |
docker --version
docker info

- name: Build Docker image
run: make docker-build

- name: Run Docker container
run: make docker-run

- name: Build
run: docker exec pgai make build

- name: Lint SQL and Python
run: docker exec pgai make lint

- name: Check Python Formatting
run: docker exec pgai make format-py

- name: Install extension and vectorizer
run: |
docker exec pgai make install
docker exec pgai make install-vec

- name: Run test server
run: docker exec -d pgai make test-server

- name: Run tests
run: docker exec pgai make test

- name: Run vectorizer tests
run: make test-vectorizer

- name: Run type check
run: make type-check-py

- name: Stop and remove Docker container
run: |
make docker-stop
make docker-rm
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
.env
.idea
.venv
build
*.egg-info
__pycache__
projects/extension/sql/ai--*.sql
projects/extension/tests/dump_restore/describe_objects.sql
projects/extension/tests/dump_restore/describe_schemas.sql
projects/extension/tests/dump_restore/dump.sql
projects/extension/tests/dump_restore/src.snapshot
projects/extension/tests/dump_restore/dst.snapshot
projects/extension/tests/*/*.actual
Loading
Loading