Skip to content

Commit

Permalink
chore(langchain): fix flaky cassette tests, skipping logic (#9768)
Browse files Browse the repository at this point in the history
This PR attempts to address the flakiness and general (lack of)
reliability of the LangChain test suite, and removes the flaky test
markers from most langchain tests. No functionality has been changed,
this PR focuses only on improving our test suite to be less flaky and be
a better signal.

Problems with LangChain test suite:
1. Tests are randomly flaking due to cassettes being read incorrectly
(See related [issue](kevin1024/vcrpy#834))
2. Tests are completely skipped except for patch tests. This seems like
something to do with our skipping logic by looking at langchain versions
and Python versions (we skip a few test cases in Python 3.9 due to
unnecessary cassette files that are required specifically for 3.9).

This PR solves the above problems by:
1. Pinning `vcrpy` to version `5.1.0` which is the version prior to the
linked issue being introduced
2. Rewriting the langchain version skipping logic to rely specifically
on the Langchain module version (instead of reusing the
`PATCH_LANGCHAIN_V0` constant from the patch file which appeared to not
be truthful)
3. Change the Python version skip checks to only use major/minor
versions rather than checking against `(3, 10, 0)` as this may result in
some edge cases. Do not skip Python 3.9 testing on
`test_langchain_community.py` or community tests on
`test_langchain_llmobs.py`.

## Checklist

- [x] The PR description includes an overview of the change
- [x] The PR description articulates the motivation for the change
- [x] The change includes tests OR the PR description describes a
testing strategy
- [x] The PR description notes risks associated with the change, if any
- [x] Newly-added code is easy to change
- [x] The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- [x] The change includes or references documentation updates if
necessary
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Newly-added code is easy to change
- [x] Release note makes sense to a user of the library
- [x] If necessary, author has acknowledged and discussed the
performance implications of this PR as reported in the benchmarks PR
comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
  • Loading branch information
Yun-Kim authored Jul 30, 2024
1 parent 16e8cce commit 7934297
Show file tree
Hide file tree
Showing 83 changed files with 1,679 additions and 6,157 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/1a15059.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/11063bf.in
#
ai21==2.9.2
ai21-tokenizer==0.11.2
Expand Down Expand Up @@ -32,7 +32,7 @@ h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
httpx-sse==0.4.0
huggingface-hub==0.24.1
huggingface-hub==0.24.2
hypothesis==6.45.0
idna==3.7
iniconfig==2.0.0
Expand Down Expand Up @@ -64,14 +64,14 @@ pluggy==1.5.0
psutil==6.0.0
pydantic==2.8.2
pydantic-core==2.20.1
pytest==8.3.1
pytest==8.3.2
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pyyaml==6.0.1
regex==2024.5.15
regex==2024.7.24
requests==2.32.3
s3transfer==0.10.2
sentencepiece==0.2.0
Expand All @@ -87,6 +87,6 @@ types-requests==2.32.0.20240712
typing-extensions==4.12.2
typing-inspect==0.9.0
urllib3==2.0.7
vcrpy==6.0.1
vcrpy==5.1.0
wrapt==1.16.0
yarl==1.9.4
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/1fda250.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/16c3b9f.in
#
ai21==1.3.4
aiohttp==3.9.5
Expand All @@ -23,7 +23,7 @@ filelock==3.15.4
frozenlist==1.4.1
fsspec==2024.6.1
greenlet==3.0.3
huggingface-hub==0.24.0
huggingface-hub==0.24.2
hypothesis==6.45.0
idna==3.7
importlib-metadata==6.11.0
Expand All @@ -50,14 +50,14 @@ pinecone-client==2.2.4
pluggy==1.5.0
psutil==6.0.0
pydantic==1.10.17
pytest==8.2.2
pytest==8.3.2
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pyyaml==6.0.1
regex==2024.5.15
regex==2024.7.24
requests==2.32.3
six==1.16.0
sniffio==1.3.1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/11f9de5.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/1761cfc.in
#
ai21==2.9.2
ai21-tokenizer==0.11.2
Expand Down Expand Up @@ -32,7 +32,7 @@ h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
httpx-sse==0.4.0
huggingface-hub==0.24.1
huggingface-hub==0.24.2
hypothesis==6.45.0
idna==3.7
iniconfig==2.0.0
Expand All @@ -45,7 +45,7 @@ langchain-aws==0.1.6
langchain-cohere==0.1.8
langchain-core==0.2.0
langchain-openai==0.1.7
langchain-pinecone==0.1.2
langchain-pinecone==0.1.3
langchain-text-splitters==0.2.1
langsmith==0.1.93
marshmallow==3.21.3
Expand All @@ -54,25 +54,26 @@ multidict==6.0.5
mypy-extensions==1.0.0
numexpr==2.8.5
numpy==1.26.4
openai==1.37.0
openai==1.37.1
opentracing==2.4.0
orjson==3.10.6
packaging==23.2
parameterized==0.9.0
pinecone-client==4.1.2
pinecone-client==5.0.0
pinecone-plugin-inference==1.0.2
pinecone-plugin-interface==0.0.7
pluggy==1.5.0
psutil==6.0.0
pydantic==2.8.2
pydantic-core==2.20.1
pytest==8.3.1
pytest==8.3.2
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pyyaml==6.0.1
regex==2024.5.15
regex==2024.7.24
requests==2.32.3
s3transfer==0.10.2
sentencepiece==0.2.0
Expand All @@ -89,6 +90,6 @@ types-requests==2.32.0.20240712
typing-extensions==4.12.2
typing-inspect==0.9.0
urllib3==2.0.7
vcrpy==6.0.1
vcrpy==5.1.0
wrapt==1.16.0
yarl==1.9.4
27 changes: 14 additions & 13 deletions .riot/requirements/5425133.txt → .riot/requirements/18bc2ac.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/5425133.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/18bc2ac.in
#
ai21==2.9.2
ai21-tokenizer==0.11.2
Expand All @@ -12,8 +12,8 @@ annotated-types==0.7.0
anthropic==0.31.2
anyio==4.4.0
attrs==23.2.0
boto3==1.34.146
botocore==1.34.146
boto3==1.34.149
botocore==1.34.149
certifi==2024.7.4
charset-normalizer==3.3.2
cohere==5.6.2
Expand All @@ -31,7 +31,7 @@ h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
httpx-sse==0.4.0
huggingface-hub==0.24.1
huggingface-hub==0.24.2
hypothesis==6.45.0
idna==3.7
iniconfig==2.0.0
Expand All @@ -40,14 +40,14 @@ jmespath==1.0.1
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.2.11
langchain-anthropic==0.1.20
langchain-anthropic==0.1.21
langchain-aws==0.1.12
langchain-cohere==0.1.9
langchain-community==0.2.10
langchain-core==0.2.23
langchain-core==0.2.24
langchain-experimental==0.0.63
langchain-openai==0.1.17
langchain-pinecone==0.1.2
langchain-openai==0.1.19
langchain-pinecone==0.1.3
langchain-text-splitters==0.2.2
langsmith==0.1.93
marshmallow==3.21.3
Expand All @@ -56,27 +56,28 @@ multidict==6.0.5
mypy-extensions==1.0.0
numexpr==2.8.5
numpy==1.26.4
openai==1.37.0
openai==1.37.1
opentracing==2.4.0
orjson==3.10.6
packaging==24.1
pandas==2.2.2
parameterized==0.9.0
pinecone-client==4.1.2
pinecone-client==5.0.0
pinecone-plugin-inference==1.0.2
pinecone-plugin-interface==0.0.7
pluggy==1.5.0
psutil==6.0.0
pydantic==2.8.2
pydantic-core==2.20.1
pytest==8.3.1
pytest==8.3.2
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pytz==2024.1
pyyaml==6.0.1
regex==2024.5.15
regex==2024.7.24
requests==2.32.3
s3transfer==0.10.2
sentencepiece==0.2.0
Expand All @@ -94,6 +95,6 @@ typing-extensions==4.12.2
typing-inspect==0.9.0
tzdata==2024.1
urllib3==2.2.2
vcrpy==6.0.1
vcrpy==5.1.0
wrapt==1.16.0
yarl==1.9.4
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/1a8a6e1.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/19f2225.in
#
ai21==2.9.2
ai21-tokenizer==0.11.2
Expand Down Expand Up @@ -33,7 +33,7 @@ h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
httpx-sse==0.4.0
huggingface-hub==0.24.1
huggingface-hub==0.24.2
hypothesis==6.45.0
idna==3.7
iniconfig==2.0.0
Expand Down Expand Up @@ -65,14 +65,14 @@ pluggy==1.5.0
psutil==6.0.0
pydantic==2.8.2
pydantic-core==2.20.1
pytest==8.3.1
pytest==8.3.2
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pyyaml==6.0.1
regex==2024.5.15
regex==2024.7.24
requests==2.32.3
s3transfer==0.10.2
sentencepiece==0.2.0
Expand All @@ -89,6 +89,6 @@ types-requests==2.32.0.20240712
typing-extensions==4.12.2
typing-inspect==0.9.0
urllib3==2.0.7
vcrpy==6.0.1
vcrpy==5.1.0
wrapt==1.16.0
yarl==1.9.4
27 changes: 14 additions & 13 deletions .riot/requirements/8dd750a.txt → .riot/requirements/1ec1dbf.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --no-annotate .riot/requirements/8dd750a.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/1ec1dbf.in
#
ai21==2.9.2
ai21-tokenizer==0.11.2
Expand All @@ -13,8 +13,8 @@ anthropic==0.31.2
anyio==4.4.0
async-timeout==4.0.3
attrs==23.2.0
boto3==1.34.146
botocore==1.34.146
boto3==1.34.149
botocore==1.34.149
certifi==2024.7.4
charset-normalizer==3.3.2
cohere==5.6.2
Expand All @@ -32,7 +32,7 @@ h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
httpx-sse==0.4.0
huggingface-hub==0.24.1
huggingface-hub==0.24.2
hypothesis==6.45.0
idna==3.7
iniconfig==2.0.0
Expand All @@ -41,14 +41,14 @@ jmespath==1.0.1
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.2.11
langchain-anthropic==0.1.20
langchain-anthropic==0.1.21
langchain-aws==0.1.12
langchain-cohere==0.1.9
langchain-community==0.2.10
langchain-core==0.2.23
langchain-core==0.2.24
langchain-experimental==0.0.63
langchain-openai==0.1.17
langchain-pinecone==0.1.2
langchain-openai==0.1.19
langchain-pinecone==0.1.3
langchain-text-splitters==0.2.2
langsmith==0.1.93
marshmallow==3.21.3
Expand All @@ -57,27 +57,28 @@ multidict==6.0.5
mypy-extensions==1.0.0
numexpr==2.8.5
numpy==1.26.4
openai==1.37.0
openai==1.37.1
opentracing==2.4.0
orjson==3.10.6
packaging==24.1
pandas==2.2.2
parameterized==0.9.0
pinecone-client==4.1.2
pinecone-client==5.0.0
pinecone-plugin-inference==1.0.2
pinecone-plugin-interface==0.0.7
pluggy==1.5.0
psutil==6.0.0
pydantic==2.8.2
pydantic-core==2.20.1
pytest==8.3.1
pytest==8.3.2
pytest-asyncio==0.23.7
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-randomly==3.10.1
python-dateutil==2.9.0.post0
pytz==2024.1
pyyaml==6.0.1
regex==2024.5.15
regex==2024.7.24
requests==2.32.3
s3transfer==0.10.2
sentencepiece==0.2.0
Expand All @@ -96,6 +97,6 @@ typing-extensions==4.12.2
typing-inspect==0.9.0
tzdata==2024.1
urllib3==2.2.2
vcrpy==6.0.1
vcrpy==5.1.0
wrapt==1.16.0
yarl==1.9.4
Loading

0 comments on commit 7934297

Please sign in to comment.