Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.37.0: Polars conv #61

Merged
merged 79 commits into from
Aug 8, 2023
Merged
Show file tree
Hide file tree
Changes from 73 commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
e37b56e
add polars dep, upgrade pyarrow min version to 8.0.0 and cfb package …
saiemgilani Jun 25, 2023
f493ca8
now using `requests` module with req options/parameters passed to `d…
saiemgilani Jun 25, 2023
e83fa07
added some timing/memory usage decorators for profiling
saiemgilani Jun 25, 2023
b42ced2
updates to add `return_as_pandas` parameters
saiemgilani Jul 9, 2023
b93274a
minor cfb package updates
saiemgilani Jul 10, 2023
b77cd8b
Function `espn_cfb_game_rosters()` added.
saiemgilani Jul 10, 2023
ecfb2ca
Update mbb package to use polars
saiemgilani Jul 10, 2023
c79a0a1
Function `espn_nba_game_rosters()` added
saiemgilani Jul 10, 2023
3234bc0
Update mbb_loaders.py
saiemgilani Jul 10, 2023
713e07a
Missing imports added
saiemgilani Jul 10, 2023
f5ad90d
simplifying the returns on the helper functions in the `mbb.espn_mbb_…
saiemgilani Jul 10, 2023
738de33
Update nba package to use polars
saiemgilani Jul 10, 2023
6b1b7dc
Documentation examples
saiemgilani Jul 10, 2023
6e19ff6
Updated nfl package to use polars
saiemgilani Jul 10, 2023
b71f572
simplifying the returns on the helper functions in the `cfb.espn_cfb_…
saiemgilani Jul 10, 2023
c711d9e
Update nhl package to use polars
saiemgilani Jul 10, 2023
2caee12
Update wbb to polars
saiemgilani Jul 11, 2023
1c6d8c6
Function nfl.nfl_game_rosters() added
saiemgilani Jul 11, 2023
daf2c7c
Update nhl_pbp.py
saiemgilani Jul 14, 2023
3d3aa23
update init files
saiemgilani Jul 14, 2023
85cad44
error in game_seconds_remaining
saiemgilani Jul 14, 2023
a9bdba2
Function `wnba.wnba_game_rosters()` added.
saiemgilani Jul 14, 2023
7dac003
Update wnba package to polars
saiemgilani Jul 14, 2023
9042beb
Update test_pbp.py
saiemgilani Jul 14, 2023
44d88d8
Update retrosheet.py
saiemgilani Jul 14, 2023
5d0fe7d
update requests for mlb module and add to init
saiemgilani Jul 15, 2023
b75333b
Function `nhl.nhl_game_rosters()` added
saiemgilani Jul 15, 2023
b9e1ef0
Update test_pbp to account for polars conventions
saiemgilani Jul 15, 2023
eead682
minor nfl_pbp updates (WIP)
saiemgilani Jul 15, 2023
96d45b7
Update CHANGELOG.md
saiemgilani Jul 15, 2023
06e5b0e
update docs
saiemgilani Jul 15, 2023
3eaf954
Update tests.yml
saiemgilani Jul 15, 2023
5519b49
Update python-publish.yml
saiemgilani Jul 15, 2023
8e6290d
tests
saiemgilani Jul 15, 2023
22fd209
Update tests.yml
saiemgilani Jul 15, 2023
6ee26ba
pl.Int64 changes per comment
saiemgilani Jul 26, 2023
4729354
black/flake/isort formatting, minor documentation details
saiemgilani Jul 26, 2023
a98074f
black
saiemgilani Jul 27, 2023
164afb1
ep_final_names/wp_final_names were not imported
saiemgilani Jul 27, 2023
f7783d0
flake things
saiemgilani Jul 27, 2023
0a748af
Update .flake8
saiemgilani Jul 27, 2023
2eede28
Update .flake8
saiemgilani Jul 31, 2023
8f05885
Update pyproject.toml
saiemgilani Jul 31, 2023
52fbbc1
Update wnba_loaders.py
saiemgilani Jul 31, 2023
32ca8f5
return_as_pandas=False
saiemgilani Jul 31, 2023
5302e8c
docs
saiemgilani Jul 31, 2023
718ecab
mlb
saiemgilani Jul 31, 2023
138e546
schedule function drops leaders key
saiemgilani Jul 31, 2023
285f2e3
changelog
saiemgilani Jul 31, 2023
cb71e1a
Function `cfb.load_cfb_betting_lines()` added.
saiemgilani Jul 31, 2023
8f2acb4
docs
saiemgilani Jul 31, 2023
9e2c8ac
schedule
saiemgilani Jul 31, 2023
1b1c254
messing around with schedule things... it's a working mess
saiemgilani Jul 31, 2023
52cb72a
updating test names
saiemgilani Jul 31, 2023
635e043
nfl_pbp blindly updated to match cfb_pbp
saiemgilani Jul 31, 2023
4fe1399
Update .gitignore
saiemgilani Jul 31, 2023
3651508
nfl_pbp was missing a few params
saiemgilani Jul 31, 2023
5b21788
removing unused vars
saiemgilani Jul 31, 2023
892ec65
updating schedule documentation per PR comments
saiemgilani Aug 1, 2023
5171d27
adding tests/function renaming
saiemgilani Aug 1, 2023
652f7bd
fixing timeouts and formatting per PR comments
saiemgilani Aug 1, 2023
c5d110a
remove mlb
saiemgilani Aug 1, 2023
444b4c1
mlb submodule moved to archive
saiemgilani Aug 1, 2023
1df7a7c
sidebars
saiemgilani Aug 1, 2023
2ae31fb
add cache documentation to teams functions
saiemgilani Aug 1, 2023
935073c
cast sequenceNumber to pl.Int32
saiemgilani Aug 1, 2023
0a15b22
None case for formatted (column) regex extraction, punt block return …
saiemgilani Aug 1, 2023
e60fa67
slight modifications for no
saiemgilani Aug 1, 2023
bd54587
Update nfl_pbp.py
saiemgilani Aug 1, 2023
dd1e6c7
more filtering to stop pipeline if len(plays_df) == 0
saiemgilani Aug 1, 2023
aab7e94
updating default overUnders basketball sports
saiemgilani Aug 1, 2023
89018c2
adding a to in case of early termination of run_processing_pipeline
saiemgilani Aug 1, 2023
e5b8ecd
adding a to in case of early termination of run_processing_pipeline
saiemgilani Aug 1, 2023
1ef7a15
updating pickcenter methods to be a bit more robust
saiemgilani Aug 2, 2023
a2fcba0
add janitor functions to polars dataframe namespace
saiemgilani Aug 8, 2023
e0b110d
no_espn_data error added
saiemgilani Aug 8, 2023
259ec39
configs updated to use fstrings
saiemgilani Aug 8, 2023
d1a0fa6
extract event parsing, janitor clean names
saiemgilani Aug 8, 2023
e45322d
add corrupt pbp check for completed games
saiemgilani Aug 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[flake8]
max-line-length = 120
max-complexity = 18
select = B,C,E,F,W,T4,B9
extend-ignore = E203, E265, E266, E400, E501, E712, W503, B905, B907
enable-extensions = G
import-order-style = google
application-import-names = sportsdataverse,test
6 changes: 3 additions & 3 deletions .github/workflows/python-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: '3.x'
python-version: '3.9'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: '3.9'
- name: Install dependencies
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,7 @@ docs/node_modules
build
dist
*.ipynb
.ipynb_checkpoints
./.ipynb_checkpoints
*/.ipynb_checkpoints
*.egg-info
47 changes: 46 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,52 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
# - repo: local
# hooks:
# - id: update-docs
# name: update-docs
# entry: bash create_docs.sh
# language: system
# types: [python]
# pass_filenames: false

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.0.1
hooks:
- id: trailing-whitespace
- id: trailing-whitespace
- id: check-merge-conflict
- id: check-ast
- id: check-toml
- id: check-json
- id: check-xml
- id: check-yaml
- id: check-symlinks
- id: check-vcs-permalinks
- id: end-of-file-fixer
- id: requirements-txt-fixer
- id: check-added-large-files
args: [--maxkb=10000]
- id: debug-statements

- repo: https://github.com/psf/black
rev: '23.1.0'
hooks:
- id: black
- repo: https://github.com/pycqa/isort
rev: '5.12.0'
hooks:
- id: isort
- repo: https://github.com/hadialqattan/pycln
rev: 'v2.1.5'
hooks:
- id: pycln
args: ['.', "--all"]
# - repo: https://github.com/pycqa/flake8
# rev: '3.9.2'
# hooks:
# - id: flake8
# - repo: https://github.com/pycqa/pydocstyle
# rev: '6.3.0'
# hooks:
# - id: pydocstyle
# args: ['.', '--ignore=D100,D203,D405']
11 changes: 10 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@

## 0.0.36-7 Release: July 9, 2023
- Switched most under the hood dataframe operations to use the python `polars` library and many functions now have a parameter `return_as_pandas` which defaults to `False` but can be set to `True` to return a pandas dataframe instead of a polars dataframe. This is a **breaking change.**
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for convenience there should probably be a way to set this globally

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't honestly know how to do that outside of a config file? It's certainly worth looking into

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you know of a way to do this that doesn't cause significant work, I'll do it but I'm about dunnn with this overhaul so don't expect this one to get resolved

- Added `**kwargs` which pass arguments to the `dl_utils.download()` function, including `headers`, `proxy`, `timeout` (default 30s), `num_retries` (default = 15), `logger` (default = None)
- Function `espn_cfb_game_rosters()` added.
- Function `espn_nba_game_rosters()` added.
- Function `espn_nfl_game_rosters()` added.
- Function `espn_nhl_game_rosters()` added.
- Function `espn_wbb_game_rosters()` added.
- Function `espn_wnba_game_rosters()` added.
- Function `load_cfb_betting_lines()` added (only 2006 through 2019).

## 0.0.34-35 Release: May 7-9, 2023
- Reconfigured some imports
Expand Down
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ recursive-include *.model
recursive-include bin *.py *.cmd *.zip
recursive-include docs *.py *.rst *.css Makefile
recursive-include features *.feature *.py
recursive-include paver_ext *.py
recursive-include paver_ext *.py
Binary file modified Sphinx-docs/_build/doctrees/environment.pickle
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.cfb.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.mbb.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.mlb.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.nba.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.nfl.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.nhl.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.wbb.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/sportsdataverse.wnba.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/tests.cfb.doctree
Binary file not shown.
Binary file modified Sphinx-docs/_build/doctrees/tests.mbb.doctree
Binary file not shown.
52 changes: 20 additions & 32 deletions Sphinx-docs/_build/markdown/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ contain the root `toctree` directive. -->
* Submodules


* sportsdataverse.cfb.cfb_game_rosters module


* sportsdataverse.cfb.cfb_loaders module


Expand Down Expand Up @@ -64,43 +67,13 @@ contain the root `toctree` directive. -->
* Module contents


* sportsdataverse.mlb package
* sportsdataverse.nba package


* Submodules


* sportsdataverse.mlb.mlb_loaders module


* sportsdataverse.mlb.mlbam_games module


* sportsdataverse.mlb.mlbam_players module


* sportsdataverse.mlb.mlbam_reports module


* sportsdataverse.mlb.mlbam_stats module


* sportsdataverse.mlb.mlbam_teams module


* sportsdataverse.mlb.retrosheet module


* sportsdataverse.mlb.retrosplits module


* Module contents


* sportsdataverse.nba package


* Submodules
* sportsdataverse.nba.nba_game_rosters module


* sportsdataverse.nba.nba_loaders module
Expand All @@ -127,6 +100,9 @@ contain the root `toctree` directive. -->
* sportsdataverse.nfl.model_vars module


* sportsdataverse.nfl.nfl_game_rosters module


* sportsdataverse.nfl.nfl_games module


Expand Down Expand Up @@ -154,6 +130,9 @@ contain the root `toctree` directive. -->
* sportsdataverse.nhl.nhl_api module


* sportsdataverse.nhl.nhl_game_rosters module


* sportsdataverse.nhl.nhl_loaders module


Expand All @@ -175,6 +154,9 @@ contain the root `toctree` directive. -->
* Submodules


* sportsdataverse.wbb.wbb_game_rosters module


* sportsdataverse.wbb.wbb_loaders module


Expand All @@ -196,6 +178,9 @@ contain the root `toctree` directive. -->
* Submodules


* sportsdataverse.wnba.wnba_game_rosters module


* sportsdataverse.wnba.wnba_loaders module


Expand All @@ -217,6 +202,9 @@ contain the root `toctree` directive. -->
* sportsdataverse.config module


* sportsdataverse.decorators module


* sportsdataverse.dl_utils module


Expand Down
52 changes: 20 additions & 32 deletions Sphinx-docs/_build/markdown/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
* Submodules


* sportsdataverse.cfb.cfb_game_rosters module


* sportsdataverse.cfb.cfb_loaders module


Expand Down Expand Up @@ -55,43 +58,13 @@
* Module contents


* sportsdataverse.mlb package
* sportsdataverse.nba package


* Submodules


* sportsdataverse.mlb.mlb_loaders module


* sportsdataverse.mlb.mlbam_games module


* sportsdataverse.mlb.mlbam_players module


* sportsdataverse.mlb.mlbam_reports module


* sportsdataverse.mlb.mlbam_stats module


* sportsdataverse.mlb.mlbam_teams module


* sportsdataverse.mlb.retrosheet module


* sportsdataverse.mlb.retrosplits module


* Module contents


* sportsdataverse.nba package


* Submodules
* sportsdataverse.nba.nba_game_rosters module


* sportsdataverse.nba.nba_loaders module
Expand All @@ -118,6 +91,9 @@
* sportsdataverse.nfl.model_vars module


* sportsdataverse.nfl.nfl_game_rosters module


* sportsdataverse.nfl.nfl_games module


Expand Down Expand Up @@ -145,6 +121,9 @@
* sportsdataverse.nhl.nhl_api module


* sportsdataverse.nhl.nhl_game_rosters module


* sportsdataverse.nhl.nhl_loaders module


Expand All @@ -166,6 +145,9 @@
* Submodules


* sportsdataverse.wbb.wbb_game_rosters module


* sportsdataverse.wbb.wbb_loaders module


Expand All @@ -187,6 +169,9 @@
* Submodules


* sportsdataverse.wnba.wnba_game_rosters module


* sportsdataverse.wnba.wnba_loaders module


Expand All @@ -208,6 +193,9 @@
* sportsdataverse.config module


* sportsdataverse.decorators module


* sportsdataverse.dl_utils module


Expand Down
Loading