diff --git a/.gitignore b/.gitignore index f6eb32e..19c8371 100644 --- a/.gitignore +++ b/.gitignore @@ -2,7 +2,7 @@ *.pyc *.egg-info -logs .vscode __pycache__ -chess_transformers/checkpoints \ No newline at end of file +checkpoints +logs \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..a8c4302 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,32 @@ +# Change Log + +## Unreleased (v0.2.0) + +### Added + +* **`ChessTransformerEncoderFT`** is an encoder-only transformer that predicts source (*From*) and destination squares (*To*) squares for the next half-move, instead of the half-move in UCI notation. +* [*CT-EFT-20*](https://github.com/sgrvinod/chess-transformers#ct-eft-20) is a new trained model of this type with about 20 million parameters. +* **`ChessDatasetFT`** is a PyTorch dataset class for this model type. +* [**`chess_transformer.data.levels`**](https://github.com/sgrvinod/chess-transformers/blob/main/chess_transformers/data/levels.py) provides a standardized vocabulary (with indices) for oft-used categorical variables. All models and datasets will hereon use this standard vocabulary instead of a dataset-specific vocabulary. + +### Changed + +* The [*LE1222*](https://github.com/sgrvinod/chess-transformers#le1222) and [*LE1222x*](https://github.com/sgrvinod/chess-transformers#le1222x) datasets no longer have their own vocabularies or vocabulary files. Instead, they use the standard vocabulary from **`chess_transformer.data.levels`**. +* The [*LE1222*](https://github.com/sgrvinod/chess-transformers#le1222) and [*LE1222x*](https://github.com/sgrvinod/chess-transformers#le1222x) datasets have been re-encoded with indices corresponding to the standard vocabulary. Earlier versions or downloads of these datasets are no longer valid for use with this library. +* The row index at which the validation split begins in each dataset is now stored as an attribute of the **`encoded_data`** table in the corresponding H5 file, instead of in a separate JSON file. +* Models [*CT-E-20*](https://github.com/sgrvinod/chess-transformers#ct-e-20) and [*CT-ED-45*](https://github.com/sgrvinod/chess-transformers#ct-ed-45) already trained with a non-standard, dataset-specific vocabulary have been refactored for use with the standard vocabulary. Earlier versions or downloads of these models are no longer valid for use with this library. +* The field **`move_sequence`** in the H5 tables has now been renamed to **`moves`**. +* The field **`move_sequence_length`** in the H5 tables has now been renamed to **`length`**. +* The **`load_assets()`** function has been renamed to **`load_model()`** and it no longer returns a vocabulary — only the model. +* The **`chess_transformers/eval`** folder has been renamed to [**`chess_transformers/evaluate`**](https://github.com/sgrvinod/chess-transformers/tree/main/chess_transformers/evaluate). +* The Python notebook **`lichess_eval.ipynb`** has been converted to a Python script [**`evaluate.py`**](https://github.com/sgrvinod/chess-transformers/blob/main/chess_transformers/evaluate/evaluation.py), which runs much faster for evaluation. +* Fairy Stockfish is now run on 8 threads and with a hash table of size 8 GB during evaluation instead of 1 thread and 16 MB respectively, which makes it a more challenging opponent. +* Evaluation results have been recomputed for [*CT-E-20*](https://github.com/sgrvinod/chess-transformers#ct-e-20) and [*CT-ED-45*](https://github.com/sgrvinod/chess-transformers#ct-ed-45) against this stronger Fairy Stockfish — they naturally fare worse. + +### Removed + +* The environment variable **`CT_LOGS_FOLDER`** no longer needs to be set before training a model. Training logs will now always be saved to **`chess_transformers/training/logs`**. +* The environment variable **`CT_CHECKPOINT_FOLDER`** no longer needs to be set before training a model. Checkpoints will now always be saved to **`chess_transformers/checkpoints`**. +* The environment variable **`CT_EVAL_GAMES_FOLDER`** no longer needs to be set before evaluating a model. Evaluation games will now always be saved to **`chess_transformers/evaluate/games`**. + + diff --git a/README.md b/README.md index 9383830..6291e60 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@
Teaching transformers to play chess
- ++ +
+ +At each strength level of the chess engine, $n=1000$ games are played by the model, i.e. $500$ games each with black and white pieces. + +Win ratios and the difference between the Elo rating of the model and the chess engine are calculated from these games' outcomes. + ++ +
-| Model Name | # Params | Training Data | Architecture | Predictions | -| :-------------------------: | :------: | :---------------------: | :---------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------: | -| [***CT-E-20***](#ct-e-20) | 20M | [***LE1222***](#le1222) | Transformer encoder only | Best next half-move (or ply)+ +
+ +This is essentially a sequence (or image) labeling task, where the sequence is the current state of the chessboard, and each square competes to be labeled as the *From* or *To* square. + +*CT-E-20* contains about 20 million parameters. + +```python +from chess_transformers.play import load_model +from chess_transformers.configs import import_config + +CONFIG = import_config("CT-EFT-20") +model = load_model(CONFIG) +``` + +You do not need to download the model checkpoint manually. It will be downloaded automatically if required. + +#### Model Strength + +*CT-EFT-20* was evaluated against the [Fairy Stockfish](https://github.com/fairy-stockfish/Fairy-Stockfish) chess engine at various strength levels [as predefined](https://github.com/lichess-org/fishnet/blob/dc4be23256e3e5591578f0901f98f5835a138d73/src/api.rs#L224) for use in the popular Stockfish chess bots on Lichess. The engine is run on an AMD Ryzen 7 3800X 8-Core Processor, with 8 CPU threads, and a hash table size of 8 GB. All other engine parameters are at their default values. + +These evaluation games can be viewed [here](chess_transformers/eval/games/CT-EFT-20/). + +| Strength Level | Games | Wins | Losses | Draws | Win Ratio | Elo Difference | Likelihood of Superiority | +| :------------: | :---: | :---: | :----: | :---: | :-------------------------: | :-----------------------: | :-----------------------: | +| $LL$ | $n$ | $w$ | $l$ | $d$ | $\frac{w + \frac{d}{2}}{n}$ | $\Delta_{Elo}$ | $LOS$ | +| **1** | 1000 | 994 | 0 | 6 | **99.70%** | 1008.63Name | Default | Minimum | Maximum | Variants |
---|---|---|---|---|
Protocol | uci | uci, usi, ucci, ucicyclone, xboard | ||
Debug Log File | ||||
Threads | 1 | 1 | 512 | |
Hash | 16 | 1 | 33554432 | |
Clear Hash | ||||
Ponder | False | |||
MultiPV | 1 | 1 | 500 | |
Skill Level | 20 | -20 | 20 | |
Move Overhead | 10 | 0 | 5000 | |
Slow Mover | 100 | 10 | 1000 | |
nodestime | 0 | 0 | 10000 | |
UCI_Chess960 | False | |||
UCI_Variant | chess | 3check, 5check, ai-wok, almost, amazon, antichess, armageddon, asean,\n", - "ataxx, atomic, breakthrough, bughouse, cambodian, capablanca,\n", - "capahouse, caparandom, centaur, chancellor, chaturanga, chess,\n", - "chessgi, chigorin, clobber, clobber10, codrus, coregal, courier,\n", - "crazyhouse, dobutsu, embassy, euroshogi, extinction, fairy,\n", - "fischerandom, gardner, giveaway, gorogoro, gothic, grand, grasshopper,\n", - "hoppelpoppel, horde, janggi, janggicasual, janggimodern,\n", - "janggitraditional, janus, jesonmor, judkins, karouk, kinglet,\n", - "kingofthehill, knightmate, koedem, kyotoshogi, loop, losalamos,\n", - "losers, makpong, makruk, manchu, micro, mini, minishogi, minixiangqi,\n", - "modern, newzealand, nightrider, nocastle, nocheckatomic, normal,\n", - "okisakishogi, opulent, placement, pocketknight, racingkings, seirawan,\n", - "shako, shatar, shatranj, shogi, shoshogi, shouse, sittuyin, suicide,\n", - "supply, tencubed, threekings, torishogi, xiangqi, yarishogi | ||
UCI_AnalyseMode | False | |||
UCI_LimitStrength | False | |||
UCI_Elo | 1350 | 500 | 2850 | |
UCI_ShowWDL | False | |||
SyzygyPath | <empty> | |||
SyzygyProbeDepth | 1 | 1 | 100 | |
Syzygy50MoveRule | True | |||
SyzygyProbeLimit | 7 | 0 | 7 | |
Use NNUE | True | |||
EvalFile | <empty> | |||
TsumeMode | False | |||
VariantPath | <empty> |
Name | Default | Minimum | Maximum | Variants |
---|---|---|---|---|
Protocol | uci | uci, usi, ucci, ucicyclone, xboard | ||
Debug Log File | ||||
Threads | 1 | 1 | 512 | |
Hash | 16 | 1 | 33554432 | |
Clear Hash | ||||
Ponder | False | |||
MultiPV | 1 | 1 | 500 | |
Skill Level | 20 | -20 | 20 | |
Move Overhead | 10 | 0 | 5000 | |
Slow Mover | 100 | 10 | 1000 | |
nodestime | 0 | 0 | 10000 | |
UCI_Chess960 | False | |||
UCI_Variant | chess | 3check, 5check, ai-wok, almost, amazon, antichess, armageddon, asean,\\nataxx, atomic, breakthrough, bughouse, cambodian, capablanca,\\ncapahouse, caparandom, centaur, chancellor, chaturanga, chess,\\nchessgi, chigorin, clobber, clobber10, codrus, coregal, courier,\\ncrazyhouse, dobutsu, embassy, euroshogi, extinction, fairy,\\nfischerandom, gardner, giveaway, gorogoro, gothic, grand, grasshopper,\\nhoppelpoppel, horde, janggi, janggicasual, janggimodern,\\njanggitraditional, janus, jesonmor, judkins, karouk, kinglet,\\nkingofthehill, knightmate, koedem, kyotoshogi, loop, losalamos,\\nlosers, makpong, makruk, manchu, micro, mini, minishogi, minixiangqi,\\nmodern, newzealand, nightrider, nocastle, nocheckatomic, normal,\\nokisakishogi, opulent, placement, pocketknight, racingkings, seirawan,\\nshako, shatar, shatranj, shogi, shoshogi, shouse, sittuyin, suicide,\\nsupply, tencubed, threekings, torishogi, xiangqi, yarishogi | ||
UCI_AnalyseMode | False | |||
UCI_LimitStrength | False | |||
UCI_Elo | 1350 | 500 | 2850 | |
UCI_ShowWDL | False | |||
SyzygyPath | <empty> | |||
SyzygyProbeDepth | 1 | 1 | 100 | |
Syzygy50MoveRule | True | |||
SyzygyProbeLimit | 7 | 0 | 7 | |
Use NNUE | True | |||
EvalFile | <empty> | |||
TsumeMode | False | |||
VariantPath | <empty> |
Name | Default | Minimum | Maximum | Variants |
---|---|---|---|---|
Debug Log File | ||||
Threads | 1 | 1 | 1024 | |
Hash | 16 | 1 | 33554432 | |
Clear Hash | ||||
Ponder | False | |||
MultiPV | 1 | 1 | 500 | |
Skill Level | 20 | 0 | 20 | |
Move Overhead | 10 | 0 | 5000 | |
Slow Mover | 100 | 10 | 1000 | |
nodestime | 0 | 0 | 10000 | |
UCI_Chess960 | False | |||
UCI_AnalyseMode | False | |||
UCI_LimitStrength | False | |||
UCI_Elo | 1320 | 1320 | 3190 | |
UCI_ShowWDL | False | |||
SyzygyPath | <empty> | |||
SyzygyProbeDepth | 1 | 1 | 100 | |
Syzygy50MoveRule | True | |||
SyzygyProbeLimit | 7 | 0 | 7 | |
Use NNUE | True | |||
EvalFile | nn-5af11540bbfe.nnue |
Name | Default | Minimum | Maximum | Variants |
---|---|---|---|---|
Debug Log File | ||||
Threads | 1 | 1 | 1024 | |
Hash | 16 | 1 | 33554432 | |
Clear Hash | ||||
Ponder | False | |||
MultiPV | 1 | 1 | 500 | |
Skill Level | 20 | 0 | 20 | |
Move Overhead | 10 | 0 | 5000 | |
Slow Mover | 100 | 10 | 1000 | |
nodestime | 0 | 0 | 10000 | |
UCI_Chess960 | False | |||
UCI_AnalyseMode | False | |||
UCI_LimitStrength | False | |||
UCI_Elo | 1320 | 1320 | 3190 | |
UCI_ShowWDL | False | |||
SyzygyPath | <empty> | |||
SyzygyProbeDepth | 1 | 1 | 100 | |
Syzygy50MoveRule | True | |||
SyzygyProbeLimit | 7 | 0 | 7 | |
Use NNUE | True | |||
EvalFile | nn-5af11540bbfe.nnue |