Skip to content

Latest commit

 

History

History
54 lines (41 loc) · 3.38 KB

README.md

File metadata and controls

54 lines (41 loc) · 3.38 KB

Repository with the results from OpenTau

Link to the main repository: https://github.com/GammaTauAI/opentau

Link to the dataset used for evaluation: https://huggingface.co/datasets/nuprl/ts-eval

744-dataset Results table

Configuration Percent Typecheck Avg Heuristic Avg type errors
Santacoder TS FT - Baseline - 2048 245/744 (32.9%) 200.7 4.7
Santacoder FIT FT - Baseline - 2048 297/744 (39.9%) 200.9 5.2
Santacoder FIT FT - Baseline - 1024 248/744 (33.3%) 200.7 5.1
Santacoder FIT FT - Baseline - 512 178/744 (23.9%) 201.2 6.3
Santacoder FIT FT - Tree, no usages - 2048 274/744 (36.8%) 168.4 3.7
Santacoder FIT FT - Tree, usages - 2048 353/744 (47.4%) 154.6 3.3

Configuration to file mapping

Configuration File name in the repository
Santacoder TS FT - Baseline - 2048 santacoder_baseline_finetuned_ts_ts-eval.json
Santacoder FIT FT - Baseline - 2048 santacoder_baseline_finetuned_fim_2048_ts-eval.json
Santacoder FIT FT - Baseline - 1024 santacoder_baseline_finetuned_fim_1024_ts-eval.json
Santacoder FIT FT - Baseline - 512 santacoder_baseline_finetuned_fim_512_ts-eval.json
Santacoder FIT FT - Tree, no usages santacoder_tree_nousages_ts-eval.json
Santacoder FIT FT - Tree, usages santacoder_tree_ts-eval.json

Notes about the results

  • Metrics in bold are the best results for each column.
  • All rows for the 744 runs had 0 average syntax errors, thus we did not include the column in the table.
  • FIT FT means that the model was fine-tuned for Fill-In-the-Type, for more information about the technique, please refer to the repository: https://github.com/GammaTauAI/santacoder-finetuning
  • TS FT means that the model was fine-tuned for TypeScript code generation, NOT for Fill-In-the-Type (FIT) or Fill-In-the-Middle (FIM).
  • The numbers at the end (e.g. 2048, 1024, 512) are the maximum number of characters for the context window when running inference with the model.

50-dataset Results table (Parser Disabled)

Both rows used the Baseline and the maximum context window size was 2048 characters.

Configuration Percent Typecheck Avg Heuristic Avg type errors Avg syntax errors
Santacoder TS FT 1/50 (2.0%) 0 121.2 42.1
Santacoder FIT FT 25/50 (50.0%) 230.0 4.6 0.2

Configuration to file mapping

Configuration File name in the repository
Santacoder TS FT santacoder_baseline_noparser_normal_ts-eval-subset.json
Santacoder FIT FT santacoder_baseline_noparser_fim_finetuned_ts-eval-subset.json