-
Notifications
You must be signed in to change notification settings - Fork 79
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update axolotl image and other dependencies (#28)
* Remove environment key from CI yaml * Update base image spec to axolotl 0.4.0 * Update deepspeed config location * Remove redundant configuration flags from merge cmdline * Disable debug mode in codellama config * Try re-enabling mistral flash attention * Revert some of the CI training overrides * Don't truncate data * Try a config without sample packing * Don't pad to sequence length * Reinstate CI data truncation * Set base GPU config to use A100-40GB * Remove sample packing and standardize batch / LR params for all models * Standardize sequence_len for mistral * Use consistent fractional val_set_size * Disable quantization in llama config * Fix CI val_set_size * Try simple torch optimizer * Try reverting deepspeed workaround * Fix type annotation * Add a step to assert that the evaluation loss is reasonable * Fix run name * Improve results table extraction * Fix direction of loss assertion * Don't call the remote data my_data * Remove huggingface secret (it's not needed for thse models) * Bump huggingface util pins * Update README
- Loading branch information
Showing
10 changed files
with
116 additions
and
89 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
from io import StringIO | ||
import re | ||
import sys | ||
|
||
import pandas as pd | ||
|
||
from modal import Volume | ||
|
||
|
||
if __name__ == "__main__": | ||
|
||
with open(".last_run_name", "r") as f: | ||
run_name = f.read().strip() | ||
|
||
vol = Volume.lookup("example-runs-vol") | ||
contents = b"" | ||
for chunk in vol.read_file(f"{run_name}/lora-out/README.md"): | ||
contents += chunk | ||
|
||
m = re.search(r"### Training results\n\n(.+?)#", contents.decode(), flags=re.DOTALL) | ||
if m is None: | ||
sys.exit("Could not parse training results from model card") | ||
else: | ||
results_text = m.group(1).strip().replace(" ", "") | ||
|
||
results = pd.read_table(StringIO(results_text), sep="|") | ||
train_loss = float(results["TrainingLoss"].iloc[-1]) | ||
val_loss = float(results["ValidationLoss"].iloc[-1]) | ||
|
||
print(f"Loss: {train_loss:.2f} (training), {val_loss:.2f} (validation)") | ||
sys.exit(val_loss > 0.25) # Arbitrary threshold |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.