Things started failing after new commit into main

[gane5hvarma]

System Info

amazon linux 2
Running it in l40s

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

model=meta-llama/Meta-Llama-3.1-8B-Instruct
volume=$PWD/data
docker run -d --gpus all --env-file .env --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/predibase/lorax:main --model-id $model
See this error after model gets downloaded

'FlashLlamaAttention' object has no attribute 'fp8_kv'

Expected behavior

webserver should spin. It worked with older version. There seems to be problem after the new commit went into main

[arnavgarg1]

cc: @ajtejankar

[ajtejankar]

Should be fixed with #662