Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add amd support #114

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ParthJadhav
Copy link

No description provided.

@AnttiRae
Copy link

AnttiRae commented Oct 28, 2023

Hey! I'm not sure if I did this correctly but here's my output from running ./run.sh --model 7b --with-rocm. It seems that something went wrong with docker not detecting my GPU. Let me know if there's something more specific I should test.
I'm running Fedora 38 with AMD RX 7900 XTX as the GPU.

Edit: I tried with Windows wsl2 (ubuntu) as well and got the same error.

@cotsuka
Copy link

cotsuka commented Nov 17, 2023

Finally found time to test this PR. Looks like I'm running into DNS resolution issues hitting several of the repos. I ran the same command as @AnttiRae above. I'll attempt to retest later to see if that clears up.

image

@cweiske
Copy link

cweiske commented Jan 24, 2024

Works here.

Speed depends on the GPU; here my GPU is slower than the CPU.
CPU: AMD Ryzen 7 7700, 16 cores with 64GiB RAM
GPU: AMD Radeon RX 7600, 8GB

Sample request timings with 70b:

llama-gpt-api-rocm-ggml-1  | llama_print_timings:       total time = 225636.50 ms
llama-gpt-api-1            | llama_print_timings:       total time = 160374.53 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants