Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor llama3 demo to the new generator API #16753

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Refactor llama3 demo to the new generator API #16753

wants to merge 19 commits into from

Conversation

mtairum
Copy link
Contributor

@mtairum mtairum commented Jan 15, 2025 •
edited
Loading

Ticket

#16752

What's changed

New Llama3 text demo that uses the Llama3 generator API.

TODO

  • Update the CI tests to use the new demo
  • Remove the old llama3 demo
  • Update test_llama_accuracy.py to use the generator API
  • Measure prefill perf

Checklist

TODO

@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 4f4c250 to decc3fa Compare January 17, 2025 17:27
@mtairum mtairum marked this pull request as ready for review January 17, 2025 17:37
@mtairum mtairum self-assigned this Jan 17, 2025
@mtairum mtairum added the llama3 label Jan 17, 2025
@mtairum mtairum requested a review from skhorasganiTT January 17, 2025 17:37
skhorasganiTT
skhorasganiTT reviewed Jan 17, 2025
View reviewed changes
Copy link
Contributor

@skhorasganiTT skhorasganiTT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all the files except simple_text_demo.py for now since that one is still being cleaned up

models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/llama_model.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
models/demos/llama3/tt/generator.py Outdated Show resolved Hide resolved
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 943dff3 to 5dfa882 Compare January 20, 2025 12:45
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from 3847b2b to 4161293 Compare January 21, 2025 14:30
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from a601774 to 4236e3a Compare January 22, 2025 11:25
mtairum added 16 commits January 24, 2025 14:53
@mtairum
#0: Refactord llama3 demo to use the new generator API
1a39143
@mtairum
#0: Fixes and cleanup to demo
3bfe69e
@mtairum
#0: New working llama3 simple text demo
dd03ab0
@mtairum
#0: Add argmax on device to llama3 generator
915b2ac
@mtairum
#0: Minor fix
ba1ed7a
@mtairum
#0: Update lt to use the new demo
08897bf
@mtairum
#0: Update benchmark api
252757f
@mtairum
#0: Moved input preprocess to llama common
054bd41
@mtairum
#0: Address review
a56ccbf
@mtairum
#0: Minor fixes
577a15c
@mtairum
#0: Address further PR comments. fix long-context demo. Removed depre…
4b1a534 
…cated op-perf script
@mtairum
#0: Moved llama3 input prompts to a separate folder. Added missing lo…
0b6dba8 
…ng prompts and tested all of them
@mtairum
#0: Fix tensor padding and moved to inside llama model
200dbba
@mtairum
#0: Add TTFT info to lt
ccc4529
@mtairum
#0: Removed unused argmax_on_device
7af323f
@mtairum
#0: Moved the dram mem config to the end
d6571e3
@mtairum mtairum force-pushed the mtairum/llama3_text_demo branch from e89d7e5 to 82e3d03 Compare January 24, 2025 15:32
@mtairum mtairum requested a review from a team as a code owner January 24, 2025 16:30
@tt-rkim
Copy link
Collaborator

tt-rkim commented Jan 25, 2025

Did you run nightly ttnn single card, t3k demos, tg demos?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skhorasganiTT skhorasganiTT skhorasganiTT approved these changes

@tt-rkim tt-rkim tt-rkim approved these changes

@cglagovichTT cglagovichTT Awaiting requested review from cglagovichTT cglagovichTT is a code owner

@yieldthought yieldthought Awaiting requested review from yieldthought yieldthought is a code owner

@uaydonat uaydonat Awaiting requested review from uaydonat uaydonat is a code owner

At least 0 approving reviews are required to merge this pull request.

Assignees

@mtairum mtairum

Labels
llama3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mtairum @tt-rkim @skhorasganiTT