Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable flash attention #20448

Merged
merged 36 commits into from
Nov 7, 2024
Merged

Enable flash attention #20448

merged 36 commits into from
Nov 7, 2024

Commits on Nov 4, 2024

  1. Enable flash attention

    @divyashreepathihalli
    divyashreepathihalli committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    37e2302 View commit details
    Browse the repository at this point in the history

  2. code reformat

    @divyashreepathihalli
    divyashreepathihalli committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    057aa66 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. address review comments

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    1abc948 View commit details
    Browse the repository at this point in the history

  2. add docstring

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    5784361 View commit details
    Browse the repository at this point in the history

  3. update docstring

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    3a47c53 View commit details
    Browse the repository at this point in the history

  4. add numerical correctness test

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    760e4b2 View commit details
    Browse the repository at this point in the history

  5. code reformat

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    045f153 View commit details
    Browse the repository at this point in the history

  6. use causal mask from call method

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    71bf0ce View commit details
    Browse the repository at this point in the history

  7. address review comments

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    3d2875f View commit details
    Browse the repository at this point in the history

  8. update if

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    6f99a57 View commit details
    Browse the repository at this point in the history

  9. fix tests

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4065bf3 View commit details
    Browse the repository at this point in the history

  10. update tests

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    a7390a6 View commit details
    Browse the repository at this point in the history

  11. enable flash attention on TPU JAX

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    71cbbd5 View commit details
    Browse the repository at this point in the history

  12. update code

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    3af4c95 View commit details
    Browse the repository at this point in the history

  13. minor fix

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    b9e075a View commit details
    Browse the repository at this point in the history

  14. address review comments

    @divyashreepathihalli
    divyashreepathihalli committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    5b4d23d View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. fix tests

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    59b0672 View commit details
    Browse the repository at this point in the history

  2. run api_gen

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    f269d06 View commit details
    Browse the repository at this point in the history

  3. code reformat

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    02f3451 View commit details
    Browse the repository at this point in the history

  4. fix mask issue

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    f622918 View commit details
    Browse the repository at this point in the history

  5. disable causal mask in dpa because it is comuted in comput_attention_… 

    …mask
    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    3cf22e4 View commit details
    Browse the repository at this point in the history

  6. fix masks tests

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    5b7f81a View commit details
    Browse the repository at this point in the history

  7. code reformat

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    53e082a View commit details
    Browse the repository at this point in the history

  8. disable tests of env is not supported

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    0ee6752 View commit details
    Browse the repository at this point in the history

  9. fix code reformat error

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    e310f22 View commit details
    Browse the repository at this point in the history

  10. fix torch GPU tests

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    cb62bbc View commit details
    Browse the repository at this point in the history

  11. Merge branch 'master' into enable_flash_attention

    @divyashreepathihalli
    divyashreepathihalli authored Nov 6, 2024
    Configuration menu
    Copy the full SHA
    d0e5de5 View commit details
    Browse the repository at this point in the history

  12. fix torch gpu tests

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    c28e460 View commit details
    Browse the repository at this point in the history

  13. make everything contigious

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    a94a9e6 View commit details
    Browse the repository at this point in the history

  14. check if mask is not before callng contigious

    @divyashreepathihalli
    divyashreepathihalli committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    36c1131 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. disable pytorch GPU test

    @divyashreepathihalli
    divyashreepathihalli committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    58ae551 View commit details
    Browse the repository at this point in the history

  2. merge master

    @divyashreepathihalli
    divyashreepathihalli committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    18d6ebd View commit details
    Browse the repository at this point in the history

  3. code reformat

    @divyashreepathihalli
    divyashreepathihalli committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    a36d119 View commit details
    Browse the repository at this point in the history

  4. set bias to None

    @divyashreepathihalli
    divyashreepathihalli committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    ae34d7f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'keras-team:master' into enable_flash_attention

    @divyashreepathihalli
    divyashreepathihalli authored Nov 7, 2024
    Configuration menu
    Copy the full SHA
    98edd93 View commit details
    Browse the repository at this point in the history

  6. disable GPU test

    @divyashreepathihalli
    divyashreepathihalli committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    9a5200e View commit details
    Browse the repository at this point in the history