New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

some questions about scaling #297

Open

miganchuanbo opened this issue Jul 10, 2023 · 0 comments

miganchuanbo commented Jul 10, 2023

DALLE2-pytorch/dalle2_pytorch/dalle2_pytorch.py

Line 869 in 00e07b7

def forward(self, x, mask = None, attn_bias = None):

It seems we need to scale up Q and K when using cosine sim. But what is the reason for scaling Q before applying rotary emb?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment