Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix type issue introduced by #28 #39

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix type issue introduced by #28 #39

wants to merge 1 commit into from

Commits on Oct 7, 2024

  1. Fix type issue introduced by xjdr-alt#28 

    Commit xjdr-alt#28 changed `apply_rotary_embed` to have dtype parameter with default float32, and forces attention softmax to be done float32. Since `attention` doesn't specify the dtype parameter when calling `apply_rotary_embed`, and output matmul doesn't convert back from float32 to match the values type, this is an issue if you're running BF16.

This specifies the existing xq.dtype for the dtype parameter when calling `apply_rotary_embed` (alternatively, we could cast keys to float32 in `scores = torch.matmul(xq, keys)`), and converts the scores to match values at the output matmul.
    @HaileyStorm
    HaileyStorm authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    7c01aa6 View commit details
    Browse the repository at this point in the history