Skip to content

Commit

Permalink
Override cast_to_fp8 in te.module.linear (#120)
Browse files Browse the repository at this point in the history
**Description**
Fix a bug in TE integration. 
Currently we only override te.cpp_extensions.cast_to_fp8 with our own
cat_to_fp8 in msamp.te.extension.

We also need to override te.module.linear.cast_to_fp8. Otherwise, it
will use the original function which does not support ScalingTensor and
will raise an exception in Megatron-LM

**Major Revision**
- Override cast_to_fp8 in te.module.linear
  • Loading branch information
tocean authored Nov 3, 2023
1 parent 4784243 commit 3b0567a
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions msamp/te/extension.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ def override():
"""Override transformer engine extension functions."""
tex.fused_cast_transpose = TeExtensionOverrider.fused_cast_transpose
te.cpp_extensions.cast_to_fp8 = TeExtensionOverrider.cast_to_fp8
te.module.linear.cast_to_fp8 = TeExtensionOverrider.cast_to_fp8
te.cpp_extensions.fp8_cast_transpose_fused = TeExtensionOverrider.fp8_cast_transpose_fused


Expand Down

0 comments on commit 3b0567a

Please sign in to comment.