Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pad audio instead of mel features to reduce word error rates #1084

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

MahmoudAshraf97
Copy link
Collaborator

This PR pads audio before feature extraction instead of padding the features, this is inline with how the whisper model was trained because reverting the zero-padded Mel spectrogram features back to time domain results in a white noise with moderate amplitude that causes hallucinations and wrong transcriptions
The distil-large-v3 WER dropped from 26.04 at 83a368e to 14.472
This figure can be reproduced by running benchmarks/yt_commons.py and switching the batched inference to sequential

as a side effect, it seems that the transcription is also faster
it took 27 minutes to evaluate the dataset vs 35 minutes before the fix

References:
openai/whisper#730 (comment)
openai/whisper#838 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant