Japanese Translations

[classmate01]

Hello guys!
First of all, I would like to thank you so much for this project and the people behind it, also the active community helping each other.
Its been days since I started learning to use standalone whisper after I chose to run it locally in my own and I know I still have a lot to learn.

I've been translating subtitles from Japanese to English animations, and sometimes the dialog is out of sync, the subtitles finishes up but the speaker was still talking, and I am missing some details especially when there is a long conversation.

For example.
Full dialog:
I'll go to the store after school, uhmm.. (like a small millisec pause).. and buy a laundry detergent, Is there anything you want to buy?

Translated dialog from my code:
I'll go to the store after school, is there anything you want to buy?

Missing:
and buy laundry detergent

Full command:
--language Japanese --model "large-v2" --task translate --output_dir source --compute_type=int8_float16 --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cuda --vad_threshold 0.35 --vad_min_speech_duration_ms=800 --beep_off

and can you guys share me a good command for JP to EN subtitle translation?

Thank you so much!

[Purfview]

I've been translating subtitles from Japanese to English animations, and sometimes the dialog is out of sync

Less accurate timestamps is expected behaviour of the translate task. Best is just transcribe and then translate with other AI, for example ChatGPT 4o.

...and I am missing some details especially when there is a long conversation.

This can happen too with whisper models, it can miss something without any reason.

...and can you guys share me a good command for JP to EN subtitle translation?

Try --sentence

--compute_type=int8_float16

I think float16 should be faster on CUDA.

--vad_threshold

It doesn't do anything for pyannote_v3 vad.

--vad_min_speech_duration_ms=800

Looks too high, better don't touch it.

[mjamil85]

Avoid using translate & Kim Vocal 2 as this cause filter to much low voice cause sub missing. I use UVR5 & then use faster-whisper.

Just use the transcribe first in japanese & then upload the srt here translate to English.

[Purfview]

this cause filter to much low voice cause sub missing

Can you share any audio example of this happening?

[mjamil85]

I tested Kim Vocal 2 at UVR5 & output the final audio result. Then I listened the audio using headphones & then I found the root cause why faster-whisper failed to transcribe the low voice because Kim Vocal 2 removed it completely or damaged the low voiced audio.

So the root cause is Kim Vocal 2, not faster-whisper. I tested MDX32-InstVoc HQ or MDX-Net-Voc-FT give good results transcribe clean vocal than Kim Vocal 2.

[Purfview]

MDX-InstVoc HQ is not good imo.
MDX-Net-Voc-FT is practically almost same as Kim Vocal 2, I don't see much difference.