Is possible that we can fix timing of subtitles using whisper models

[Nisekoi-1]

Is it possible that using Whisper AI model we only sync/fix subs timing to audio rather than generating them from scratch?

[niksedk]

That would work very poorly as Whisper is really bad at finding timestamps.

[Baenwort]

There is a project working on improving the timestamps: https://github.com/jianfch/stable-ts perhaps their work could replace the python whisper you use?

[niksedk]

SE uses similar stuff, but it only works when there is silence between speech (waiting for this to be fixed in Whisper).

[Baenwort]

There is another project that is trying to improve timings and seems to work according to initial reviews: openai/whisper#435 (comment) references https://github.com/m-bain/whisperX

[dgoryeo]

@Baenwort , just a note: WhisperX at the moment supports very limited number of languages: EN, DE, IT, FR. Hopefully the community finds a way to add more languages to that appoach.

[dgoryeo]

Just an update to my earlier message: WhisperX now supports additional non-European languages.

[chenlung]

Something was recent committed from WhisperX to Subtitle Edit (partial support? Not sure if it's available/enabled in builds?), so it would be interesting to read your thoughts, both on this issue and audio-to-text in general (barriers; what might be coming, etc.), @niksedk.