Amazing! I've made a similar ebooks-audiobooks aligner years ago: https://github.com/r4victor/syncabook. At that time, I chose to synthesize the text and align two audio sequences because I found texts-alignment approaches (including ML-based ones) too compute-intensive and inadequate for long texts. I see Storyteller works by aligning the texts. Could you give some view on how long it takes to sync a book?
Also, my experience was that audio and text versions are often very different (e.g. the audio having an intro missing from the text). It'd be very interesting to know how well Storyteller handles such cases. Does it require manual audio/text editing or handle the differences automatically?
The docs say it's usually 1-4 hours depending on the book and the hardware: https://smoores.gitlab.io/storyteller/docs/syncing-books
The docs also have a detailed section about the algorithm that goes into how it auto-handles differences between the audio and the text.
One obvious optimization is to sample the audio file at regular intervals and transcribe only a part of the text. Then just interpolate the locations. This can speed it up by a couple of orders of magnitude.
This is true, but it really limits the ability to highlight the current sentence visually while it's being read, which is great for language learning and for reducing cognitive load. I actually spent a lot of time trying to get the timing as precise as I could to make this feel as natural as possible, and I think the effect is really nice!
Ah, makes sense. Maybe have it as an option?
And I haven't realized that you can actually see sentences highlighted as they are being read. I'd love that for Chinese (I'm learning it, so it'd help me a lot). I'll try and see if it "just works", and contribute a patch if it doesn't.
A few years ago, I made that as a YouTube channel based on LibriVox audiobooks, maybe you'll enjoy it:
Simplified Mandarin: https://youtube.com/playlist?list=PLVlVz7EDz7fprPeVpqQCvlkMI...
Traditional Mandarin: https://youtube.com/playlist?list=PLVlVz7EDz7fpQZr29P5hqVveL...
... also 33 other languages https://youtube.com/@literature_for_eyes_and_ears/playlists
I left it to languish once I discovered the demand wasn't that great and I was spending more time making the videos than people ultimately spent watching them.
There's an open ticket for languages other than English! https://gitlab.com/smoores/storyteller/-/issues/10. If you want to take a look, please do! I don't have any contributing guidelines yet, unfortunately, but they'll probably come soon. I think Whisper does have Cantonese and Mandarin support, so it should be possible to add support for those languages, though we'll have to look into nltk support for sentence tokenization as well!
Hello! syncabook is awesome, and indeed Storyteller does take "the opposite" approach when it comes to forced alignment.
Others have linked to the docs, where I go into detail about the syncing algorithm, but at a high level:
Storyteller uses Whisper to transcribe the audio to text (this is the most computationally expensive part of the process)
Then we use a Levenshtein-distance-based fuzzy search algorithm to find each chapter in the text (this is attempting to account for the difference between audio and text versions, as you said!)
Then for each chapter, we find the start and end timestamp of each sentence, again using a fuzzy search across the transcription.
In general, Storyteller does a pretty good job; it treats the ebook as the source of truth, which means that at the moment it sometimes misses introductory and ending pieces of the audiobook, though it's on the roadmap to have some support for explicitly triggering those when that happens.