Forget subtitles: YouTube now dubs videos with AI-generated voices(restofworld.org)
restofworld.org
Forget subtitles: YouTube now dubs videos with AI-generated voices
https://restofworld.org/2023/youtube-ai-dubbing-automated-translation/
17 comments
Anyone outside of English speaking countries should get ready to be detaulted to crappy AI voices instead of the original English without any option to disable it. Happened with subtitles as well. I don't think there's any way to disable subtitles once and for all on all devices. They come back even if you turn them off everywhere. I constantly have to disable them manually. If that crap starts with sound as well where I cannnot set a default, I'll just use a third party way to watch the videos.
It's cool and all but an awkward framing of the feature. Dubs don't really have the same functios as subtitle text. It's apples and oranges.
As for the impact on people who want money to perform audio to audio translation services it sounds a lot like the luddites vs the power-looms. It's too bad for the people in that service industry but I'm 100% happy auto-translation is making everyone more able to communicate.
As for the impact on people who want money to perform audio to audio translation services it sounds a lot like the luddites vs the power-looms. It's too bad for the people in that service industry but I'm 100% happy auto-translation is making everyone more able to communicate.
I am wondering though: capitalism functions because of comparitive advantage. What happens when comparitive advantage vanishes in the realm of most people except the 0.1% in the tech companies.
It's almost as if AI/technology is reaching a phase where it can reap every advantage, causing a sort of heat death analogous to the heat death in thermodynamics, where trade will eventually cease because no one has any meaningful contribution since AI can do it better.
Maybe this will create a system where no one has to do any work and UBI will take care of everyone, but humans need a purpose and what happens to society when there is mass depression due to a widespread lack of purpose?
It's almost as if AI/technology is reaching a phase where it can reap every advantage, causing a sort of heat death analogous to the heat death in thermodynamics, where trade will eventually cease because no one has any meaningful contribution since AI can do it better.
Maybe this will create a system where no one has to do any work and UBI will take care of everyone, but humans need a purpose and what happens to society when there is mass depression due to a widespread lack of purpose?
Thank you for this comment, sums up my thoughts perfectly.
I fear a regression of society to the days of zero-sum games. As an example, even though folks may say "open source models" are the answer, the only way to run said models at scale is to acquire physical resources such as electricity and hardware, and physical resources, by their very nature, are subject to zero-sum games.
I fear a regression of society to the days of zero-sum games. As an example, even though folks may say "open source models" are the answer, the only way to run said models at scale is to acquire physical resources such as electricity and hardware, and physical resources, by their very nature, are subject to zero-sum games.
> AI/technology is reaching a phase where it can reap every advantage
No, the one who can reap every advantage of the new skills and knowledge learned by AI is actually you. They are all just one prompt away.
> no one has any meaningful contribution since AI can do it better
It won't be human vs AI, but human+AI vs human+AI.
Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Looks like AI is only going to scale as much as we can scale oversight. It will have to be a human+AI team effort.
No, the one who can reap every advantage of the new skills and knowledge learned by AI is actually you. They are all just one prompt away.
> no one has any meaningful contribution since AI can do it better
It won't be human vs AI, but human+AI vs human+AI.
Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Looks like AI is only going to scale as much as we can scale oversight. It will have to be a human+AI team effort.
> It won't be human vs AI, but human+AI vs human+AI.
I think the metaphor is "centaur"; in chess we started with humans dominating, then AI got better and (after DeepBlue) centaurs dominated, then AI got better and humans only held them back.
> Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios.
Well, that's one way to prevent misaligned AI paper-clipping us. (Pity some of us keep being lazy and letting the fragile AI break stuff…)
Given we don't know how hard it is to make a human-level general AI, there's no way to know right now if we're going to be in a world of (for any skill not already automated) human dominance, centaur dominance, or AI dominance, nor when any transitions between states will happen.
I think the metaphor is "centaur"; in chess we started with humans dominating, then AI got better and (after DeepBlue) centaurs dominated, then AI got better and humans only held them back.
> Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios.
Well, that's one way to prevent misaligned AI paper-clipping us. (Pity some of us keep being lazy and letting the fragile AI break stuff…)
Given we don't know how hard it is to make a human-level general AI, there's no way to know right now if we're going to be in a world of (for any skill not already automated) human dominance, centaur dominance, or AI dominance, nor when any transitions between states will happen.
Short-term thinking only. human+AI will give way to AI...of couse, I can reap the advantages of AI. THAT is the problem: thinking about what I can do. That is the problem that plagues us.
I was not thinking of I. I could easily reap the advantages of AI now, but what about future generations?
I was not thinking of I. I could easily reap the advantages of AI now, but what about future generations?
It's another case of "it is easier to imagine an end to the world than an end to capitalism". I for one can absolutely imagine having a "purpose" without a "job".
Here are a few examples of activities that might give people a sense of purpose: art, traveling and exploring the world, lifelong learning, educating kids/adults about something you love, political activism, socializing in general, scientific inquiry, connecting with and helping animals, catching up on old literature/films/video games. And this is just me brainstorming for 2 minutes.
Here are a few examples of activities that might give people a sense of purpose: art, traveling and exploring the world, lifelong learning, educating kids/adults about something you love, political activism, socializing in general, scientific inquiry, connecting with and helping animals, catching up on old literature/films/video games. And this is just me brainstorming for 2 minutes.
I am sure you can imagine such ways, and so can I. I personally hate working at a job and I don't even have a traditional full-time job anyway. I love doing a lot of things on your list as well and I feel like I have a purpose even without one.... but a lot of people are not so self-directed. And even if people are, a lot of them derive some satisfaction from taking their passion and sharing it with others.
How are you going to share your passion in creating art when most people will be rather indifferent to art due to AI? And will most people even go to universities given that they don't need to go to get a job?
Do you really think that if no one has any job whatsoever, they will all be self-directed enough to find true meaning in life? I could, and maybe you could to, but I really doubt if we are going to be transformed into a planet of traveling, science-teaching, animal caring, old video-game playing happy creatures any time soon.
How are you going to share your passion in creating art when most people will be rather indifferent to art due to AI? And will most people even go to universities given that they don't need to go to get a job?
Do you really think that if no one has any job whatsoever, they will all be self-directed enough to find true meaning in life? I could, and maybe you could to, but I really doubt if we are going to be transformed into a planet of traveling, science-teaching, animal caring, old video-game playing happy creatures any time soon.
> How are you going to share your passion in creating art when most people will be rather indifferent to art ~~due to AI~~?
This is already true, regardless of AI. Most hobbies or pursuits people act on have quite a small group (compared to the total population) of people they can share their interest with. It's not like everyone loves art (or any particular interest) right now or cares to talk about it for more than maybe an hour in a month.
People form relationships based on their interests (or at least what they spend a lot of time doing), deliberately or not. I play video games and write software. Most of my friends share at least partial interest with one of those two things. I almost never talk about either of these thing with family, since they don't have interest.
None of my friends are into Muay Thai. This would likely be different from someone who is very into Muay Thai.
This is already true, regardless of AI. Most hobbies or pursuits people act on have quite a small group (compared to the total population) of people they can share their interest with. It's not like everyone loves art (or any particular interest) right now or cares to talk about it for more than maybe an hour in a month.
People form relationships based on their interests (or at least what they spend a lot of time doing), deliberately or not. I play video games and write software. Most of my friends share at least partial interest with one of those two things. I almost never talk about either of these thing with family, since they don't have interest.
None of my friends are into Muay Thai. This would likely be different from someone who is very into Muay Thai.
I don't have a full answer, and acknowledge that this will be a difficult transition, but humanity had lived for millennia before the modern concept of "a job", and many tribal societies still do, so we know it's possible.
As for "happy", I don't think humans will ever be generally happy, but there's nothing new about that - our neural chemistry keeps us at a state of want as an adaptation; c'est la vie. As I see it, if you want something resembling lasting happiness, you need either drugs, or to relinquish "the self" via deep mindfulness, or to engineer ourselves beyond being human.
As for "happy", I don't think humans will ever be generally happy, but there's nothing new about that - our neural chemistry keeps us at a state of want as an adaptation; c'est la vie. As I see it, if you want something resembling lasting happiness, you need either drugs, or to relinquish "the self" via deep mindfulness, or to engineer ourselves beyond being human.
This is an old question in economics, but undoubtedly relevant now:
"Is this new technology change different or similar to other technology changes?"
In other technology changes, there are hundreds of jobs that disappear, but hundreds that appears...and, as far as I know, we haven't observed an increase in joblessness in the last, say 30-50 years. We went through the industrial revolution, the invention of personal computers and a big etc and it didn't happen.
Is this one different? I don't have an answer, but I don't think the answer is obvious.
"Is this new technology change different or similar to other technology changes?"
In other technology changes, there are hundreds of jobs that disappear, but hundreds that appears...and, as far as I know, we haven't observed an increase in joblessness in the last, say 30-50 years. We went through the industrial revolution, the invention of personal computers and a big etc and it didn't happen.
Is this one different? I don't have an answer, but I don't think the answer is obvious.
If auto dub is as bad as auto generated subtitles, I am not holding my breath...
What are you on about? Youtube's auto generated subtitles are pretty good. granted, 5-10% of them are miss, but they usually convey at least enough to grasp the context. I would be lost without them (I'd have to be listening to really loud sound and/or focus more intensively to what is said)
This has really not been my experience... until just now.
A few times before I've tried using auto generated subs and every time I turned them off within a minute because so much of it is nonsense. Last time was about a month ago.
It seemed to get a lot of simple sentences right but every few it completely misparsed. Additionally, it seemed like every uncommon name or technical term became something else.
In light of your comment I went to YouTube to evaluate it again and... it's a lot better than I remember! I watched a few minutes of a video analysing some music and everything, including music terms, was correct.
A few times before I've tried using auto generated subs and every time I turned them off within a minute because so much of it is nonsense. Last time was about a month ago.
It seemed to get a lot of simple sentences right but every few it completely misparsed. Additionally, it seemed like every uncommon name or technical term became something else.
In light of your comment I went to YouTube to evaluate it again and... it's a lot better than I remember! I watched a few minutes of a video analysing some music and everything, including music terms, was correct.
Seconded. The auto generated subtitles were basically useless until literally a matter of weeks ago, but there has clearly been some kind of massive advance.
The auto-generated subtitles still struggle with very strong regional accents, very poor sound quality or people talking over each other, but they're good enough for the vast majority of content. It doesn't seem to make dumb errors any more, just the kind of understandable errors you'd expect from a human stenographer.
The auto-generated subtitles still struggle with very strong regional accents, very poor sound quality or people talking over each other, but they're good enough for the vast majority of content. It doesn't seem to make dumb errors any more, just the kind of understandable errors you'd expect from a human stenographer.
Your estimate is way too optimistic. But also what it doesn't get is usually key subject content and what it does get is low hanging fruit guessable from context.
Sometimes I try subtitles on something I cannot understand (because English is my second language) and that has not helped ever, it's always gibberish.
I tried going subtitles only a few times and gave up because I have to rewind and turn on sound every few seconds.
Sometimes I try subtitles on something I cannot understand (because English is my second language) and that has not helped ever, it's always gibberish.
I tried going subtitles only a few times and gave up because I have to rewind and turn on sound every few seconds.
For me YouTube captions/transcription have always been really bad, but whenever YouTube switched to some Whisper equivalent (or Whisper itself, ha) it has improved tremendously and rarely gets things wrong, in my own experience.
Maybe for american English, but put an accent in there, or even just British English and it goes down, fast. I'm currently watching Taskmaster on YouTube and every time it's not crystal clear audio with no background noice, it's closer to 50% miss
For English I only notice it miss technical words, confusing them for more common words. Regardless of accent.
For Russian, which I don't understand, the result is maybe 50% intelligible (particularly when technical jargon is probably involved). But 50% is still infinitely better than what I can do with Russian myself (0%).
For Russian, which I don't understand, the result is maybe 50% intelligible (particularly when technical jargon is probably involved). But 50% is still infinitely better than what I can do with Russian myself (0%).
50% intelligible is not infinitely better than 0% intelligible. In fact it could be that you're completely misled into believing something that's not true at all. Whereas if you are aware you understand 0% you're not jumping to these wild conclusions as confidently.
For context, I'm watching videos of a guy making jet engines in his garage. The consequence of being mislead is about zero. I'm not concerned about the 50% I miss. There are no wild conclusions being jumped to. If anybody jumps to any sort of wild conclusions on the basis of a 50% intelligible video, that's squarely on them.
Unless the unintelligible 50% happen to align to appear intelligible... but wrongly:)
Idk, my English is not great but apparently I am better than YouTube subs because when I need help understanding an English word it turns out YT can't transcribe it either :/
Some of us who are English speaking still read subtitles on English speaking media because of lack of clarity of the voices or we are hard of hearing.
If the dubbing machine can understand, it could produce a more readable version in the original language, not only a different one.
If it's the same machine as the one that generates the captions, then it's not very reliable yet.
Have you tried recently? YouTube captions have improved substantially over the last year. In my experience it now performs at human level which might not be good enough for every use case but it does make the "transcript" button and full text search practical
subtitles can be anything you want if you have an editor with subtitle functionality. it could be the dialogue from the video; it could be "thought bubbles" for the characters, or just ascii strings of whatever code.
in a "good enough" sense end user subtitled video has potential use for steganographic methods.
in a "good enough" sense end user subtitled video has potential use for steganographic methods.
Some of us use subtitles because we don't want to listen to the audio in situations where that would be disrespectful, like in the gym.
In Spain there is a big dubbing industry with very good professionals that are able to transmit the acting emotions nearly as good as the original (but never the same). But listening to automated AI dubbing would never reach that level of acting, sounds awkward. Even listening to "latin" spanish dubbing seems out of phase to most of us.
Anyway, YouTube should put some efforts on reviewing the (at least) iOS application localization in spanish, because it's impossible to see at a glance the age of a video in a video listing.
In Spain there is a big dubbing industry with very good professionals that are able to transmit the acting emotions nearly as good as the original (but never the same). But listening to automated AI dubbing would never reach that level of acting, sounds awkward. Even listening to "latin" spanish dubbing seems out of phase to most of us.
Anyway, YouTube should put some efforts on reviewing the (at least) iOS application localization in spanish, because it's impossible to see at a glance the age of a video in a video listing.
Funny, for me the Latin Spanish acting sounds better and the Spain ones sound stiff. And for video games, the acting from Japanese voice actors usually feels better than the English ones.
I guess it depends more on what you are used to than the actual acting.
I guess it depends more on what you are used to than the actual acting.
Spanish dubs are consistently the worse you can ever hear. Latin dubs are amazing sometimes even better than the original.
AI dubbing would never reach that level of acting, sounds awkward. - you should consider being more conservative in this direction)) I've said the same thing abt generating of images 5 years ago but the last progress of midjourney/dalle proved me wrong. We should not underestimate what hights human progress can achieve
Op also clearly hasn't used Elevenlabs or similar tools. If you clone a professional narrator it already sounds incredibly good and effectively indistinguishable from a human. Giving acting directions to the model to steer the output (kind of like ControlNet does for Stable Diffusion) seems like a logical next step.
But in this case, they want to avoid the human input. So, I guess, it would rather work by reading and copying the intonation of the source voice.
Not Youtube but I saw EZDubs on Twitter recently and honestly the results are not that bad
Here is the Messi post match interview after his first game, mind you him and the interviewer not even speaking the same spanish dialect
https://twitter.com/MLSes/status/1682579623089930241
And this was the english dubbed version https://twitter.com/ezdubs_bot/status/1682612908604157952 There are some mistakes like free throw instead of free kick but apart from that it's not that bad
Here is the Messi post match interview after his first game, mind you him and the interviewer not even speaking the same spanish dialect
https://twitter.com/MLSes/status/1682579623089930241
And this was the english dubbed version https://twitter.com/ezdubs_bot/status/1682612908604157952 There are some mistakes like free throw instead of free kick but apart from that it's not that bad
Great, in a few years Youtube will be able to generate new videos with some sort of LLM for video equivalent and stop paying creators. And include as much sponsorships and ads as possible too.
It is great because the demand for authentic content won't go away so this will create new networks which focus on promoting and ensuring that.
I think I'd have a hard time coming up with a feature I wanted less than this even if I tried.
I wonder if this could be used for language-learning. One of the biggest problem with learning certain small languages is that there's basically no interesting content produced in them (depending on your interests of course, but I ran into this problem multiple times.) For example, everything technical is done in English these days so you won't find too much advanced stuff on certain topics in Swedish, so you get stuck with repeating basics which gets boring after a while. There's also only certain countries which produce the kind of drama series or movies you want to watch. Sure, every country has some, but not always a lot, because again they import.
As a native Continental Portuguese speaker, I find the Brazilian Portuguese translation jarring, and would never use it.
While this may not be available to everyone right now, I am using a service from Finland https://www.hei.io/ to help me auto generate subtitles and "AI" voiceover.
Upload youtube link/video, Generate eng subs. Fix english subs manually, auto generate subs into any of the popular languages (we have req. for Asian langs.). Manually fix the auto-generated asian subtitle from minor errors. Reupload it and get perfect true "AI" Voiceover. All AI voice over services will have errors in translation/meaning. If you want quick job, this does it. If you want accurate job, manually fix the subs created by such services.
Anyways, what used to take us 2 weeks, takes us a day or two.
Upload youtube link/video, Generate eng subs. Fix english subs manually, auto generate subs into any of the popular languages (we have req. for Asian langs.). Manually fix the auto-generated asian subtitle from minor errors. Reupload it and get perfect true "AI" Voiceover. All AI voice over services will have errors in translation/meaning. If you want quick job, this does it. If you want accurate job, manually fix the subs created by such services.
Anyways, what used to take us 2 weeks, takes us a day or two.
I want subs. I learned English through watching English TV with subtitles in my native language.
And the generation after me started watching dubbed kids tv. Large decline in their English language skills.
And the generation after me started watching dubbed kids tv. Large decline in their English language skills.
I grew up used to reading subtitles. Listening to the original voices and languages is super interesting to me. Besides some very specific cases (I can't look at the screen all the time) I don't have any use for dubbing.
If they allow this to work with English -> English, I bet it will be very useful for lecture videos where the speaker does the "uh uh uh um um umm" thing (since those utterances don't get added to the subtitles.) I'm looking forward to trying this.
Also, I think some people are fixating on "forget subtitles" in the headline. Youtube obviously aren't going to take subtitles away. The article doesn't suggest that people stop using subtitles. It's just a meaningless figure of speech.
Also, I think some people are fixating on "forget subtitles" in the headline. Youtube obviously aren't going to take subtitles away. The article doesn't suggest that people stop using subtitles. It's just a meaningless figure of speech.
Subtitles are useful when you have children playing in the living room. Sometimes they will scream and you won't miss anything because of that