Generative AI - Short & Sweet 09

First of all, apologies for skipping last week’s episode. I went on vacation and couldn’t write a post with the quality that I expect from myself. This week I am back in the right headspace and went wild on the keyboard. Happy reading.

🔮 The GAI topic of the week is ..

OpenAI’s Whisper. It is an Automatic Speech Recognition (ASR) system that aims to recognize what has been said and transcribes it to text. 

Tech terms to know 📙
LLM = Large Language Model
GPT = Generative Pre-trained Transformer [1]
PaLM = Pathways Language Model [2]
Token = smaller text units like words, characters, or subwords.

Whisper’s power 🔥
Its high accuracy and robustness. Compared to Siri, Alexa, and Google Assistant, Whisper understands fast-spoken, mumbling, or jargon-filled voice recordings very accurately. Andrej Karpathy, former director of AI at Tesla and OpenAI, now an independent researcher (that’s cool), has tested it and the results are striking. See his results [3].

In the following figure, one can see that Whisper’s performance (left) is close to the performance of a professional human transcriber. It is best in class of LLMs (A-D in figure), but can’t beat computer-assisted (E in figure) transcription [6].

It is easy to use. For example, go on the Hugging Face space that hosts Whisper [4], enable your mic, speak in whatever language you want to, and click transcribe.
I tested it in the following languages: German (my mother tongue), Polish (I am raised bilingual), English (my working language), and Italian (I am learning it right now). It worked like a charm. Whisper could even transcribe the Polish dialect, Silesian, correctly.

Also, it is multilingual and multitasking, meaning an AI that performs multiple tasks at the same time. With regards to this, Whisper for example detects the language it is being presented with AND transcribes it to text.
There is a limit to this though. Whenever, I switched the language within a sentence, e.g. starting the sentence in German and ending in Polish, it would result in an error message.

OpenAI open-sourced Whisper. They open-sourced code [5] and the respective paper [6]. It helps tremendously when companies disclose the code of their models. Implementing and tweaking the models is easier, utilizing as well as customizing their capabilities best.
However, OpenAI didn’t disclose the dataset that Whisper was trained on. 🤔

Rumors about OpenAI’s data strategy 🤐
Why wouldn’t they? Some say it is because of legal issues (web scraping can be tricky at times) and some say it’s because they want to monetize it later. There might be another reason though: the performance of upcoming LLMs with next-in-line GPT-4. Let me unpack.

The 3 key aspects of improving an LLM. Generally speaking, there are 3 levers to get the most out of an LLM like GPT-3 [1] and PaLM [2], etc. For all 3 levers, the rule of thumb is the more, the better the performance.

• Computing power.

  • Regarding OpenAI: they partnered up with Microsoft. So, there are virtually no computing limits.

• Number of trainable parameters aka the size of a model.

  • Looking back at the LLMs developed, one can clearly recognize an upside trend in this field. It’s not a technical problem to further increase the model size, but it’s currently not helpful. I’m going to explain the Chinchilla scaling laws in just a second.

• Data.

  • For all LLMs, this is currently the limiting factor.

An Important insight from DeepMind. The London-based Google company DeepMind has investigated what the compute-optimal number of tokens for training a certain LLM size is. In simpler tokens 🫠, I have a model with X trainable parameters, how much training data do I need in order to have an optimally trained language model? These are the so-called Chinchilla scaling laws [7].
In the table below it’s quite interesting to see that, for example, OpenAI’s GPT-3 model with 175B parameters would be compute-optimally trained if the training set would have 3.7T tokens. In fact, it is just trained on 300B tokens [8]. Is it now undertrained? Most likely, yes, but keep in mind that DeepMind published their work [7] after GPT-3’s release. Now, the world looks different.

OpenAI’s GPTs. OpenAI is known for its strong language models. When released, they outperform current state-of-the-art LLMs significantly. I.e. OpenAI released GPT in 2018, GPT-2 in 2019, and GPT-3 in 2020. GPT-4 has been announced, but is quite delayed, presumably due to the attention that DALL-E and CLIP have drawn in the last 18 months or so. Earlier I wrote AI image generation [9] to provide a big-picture perspective .
Assuming GPT-4 has at least the size of GPT-3 and taking the Chinchilla scaling laws into account, they would need 3.7T tokens for training. 12 times as much training data as OpenAI had available for GPT-3, and probably more.

Training data is needed. OpenAI is missing training data. Moreover, I think it is a fair assumption that all other companies building next-gen LLMs are missing training data.
How to bridge the data gap? According to some resources [8], Wikipedia is around 3B tokens, accessible books around 67B tokens, and web crawls around 430B tokens, resulting in a little over 500B tokens. Still, 3.2T token missing.
Then comes synthetic data into play which should not provide the rest of the missing data, because of 2 reasons: first, current data augmentation models are good, but don’t provide enough data diversity to make up for 86% of the training data. Second, there is more real-world data out there. I am talking about video data.
Why not transcribe all available video data? This is deffo one of the main reasons why Whisper exists. There are well-reasoned estimates that Youtube alone has roughly 12T tokens [10]. As good fortune would have it, there is already a Hugging Face space that implemented a Youtube-Whisper model [11]. Plus, it’s fun to try it out.
I reckon it’s better to stop here, as I will follow up on this once GPT-4 is out.

What could Whisper’s impact be? 💡
Besides the point I made above about how Whisper is a strategic move in order to get bigger and better datasets, there is a plethora of other applications to think about.

• Imagine you could take study notes with your voice without having to check them constantly.

• Accurate subtitles in videos/ movies for all languages.

• Whisper could signify a life quality improvement for people with speech or hearing impairment. For instance, you could instantly transcribe your words for a deaf person.

• Further, to language barriers, do you know/ remember the movie The Hitchhiker’s Guide to the Galaxy? There was the Babelfish [12], a fish that once you stick it in your ear translates every galactic language in realtime. Imagine a headphone-like device that tears down all language barriers. And, it is even called Whisper!

• Lastly, just think about the innumerable applications in law, healthcare, banking, marketing, education. There are plenty of ways how companies could integrate Whisper into existing services.

What other applications seem obvious to you?

Generative AI Top 3 Gems 💎

1. NVidia Maxine for real-time audio and video communication. Particularly interesting is the feature for turning on/ off the eye contact of the other person, improving personal touch in remote video calls.

2. Editorial feature from Google Arts & Culture: 12 great AI-created songs. The whole web page is a piece of art, I think.

3. This Photoshop Plugin integrates stable diffusion. You can generate high-quality images in the tool like you do with DALL-E, Parti AI, and other models, see [13].

Subscribing to, giving feedback about, and sharing the newsletter as well as our renowned online course will be highly appreciated and helps a lot. 🌝

You know about 💎 GAI Gems that we should consider? Or other matters?
Please, respond to this email.

Thank you for reading,

Martin

References:
[1] OpenAI’s language model GPT-3.
[2] Google’s Pathways Language Model.
[3] Andrej plays with Whisper.
[4] Hugging Face Demo.
[5] Whisper’s code.
[6] OpenAI’s paper about Whisper.
[7] DeepMind’s paper: Training Compute-Optimal Large Language Models.
[8] GPT-3: all you need to know.
[9] Episode 2 of this newsletter about AI image generation.
[10] Estimates how many tokens there are in Youtube.
[11] Hugging Face space for Youtube-Whisperer.
[12] Babelfish in The Hitchhiker’s Guide to the Galaxy - video.
[13] Ipython Notebook to try it out yourself.