The video discusses the highlights from the 2024 State of AI report, touching on advancements and predictions in Artificial Intelligence. Key updates include the competition among AI models like Claude 3.5 and GPT-4, and significant financial data regarding AI training expenses projected for 2024. The video also references past predictions and developments that have occurred over the past year, including the training costs for future AI models, and mentions notable achievements in multimodality through Meta's MovieGen, further emphasizing AI's increasing integration into various scientific fields.

The video addresses the concerns raised by prominent figures in AI research about the safety and ethical considerations of AI. It explores the possibility of AI surpassing human intelligence in the near future and the potential consequences of such a development. Notable researchers have issued warnings comparing AI threats to nuclear war, raising ethical questions and emphasizing the necessity of safety measures. Highlights from the video also include technological advancements such as BrainLM's ability to predict clinical variables from brain activity and developments from Google DeepMind.

Main takeaways from the video include:

Please remember to turn on the CC button to view the subtitles.

Key Vocabularies and Common Phrases:

1. gratuitous [ɡrəˈtuːɪtəs] - (adjective) - Done without good reason; uncalled for. - Synonyms: (unwarranted, unnecessary, superfluous)

So now that I am done with the gratuitous special effects courtesy of a new Pica AI tool.

2. converging [kənˈvɜːrdʒɪŋ] - (verb) - Coming together from different directions so as eventually to meet. - Synonyms: (merging, uniting, meeting)

All models are converging, they thought, because of the heavy overlaps in pre training data.

3. amortization [ˌæmərtəˈzeɪʃən] - (noun) - The process of gradually writing off the initial cost of an asset. - Synonyms: (depreciation, reduction, decrement)

Note that that does not include research compute amortization as in training miniature models.

4. multimodality [ˌmʌltiˌməʊˈdæləti] - (noun) - The use or availability of multiple modes or methods. - Synonyms: (multiplicity, plurality, diversification)

The next highlight for me was the section on multimodality and Meta's movie gen.

5. perturbation [ˌpɜːrtərˈbeɪʃən] - (noun) - A disturbance or deviation of a system, moving it from its regular or normal state. - Synonyms: (disturbance, disruption, deviation)

Although those other two are pretty impactful, it can do in silico perturbation analysis

6. cognition [kɑːɡˈnɪʃən] - (noun) - The mental action or process of acquiring knowledge and understanding through thought, experience, and the senses. - Synonyms: (perception, discernment, insight)

You need to have a deep interest in machine cognition and consciousness.

7. biologically [ˌbaɪəˈlɒdʒɪkli] - (adverb) - In a way that relates to biology or living organisms. - Synonyms: (biogenically, organically, naturally)

BrainLm has the ability to simulate brain responses in a biologically meaningful manner.

8. disaggregate [dɪsˈæɡrɪˌɡeɪt] - (verb) - To separate something into its component parts. - Synonyms: (disassemble, divide, break down)

The research talent that is probably most able to do the kind of engineering necessary to ascertain the most useful, unique sources of data for the models is the very same talent least incentivized to do that research.

9. extrapolate [ɪkˈstræpəˌleɪt] - (verb) - To estimate or conclude by extending known information. - Synonyms: (infer, deduce, predict)

We wouldn't want to extrapolate too far.

10. externalities [ˌɛkstɜːrˈnælɪtiz] - (noun) - Consequences of an industrial or commercial activity which affects other parties without this being reflected in market prices. - Synonyms: (spillover, side effects, consequences)

Speaking of kind of externalities like copyright payment, how about environmental effects? Well, all of these companies, as I have covered before, made various pledges saying that they're going to be completely carbon neutral

AI - 2024AD - 212-page Report (from this morning) Fully Read w- Highlights

This morning the 212 page state of AI report 2024 was released and as I did last year, I want to bring you just the highlights. Yes, I read it all and it's been going six years and it gives a powerful framework around which I can also weave in some of the most interesting developments of the past few days. So now that I am done with the gratuitous special effects courtesy of a new Pica AI tool that I upgraded my account to bring you guys, here is the report. The report has been going going six years and it's from Airstreet Capital. These are of course my highlights. You will find plenty more if you read the full report yourself linked in the description. But this first highlight and this page, I suspect initially read OpenAI's reign of terror came to an end and the authors, and they can correct me if they are watching, probably had to edit that with a comma until when zero one came out. Essentially, this page is stating that models like Claude 3.5, Sonic Grok two, Gemini 1.5 caught up with GPT four. All models are converging, they thought, because of the heavy overlaps in pre training data.

Before we get to the research section though, I covered their 2023 predictions in a video from about a year ago. Their predictions are on the left and their rating of their own predictions is on the right. There was one prediction that they rated as having failed that I think is pretty harsh. They said the Gen AI scaling craze would see a group spending over a billion dollars to train a single large scale model, and the rating was give that another year. But just yesterday we got this from the information, which is that OpenAI's projected 2024 costs for training a model is $3 billion. Note that that does not include research compute amortization as in training miniature models to see what works. This is full on training of frontier models. I would be pretty surprised if the full O one model, if you count the base model generator and the fine tuned final model, didn't cost at least $1 billion. That report in the information, by the way, said that OpenAI likely won't turn a profit until 2029, according to internal sources. Oh, and if you thought 1 billion or even 3 billion was quite a lot for training LLMs training chatbots, you might think of them. Well, how about almost 10 billion a year in 2026? Note that doesn't include the playing around research compute costs of more than $5 billion in that same year. Just imagine how many experiments or tinkering about $5 billion can bring you the next highlight for me was the section on multimodality and Meta's movie gen. We can't actually play about with the model, which is why I haven't done a whole separate video on it. But I will play this five second extract because it is impressive how it can produce audio at the same time as video. You might have caught there the chainsaw sound effect when she was doing the garden and the car racing sounds as the car was racing on the carpet. To be honest, compared to reading the movie gentle paper, I found it more fun to play about with tools like pika AI, which are available now. And all you have to do is upload an image and then pick a pika effect like melt or explode or squish, and you get these amazing little clips. Look at Mudang here, picked up and squished all my AI insiders on Patreon. The logo. Look at this. Boom. There is the full screen explosion. And I think that is pretty cool.

Now, of course, I'm not going to ignore the fact that the Nobel Prize in physics was won by a pair of neural network scientists, AI scientists. And likewise, Sir Demis Hasabis of Google DeepMind fame co won the Nobel Prize in chemistry for alphafold. One commentator said, this represents AI eating science, and it's hard to disagree with that. In fact, now I think of it, I wonder what the odds in prediction markets are for most Nobel prizes in the sciences in the 2030s being won by people involved in AI. Obviously, some of you may argue that the AI itself should win the prize. Actually, that reminds me on a complete tangent. It's kind of related to Google DeepMind. I saw this job advert for Google DeepMind. Here it is. It's a new one. It's a research scientist job position. But you need to have. Where is it? Down here. You need to have a deep interest in machine cognition and consciousness. So maybe the prospect of AI winning the Nobel Prize isn't so far fetched.

I will also note that almost all of these figures have issued stringent warnings about the future of AI. Sir Demis Hassabis comparing the threat of AI going wrong to nuclear war. John Hotfeld has recently issued a warning about the world turning into something like 1984 because of AI controlling the narrative. And of course, Jeffrey Hinton has issued a myriad warnings about the inherent superiority of artificial intelligence compared to human intelligence. I watched the interviews that he did today and yesterday, and I did pick out these two or three highlights. First, he is proud of his former student Ilya Sotskava firing Sam Altman I'm particularly proud of the fact that one of my students fired Sam Altman, and I think I better leave it there and leave it for questions.

Can you please elaborate on your comment earlier on the call about Sam Altman? OpenAI was set up with a big emphasis on safety. Its primary objective was to develop artificial general intelligence and ensure that it was safe. One of my former students, Ilya Sutskyver, was the chief scientist, and over time it turned out that Sam Altman was much less concerned with safety than with profits. And I think that's unfortunate. In a nutshell, though, here was his warning. Most of the top researchers I know believe that AI will become more intelligent than people. They vary on the timescales. A lot of them believe that that will happen sometime in the next 20 years. Some of them believe it will happen sooner. Some of them believe it will take much longer. But quite a few good researchers believe that sometime in the next 20 years, AI will become more intelligent than us, and we need to think hard about what happens then.

My guess is it probably happens sometime between five and 20 years from now. It might be longer. There's a very small chance it'll be sooner, and we don't know what's going to happen then. So if you look around, there are very few examples of more intelligent things being controlled by less intelligent things, which makes you wonder whether when AI gets smarter than us, it's going to take over control. One thing I am pretty confident in is that a world of superior artificial intelligence intelligence will be a hell of a lot weirder than AR R1. Here's an example from page 54 about brainlm, which I'll be honest, I hadn't even heard of. I did read the brainlm paper after this, though, because this line caught my eye. This model can be fine tuned to predict clinical variables, for example, age and anxiety disorders, better than other methods. In simple terms, it can read your brain activity and predict better than almost any other method, whether you have, for example, mental health challenges. That is not actually something I knew existed. And the paper is also very interesting. Not only, by the way, is brainlm inspired by natural language models, it leverages a transformer based architecture. They mask future states, allowing the model to do self supervised training and predict what comes next. If that rings a bell, with enough data and pre training, brainlm can predict future brain states and decode cognitive variables, and possibly most impactfully.

Although those other two are pretty impactful, it can do in silico perturbation analysis. In other words, imagine testing medications for depression in silicon on GPU's rather than initially with patients. BrainLm has the ability to simulate brain responses in a biologically meaningful manner. The next highlight is a fairly quick one, and you may have heard of alphafold three from Google DeepMind about predicting the structure of proteins, DNA and more. But I didn't know that there was chai one from chai discovery backed by OpenAI, which is an open source alternative. Nor did I know that its performance in certain domains is comparable or superior to alphafold three. Alphafold three, don't forget, has not been fully open sourced. One other highlight I'm going to pick out on page 96, I'm going to choose not because it's particularly revealing, but because it exemplifies the kind of stacking accelerations that we're experiencing. The chart below shows the generations of Nvidia data center GPU's. And you can see the number of months between releases is gradually, on average declining. You may wonder about the next generation, which is apparently the Rubin R 100, which is going to come in late 2025. So yes, we have the release dates coming closer and closer together on average. But also, as you can see from the right line, the accelerating number of teraflops within each GPU.

You can think of this as charting the thousands of trillions of floating point operations or calculations per second per GPU that's already fairly accelerated, right? Until you learn that the number of GPU's that we're clustering not just in one data center, but combining data centers is also massively increasing. And of course the amount of money that people are spending on these GPU's. And that's all before we consider the algorithmic efficiencies within models like the O one series. This is why, and I've said this before on the channel, I think the next two years of progress is pretty much baked in. I did a massive analysis video on my Patreon, but suffice to say the next 10,000 x of scale is pretty much baked in. Which brings me to the next highlight. And it's a fairly quick one, but China wants some of that scale.

I just thought you guys might enjoy this quick anecdote given that h 100s aren't allowed to be sold to China. But a malaysian broker had a way of sticking to those rules, kind of. Nvidia coordinated the rental, installation and activation of servers based in a malaysian town adjacent to the Singapore or border. Nvidia inspectors checked the servers there and left shortly afterward. The servers were whisked away to China via Hong Kong, depending on the modality. China, it seems to me, is between three and twelve months behind the frontier, but not much more than that. I'm not drawing any conclusions. It was just interesting for me to read this and hopefully for you too.

Speaking of renting and costs, there is this chart going around about the hundred x drop in price of frontier models from OpenAI or with anthropic a 60 x drop from the cost of Claude three Opus to Claude three haiku. But I've never believed those kind of drops because we're not comparing like for like in simple bench performance. For example, Opus is just streets ahead of haiku. It's not even close, but on page 110 the report had this comparison which for me feels a bit more accurate to what has been achieved, and it is still dramatic even comparing, quote the same model, Gemini 1.5 Pro from launch to second half of 2024, it's a 76% price cut. Likewise for Gemini 1.5 flash, it's an 86% price cut. This, remember, is for roughly equivalent, if not superior performance. So if we try to keep the average bar of performance the same, that feels about right.

An 80% cut in price for the same actual performance. We wouldn't want to extrapolate too far. But if that rate of price cut carried on for one, two, five years, it would be a wild world we would live in. And yes, by the way, it would be a world largely dominated, I think still by transformers. As we've seen with brainlm and zero one, the full capabilities of transformers are still being discovered and they represent a three quarters market share. Next highlight is a quick one on copyright, and we all know probably that OpenAI transcribed millions of hours of YouTube videos to power its whisper model. Did you know that runwayml and Nvidia also mass scraped YouTube? But I thought some of you might be interested in the fact that people are trying to create business models now, like Caliop networks, so that creators can sell their YouTube videos to be scraped, basically get paid for what's already happening for free.

I know there are plenty of YouTube creators who wouldn't mind a tiny slice of that hundred billion in revenue that OpenAI are projecting. However, we probably should bear in mind Mark Zuckerberg's words, which are that creators and publishers overestimate the value of their work for training. Aih, obviously it's too long to go into here, but I think zero one slightly justifies this claim, it seems to me, and obviously this is a very early take, but as long as there is some data within the model that pertains to the kind of reasoning that must be done to answer a particular problem, the O method will find it doesn't necessarily need abundant examples, just some. I will likely not be making that point though.

I'm just wondering if anyone offers me a check for scraping my old YouTube videos. By the way, it's not just videos. Teams are working on getting authors paid for their works use by AI companies now I've just had a thought that if it were possible somehow to reverse engineer the data sources that were most responsible for model performance, I don't think any of these companies would have an incentive to find out those data sources. While it's all just this mass aggregated data scraped from the Internet, there's little kind of clear responsibility to pay this person or that. I'm basically saying that the research talent that is probably most able to do the kind of engineering necessary to ascertain the most useful, unique sources of data for the models is the very same talent least incentivized to do that research. Anyway.

Speaking of kind of externalities like copyright payment, how about environmental effects? Well, all of these companies, as I have covered before, made various pledges saying that they're going to be completely carbon neutral. I should probably reflect on my use of the word pledges there. They're more kind of of aspirational brainstorming notes without making things too complicated. Power usage because of AI, among other things, is going to skyrocket. Now, bear with me though, that doesn't matter, according to Eric Schmidt, formerly CEO of Google, because we weren't going to hit our targets anyway.

My own opinion is that we're not going to hit the climate goals anyway because we're not organized to do it. We're not. And the way to do it is with the things that we're talking about now. And yes, the needs in this area will be a problem, but I'd rather bet on AI solving the problem than constraining it and having the problem. Now, just in case that doesn't reassure you, here is Sam Altman and Ilya Sutskva, back when they still worked together, saying that AI will solve climate change. I don't want to say this because climate change is so serious and so hard of a problem, but I think once we have a really powerful superintelligence, addressing climate change will not be particularly difficult for a system like that. We can even explain how. Here's how we solve climate change. You need a very large amount of carbon cap of efficient carbon capture. You need the energy for the carbon capture, you need the technology to build it, and you need to build a lot of it.

If you can accelerate the scientific progress, which is something that a powerful AI could do, we could get to a very advanced carbon capture much faster. We could get to a very cheap power much faster. We could get to cheaper manufacturing much faster. Now, combine those three. Cheap power, cheap manufacturing, advanced carbon capture, and we build lots of them. And now you sucked out all the excess CO2 from the atmosphere. You know, if you think about a system where you can say, tell me how to make a lot of clean energy cheaply with one addition that not only you ask you to tell it, you ask you to do it.

Of course, I don't want to be too facetious. AI genuinely does lead to efficiencies which reduce energy usage. I just thought that it might be worth flagging that until a super intelligent AI solves climate change, AI data centers might raise your electricity bills and increase the risk of blackout. For many of you, this will be just a trivial externality. For others, pretty annoying. Drawing to an end now, just three more points. The first is a quick one, and it's basically this. In a nutshell.

Jailbreaking has not been solved. There was page after page after page proving that many, many jailbreaking techniques still work. Stealth attacks, sleeper agents, instruction hierarchies being compromised within hours, and on and on and on. I just wanted to summarize, for anyone who wasn't up to date this, that jailbreaking has definitely not been solved. It brings to mind early last year when some researchers thought that jailbreaking was essentially a solved problem. Amade, to his credit, said at the time, this is 2023. We are finding new jailbreaks every day. People jailbreak Claude, they jailbreak the other models. I'm deeply concerned that in two to three years, that would be one to two years from now, a jailbreak could be life or death. But as page 203 points out, are we focused on the wrong harms?

And this was such an interesting chart on the right of the actual misuses of Genai that I tried to find the original source. I will say, as a slight critique of this report, they do make finding the original sources a bit harder than I would have thought. But anyway, it was this June paper from Google, DeepMind generative AI, a taxonomy of tactics and insights from real world data. So, in short, these are the actual misuses that are happening now with Gen AI. I'll put the definitions up on screen, but you can see that the most frequent one is impersonating people like those robocalls in New Hampshire pretending to be Joe Biden, but obviously not being from him, or generating non consensual intimate images. Obviously the chart when we get AGI and ASI, artificial superintelligence might look very different, but for 2024, these are the current harms.

Last, of course, I had to end on the report's predictions for next year, and I do have one slight critique of these predictions. There are a lot of unquantitative words thrown in. Frontier labs will implement meaningful changes to data collection practices. Well, who defines meaningful? Lawmakers will worry that they've overreached with the EU AI act. This one is more firm. An open source alternative to OpenAI zero one surpasses it across a range of reasoning benchmarks. That one I'm going to go with. Probably not. That's a close one. I think end of 2025 it might get really close, but zero one is going to be something quite special.

Even zero one preview is already yes, OpenAI have given a lot of hints about how they did it, but I will stick my neck out and say that an open source alternative won't surpass zero one. Maybe llama four mid to late next year gets close, but I don't think surpasses it. Anyway, back to subjective words. We've got challengers failing to make any meaningful dent. Well, what does that mean? In Nvidia's market position? Levels of investment in humanoid robots will trail off. Does that mean go down or just increase less quickly? I don't think levels of investment in humanoid robots will go down for sure. I think it will keep going up. Nine is obviously a very firm prediction, which is a research paper generated by an AI scientist is accepted at a major ML conference or workshop. I'm going to say no to that one. So that one was a clear, objective prediction, and I'm going to take the other side of it.

I'll give you one quick reason why, and you can let me know what you think in the comments. But I think the authors of the paper will have to make it clear that an AI wrote it for ethical reasons, and then the conference will likely reject it, even if it's a good paper, because it was done by AI. That's even assuming it's good enough for a proper major ML conference, which I don't think it would be. Not in 2025 2027. Very different story. And again, we have the language. A video game based around interacting with genai based elements will achieve breakout status. Well, what does that mean? Before I make my own cheeky prediction, here's how I ended last year's video. Now if I'm criticizing them, it would be remiss of me not to end the video with my own prediction. I predict that a model will be released in the next year that will break state of the art benchmarks across at least four different modalities simultaneously.

I would claim that was passed with Gypsy 4.0 in May, which broke certain benchmarks across text domains and in audio vision, understanding and other domains. But what's my prediction, which I think at the very least is quantitative, that OpenAI's valuation will double again next year absent an invasion of Taiwan by China. When you combine the zero one method with a GPT five or Orion scale model, amazing things will happen and I think the world will hear about it.

Thank you so much for watching.

Artificial Intelligence, Technology, Innovation, Openai, Ethical Ai, Ai Predictions, Ai Explained