This wasn't discussed in the episode, but I found the Doom bit in the compiler video extremely weird. The game is _ancient_, if CCC outputs basically any kind of code at all, I'd expect it'd be able to just run Doom normally on modern machines - instead of it having _even lower_ framerate and resolution than the 1993 original. Somehow that makes the original claim more fantastical, not less.
Yes programming is cooked not because ai is good, but it can spit out bang average code is seconds that is good enough of majority of big tech companies, so i guess getting paid to do programming is essentially dead
Nice talk again! Lots of interesting points, starting with the two-axis framework to assess AI-related claims. I also liked the perspective of interpreting LLMs as decompressors from natural language to whatever artifacts they produce.
With respect to the Dota 2 segment, another caveat is that the game was significantly simplified by removing most of the hero pool (from 100+ options down to 17).
This takes many mechanics out of the game and makes the pro players play at a disadvantage since competitive strategies have to be reinvented.
> We spent several weeks training with hero pools up to 25 heroes, bringing those heroes to approximately 5k MMR (about 95th percentile of Dota players). Although they were still improving, they weren’t learning fast enough to reach pro level before Finals. We haven’t yet had time to investigate why, but our hypotheses range from insufficient model capacity to needing better matchmaking for the expanded hero pool to requiring more training time for new heroes to catch up to old heroes. Imagine how hard it is for a human to learn a new hero when everyone else has mastered theirs!
> We believe these issues are fundamentally solvable, and solving them could be interesting in its own right. The Finals version plays with 17 heroes—we removed Lich because his abilities were changed significantly in Dota version 7.20.
--------------------------
So... an incomplete version of the game works, and we _believe_ it can be solved fully, and we leave it at that :)
This wasn't discussed in the episode, but I found the Doom bit in the compiler video extremely weird. The game is _ancient_, if CCC outputs basically any kind of code at all, I'd expect it'd be able to just run Doom normally on modern machines - instead of it having _even lower_ framerate and resolution than the 1993 original. Somehow that makes the original claim more fantastical, not less.
Yes programming is cooked not because ai is good, but it can spit out bang average code is seconds that is good enough of majority of big tech companies, so i guess getting paid to do programming is essentially dead
Nice talk again! Lots of interesting points, starting with the two-axis framework to assess AI-related claims. I also liked the perspective of interpreting LLMs as decompressors from natural language to whatever artifacts they produce.
With respect to the Dota 2 segment, another caveat is that the game was significantly simplified by removing most of the hero pool (from 100+ options down to 17).
This takes many mechanics out of the game and makes the pro players play at a disadvantage since competitive strategies have to be reinvented.
In their [blog post](https://openai.com/index/openai-five-defeats-dota-2-world-champions/) they say:
> We spent several weeks training with hero pools up to 25 heroes, bringing those heroes to approximately 5k MMR (about 95th percentile of Dota players). Although they were still improving, they weren’t learning fast enough to reach pro level before Finals. We haven’t yet had time to investigate why, but our hypotheses range from insufficient model capacity to needing better matchmaking for the expanded hero pool to requiring more training time for new heroes to catch up to old heroes. Imagine how hard it is for a human to learn a new hero when everyone else has mastered theirs!
> We believe these issues are fundamentally solvable, and solving them could be interesting in its own right. The Finals version plays with 17 heroes—we removed Lich because his abilities were changed significantly in Dota version 7.20.
--------------------------
So... an incomplete version of the game works, and we _believe_ it can be solved fully, and we leave it at that :)