Training an AI takes three things:
- Compute (ie computing power, hardware, chips)
- Electricity (to power the compute)
- Training data
Compute
Compute is measured in floating point operations (FLOPs). GPT-3 took 10^23 FLOPs to train, and GPT-4 plausibly 10^25.
The capacity of all the computers in the world is about 10^21 FLOP/second, so they could train GPT-4 in 10^4 seconds (ie two hours). Since OpenAI has fewer than all the computers in the world, it took them six months. This suggests OpenAI was using about 1/2000th of all the computers in the world during that time.
Energy
GPT-4 took about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million
Training Data
This is the text or images or whatever that the AI reads to understand how its domain works. GPT-3 used 300 billion tokens. GPT-4 used 13 trillion tokens (another source says 6 trillion).
- GPT-5 might need about 1% the world’s computers, a small power plant’s worth of energy, and a lot of training data.
- GPT-6 might need about 10% of the world’s computers, a large power plant’s worth of energy, and more training data than exists. Probably this looks like a town-sized data center attached to a lot of solar panels or a nuclear reactor.
- GPT-7 might need all of the world’s computers, a gargantuan power plant beyond any that currently exist, and way more training data than exists. Probably this looks like a city-sized data center attached to a fusion plant.
- Building GPT-8 is currently impossible. Even if you solve synthetic data and fusion power, and you take over the whole semiconductor industry, you wouldn’t come close. Your only hope is that GPT-7 is superintelligent and helps you with this, either by telling you how to build AIs for cheap, or by growing the global economy so much that it can fund currently-impossible things.
GPT = Generative Pre-trained Transformer = කලින් පුහුනු කල තොරතුරු ඇසුරෙන් දෙයක් මනුස්සයෙක් කරන විදියත විස්තර කරන්න හදන එක වගේ තෙරුමක්
https://www.cnbc.com/2023/05/10/mic...er-from-sam-altman-backed-helion-in-2028.html
https://www.astralcodexten.com/p/sam-altman-wants-7-trillion
Last edited:
