Reply to thread

Message: <blockquote data-quote="SLHodahitha" data-source="post: 29592163" data-attributes="member: 565060">Training an AI takes three things:<ul> <li data-xf-list-type="ul">Compute (ie computing power, hardware, chips)</li> <li data-xf-list-type="ul">Electricity (to power the compute)</li> <li data-xf-list-type="ul">Training data Compute Compute is measured in floating point operations (FLOPs). GPT-3 took 10^23 FLOPs to train, and GPT-4 plausibly 10^25. The capacity of all the computers in the world is about 10^21 FLOP/second, so they could train GPT-4 in 10^4 seconds (ie two hours). Since OpenAI has fewer than all the computers in the world, it took them six months. This suggests OpenAI was using about 1/2000th of all the computers in the world during that time. Energy GPT-4 took about <a href="https://www.ri.se/en/news/blog/generative-ai-does-not-run-on-thin-air" target="_blank">50 gigawatt-hours</a> of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million Training Data This is the text or images or whatever that the AI reads to understand how its domain works. <a href="https://lambdalabs.com/blog/demystifying-gpt-3" target="_blank">GPT-3</a> used 300 billion tokens. <a href="https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/" target="_blank">GPT-4</a> used 13 trillion tokens (another source says 6 trillion). <img src="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e29cdd8-3523-4386-b574-7a6f7c7fb0e4_1083x748.png" alt="" class="fr-fic fr-dii fr-draggable " style="" /></li> <li data-xf-list-type="ul">GPT-5 might need about 1% the world’s computers, a small power plant’s worth of energy, and a lot of training data.</li> <li data-xf-list-type="ul">GPT-6 might need about 10% of the world’s computers, a large power plant’s worth of energy, and more training data than exists. Probably this looks like a town-sized data center attached to a lot of solar panels or a nuclear reactor.</li> <li data-xf-list-type="ul">GPT-7 might need all of the world’s computers, a gargantuan power plant beyond any that currently exist, and way more training data than exists. Probably this looks like a city-sized data center attached to a fusion plant.</li> <li data-xf-list-type="ul">Building GPT-8 is currently impossible. Even if you solve synthetic data and fusion power, and you take over the whole semiconductor industry, you wouldn’t come close. Your only hope is that GPT-7 is superintelligent and helps you with this, either by telling you how to build AIs for cheap, or by growing the global economy so much that it can fund currently-impossible things. GPT = Generative Pre-trained Transformer = කලින් පුහුනු කල තොරතුරු ඇසුරෙන් දෙයක් මනුස්සයෙක් කරන විදියත විස්තර කරන්න හදන එක වගේ තෙරුමක් <a href="https://www.cnbc.com/2023/05/10/microsoft-agrees-to-buy-power-from-sam-altman-backed-helion-in-2028.html" target="_blank">https://www.cnbc.com/2023/05/10/microsoft-agrees-to-buy-power-from-sam-altman-backed-helion-in-2028.html</a> <a href="https://www.astralcodexten.com/p/sam-altman-wants-7-trillion" target="_blank">https://www.astralcodexten.com/p/sam-altman-wants-7-trillion</a></li> </ul></blockquote>

Verification: Winadiyakata thappara keeyak tibeda?

Top Bottom