Way back in the day, some people decried the amount of energy used to perform a search on a search engine. Those people’s heads would explode at the energy consumed by an LLM to respond to a message that just said “hello” or “thanks”. If that doesn’t do the trick, just tell them the amount of energy required to train one of the models. Does this mean it would be terrible to replace a call centre with AIs?
Training
First off, a quick estimate. GPT-4 took somewhere around 20 million GPU-hours to train. At around 1kW per GPU (including cooling and the rest of the system attached to it), a model takes about 20GWh to train. At the end of it all, we have a system that knows a fair bit about most topics such that it can hold a conversation about many of them, in multiple languages.
The total power consumption in North America in 2020 (selected since it was prior to widespread LLM usage) was about 4PWh (4 million GWh). That works out to about 10MWh per person per year, give or take.1 That’s about 1.7kW per waking hour and 6.5kW per working hour.
GPT-4’s training therefore took 2000 people’s annual power consumption.2 On the other hand, GPT-4 has multiple areas it knows a fair bit about to an undergraduate level and it takes around 20 person-years for a typical person to obtain a similar level of knowledge in a particular domain. I think it wouldn’t be too far off the mark to say that it has a similar amount of knowledge to undergraduates and trades people from 100 different field along with being a polyglot.3 That’s 2000 peoples’ knowledge crammed in a single model.
Inference
From a marginal energy cost per query, ChatGPT does even better. You want to ask your colleague a question or make a call to a call centre? An LLM can probably response to each round of your queries at least 6x faster than a human. Unlike the context switching penalty incurred by humans, the servers running the LLM don’t skip a beat or idle between answering questions from different people. Plus, there are energy savings because you’re not idle or context switching while you’re waiting for an answer.
Let’s suppose a typical query takes about an LLM server about 10 seconds to answer vs 60 seconds for another human at your beck and call. With an LLM server consuming about 4kW for inference, that’s 11Wh for an LLM to answer your query. For a human at 1.7kW per waking hour, that’s 28Wh. You also spent 10s waiting for an LLM’s answer and 60s waiting for a human (plus even more due to avoiding context switching time). So, really, we’re looking at 16Wh vs. 56Wh — and that’s not even counting the other person’s time waiting for your response if it’s a dialog!
LLMs come out 3.5x better for inference!
Lifecycle cost of an LLM
The shelf-life of an LLM is fairly short; newer models get released fairly regularly. There are also many competing models. While LLMs have, at least to date, found some continued usage, because they are less resource-heavy and good enough for a particular task, let’s call the working life 180 days for an LLM.
We know LLMs are around 3.5x more efficient than humans at inference from an energy cost. If a human can get to an answer faster overall (maybe an LLM just doesn’t understand the problem and you need many back-and-forths to achieve the same results), that muddies the equation a bit, but let’s ignore that in this post.
How many messages do we need to send to an LLM before we break even, ignoring the amount of time required to raise a human to answer the question?
500 million messages. ChatGPT has about 500 million users, so the average user just needs to send a handful of messages for a net savings!
- This doesn’t account for other forms of energy consumed by humans like gas and food, nor does it account for embodied energy from imports. And North Americans import a lot of things. ↩︎
- Newer, larger models require more power to produce. ↩︎
- If people’ memories were good enough and we had longer lifespans, we could get 100 certifications in about 200-300 years Add on another couple hundred years for learning all those languages. LLMs still fare pretty well. ↩︎