Right now, ChatGPT lives in data centres around the world. But in the not-to-distant future, large language models (LLMs) like ChatGPT will run locally on portable, ubiquitous devices.
Wikipedia on the go
I remember the first time I held a microSD card. I needed one to store a tiny Python program and the device I was working with could only load Python programs from external storage.
Professor Khai Truong had gone to Best Buy and bought a few cards for the lab. He handed me an 8GiB card, the smallest capacity Best Buy had available, and said I could ask for a larger-capacity card if I needed it. For a moment, I laughed; this card had about a million times more storage than I needed. But that feeling of amusement was quickly replaced by a feeling of awe: 8GiB was enough to hold the entirety of Wikipedia’s text content at the time: the sum of all (general) human knowledge in something about the size of my pinky nail. The original iPod from about five years earlier had less than a tenth of that capacity.
The thought of carrying Wikipedia around was quite appealing: I didn’t have a data plan (I was extremely late onto the bandwagon for that one), so I installed an off-line Wikipedia reader to have access to Wikipedia from everywhere. Even if I found myself in some post-apocalyptic scenario, I could still read the entries on farming and get a garden going. At least assuming I still had electricity to run my devices and someone hadn’t defaced the relevant Wikipedia pages. I’m pretty clumsy, so I’d probably also sever a limb in the process, but at least I’d also have the Wikipedia entries on first aid.
ChatGPT on the go
Just like I originally needed to be connected to the Internet to have access to Wikipedia, I currently need to be connected to the Internet to access ChatGPT. But I’m betting that a future where an LLM is running on your phone or even wireless headphones is pretty close. Right now, data centres are doing all the AI heavy lifting with powerful GPUs. But as you read this, I’d be surprised if companies like Apple, Google, Nvidia, Samsung, and Meta weren’t all working on getting chips optimized for running inference (the generation step of LLMs, as opposed to the training side of things) onto consumer devices.
Up until now, there hadn’t been a siren call to a singular solution for providing AI in your pocket. A central processing unit (CPU) can perform all sorts of computation: It’s the “compute” part of a computer that runs the code for you to fire up a web browser or type up a document in a word processor. A CPU can do all sorts of computes including AI things like finding the shortest route between a monster and you in a computer game or to evaluate chess positions for an AI to beat you at chess. And, while a CPU is a generalist, we have other hardware to do things like efficiently create 3D worlds; that’s often done using graphics processing units (GPUs) that are designed to do the types of computations used heavily in graphics processing (hence the name) that would otherwise overwhelm a CPU. And it turned out that GPUs were also better at doing the things we needed for LLMs than a CPU, so we have been using them there, too. But just as CPUs weren’t optimized for graphics processing, there’s probably room for targeting running (large) language models.
Companies are probably trying to crack the nut of running language models efficiently. And notice I dropped the word “large” in relationship to language models. I suspect that, over time, we’ll get better at cramming more capability into smaller models and that we’ll see small models become capable enough to do many tasks. Like, on my wireless ear buds, maybe I just need enough intelligence to have it understand requests to find my missing phone, set a reminder (that lives in the ear buds), or translate some sentences between English and a local language if I’m travelling. On my phone, I might cram a much more capable digital assistant. On my thermostat, maybe just enough language to understand if someone in the room is complaining about the temperature.
Money and resources are pouring into this, I’m sure, and the rate of progress is going to be staggering. In data centres, I’m sure even GPUs will be displaced to at least some extent by inference accelerators and the fine folks at OpenAI (and other tech companies) will be able to bounce their ideas off of a future ChatGPT running with these accelerators to create a virtuous cycle of improvement. We’re still bootstrapping ourselves into an AI-accelerated world, so the change isn’t going to happen overnight. New chips take time; we’ll probably see some visible gains in the next year or two, especially in consumer-grade equipment, but it will likely be the better part of a decade from now before the accelerators become dominant in data centres.
Having a digital assistant on my phone with which I can have a realtime conversation with that has similar intelligence to the ChatGPT of the present is probably about six or seven years out. That’s going to require a combination of hardware performance improvements (for realtime speech), hardware efficiency improvements (for running on a battery), hardware capacity (for storing a model), and model efficiency. If we drop the realtime requirement and allow a bit of a pause between user request and computer response, we’re probably just three or four years out. And if we’re willing to keep things running in the cloud? I think many of us will be willing to pay a few dollars a month to run the occasional voice query in the cloud. Or even, as people have been working for decades towards, a fair bit more to have an always running digital assistant ready to schedule appointments or screen calls using cloud compute resources.
The idea of ubiquitous LLMs excites me more than Wikipedia on the go: If I sever my limb in a post-apocalyptic farming accident, I can have my first-aid conversation hands-free with an AI and I’ll only have to worry that about a 10% chance that the AI hallucinates a cure to regenerate my arm with lizard DNA.