On the Von Neumann Tax

I have always been fascinated by the Von Neumann tax. Virtually every computer today separates "thinking" from "remembering." The processor does the math, and the memory holds the numbers. Every time a neural network generates a word, it must fetch billions of weights, carry them across a wire, multiply them, and write the answer back. Naturally, this constant shuttling of data leads to massive energy costs and compromised inference speeds.

This is why I was so intrigued by AMD's acquisition of Taalas (Canada proud :P) a few days ago. Taalas bypasses this memory wall by physically etching AI weights directly into silicon. This way, the chip does not need to load the model, because it is the model. Eliminating that travel time allows an 8-billion parameter model to run at 17,000 tokens per second.

While this is an exciting approach, there is a pretty big catch. Because those weights are literally hardwired into the metal of the chip using ROM, the model is permanently frozen. If you want to update the AI, you cannot just download a software patch; you have to design and manufacture a brand-new physical chip, or at least new metal mask layers.

This is exactly why memristors might be a more sustainable way to solve the Von Neumann tax. A memristor is essentially a microscopic resistor with a memory. Instead of storing a digital 1 or a 0, you can set its physical conductance (the inverse of its resistance) to represent a specific weight in a neural network. When you pass a tiny voltage through it, the memristor's conductance naturally computes the output current via Ohm's law (I = V × G).

This is a significant shift because it allows for extreme inference speeds, but unlike the permanently etched silicon from Taalas, memristors are completely reprogrammable. By applying a specific programming voltage, you can rewrite the resistance state. You get the exact same in-memory efficiency, but you retain the flexibility of software.

We have spent decades forcing physics to emulate perfect digital logic to run our models. As we hit the physical and thermal limits of modern hardware, perhaps the next massive leap in AI will come from simply letting nature do the math and giving the crown back to analog computers.

Here is AMD's press release: AMD Newsroom