On Cheap Software and Expensive Compute

A few weeks back, I attended a panel at Toronto Tech Week with Reynold Xin (co-founder of Databricks) and Mike Murchison (CEO of ada CX). I expected a run-of-the-mill AI hype conversation, but I was actually very pleasantly surprised by the lucidity and bluntness of the conversation. I was especially impressed by a point Reynold made.

He pointed out that, while you can argue as much as you want about the efficiency and quality of the code that is being written by AI, it is clear that the industry is really working hard to make software cheaper to build. Yet, the reality is that compute keeps getting more expensive to run. It seems that everyone obsesses over the first curve and underrates the second, even though the gap between the two is probably where the most interesting developments of the next few years are.

What I keep coming back to is what this means for who ends up winning. We tend to assume abundance is the advantage, that the team with the most GPUs, the most capital, and the most engineers takes it. However, if compute really is the limiting factor, then that logic should invert, and the advantage should lie with whoever can build the most with the least.

I also think that building with less is more promising than people give it credit for. We already saw a preview of this with the wave of models reaching near-frontier performance at a fraction of the expected training cost, largely through smarter methods rather than bigger clusters, whether that is only activating the parts of a network you actually need, or shrinking models so they run on far cheaper hardware without losing performance. The same logic applies to serving them, where routing simple queries to small models and reserving the expensive ones for genuinely hard problems can make things cheaper and require less infrastructure.