At Open Source Summit in Minneapolis, I caught up with Jonathan Bryce, executive director of Cloud Infrastructure and the Cloud Native Computing Foundation (CNCF) at the Linux Foundation.
I first interviewed Jonathan 17 years ago, shortly after Rackspace acquired his early cloud company, Mosso. Since then, he helped establish OpenStack and led the Open Infrastructure Foundation, which joined the Linux Foundation last year.
What we talked about
We covered a lot of ground: why hardware matters again, the shift toward inference, the gap between the hype around AI agents and their actual adoption, where Jonathan sees things heading, and how you can get in on the ground floor.
A week earlier, I’d seen Jonathan present at Kubernetes Community Day in Austin, where he laid out three pillars of open source AI: training, inference, and agents. The slide he created does a good job of framing some of the topics we covered.
Hardware Matters Again
Jonathan talked about the early days of cloud, around the time they sold their company to Rackspace, when there was a growing assumption that hardware would become increasingly abstracted as value moved to the software layer above it. Servers were becoming commoditized, and data centers filled up with interchangeable, “pluck and chuck” systems that could be swapped out when they failed.
AI is changing that. GPUs, specialized accelerators, networking architectures, and other hardware choices now have major implications for performance and cost. As Jonathan put it, hardware is becoming cool again.
From Training to Inference: The Challenge of Scaling Out
Jonathan pointed to two related shifts: from training to inference, and from vertical to horizontal scaling.
For years, AI infrastructure investment went toward training ever-larger models on massive GPU clusters. Now Jonathan is seeing it shift toward inference, which can run on a broader mix of CPUs, GPUs, and accelerators. That diversity helps with cost and performance, but it also makes inference harder to scale, which brings us to the second shift.
Early on, workloads can be handled by scaling vertically, adding more processing power and memory to individual systems. As demand grows, organizations need to scale horizontally, distributing workloads across many machines. This is where cloud-native technologies come in. Scheduling, scaling, and orchestrating resources across distributed systems are exactly the problems the cloud-native community has spent more than a decade solving.
Enter llm-d
Scaling inference horizontally is no simple task. One project tackling it is llm-d, a new CNCF sandbox project that builds on vLLM, the open source inference engine. (The CNCF sandbox is the entry tier for early-stage open source projects joining the foundation.)
While vLLM handles model execution, llm-d coordinates inference across multiple systems. Its capabilities include intelligent request routing, KV-cache management, and Kubernetes-based orchestration, helping organizations distribute workloads while improving performance and resource utilization.
An Opportunity to Shape the Technology
Jonathan closed by noting that projects like llm-d are still young, with their architecture, communities, and direction all evolving. For developers and organizations interested in contributing, there’s a real opportunity to help shape the technology while it’s still taking form.
As Jonathan put it, “The future is being written right now.”
Pau for now…
Leave a Reply