Clayton Coleman

@smarterclayton

Kubernetes maintainer, shipper, and ancient grey beard

Joined June 2016

16 Following

2,129 Followers

Replies Reposts Media

Clayton Coleman
@smarterclayton
Mar 31
I’m very proud of this work by Kaushik and Ben in llm-d inference scheduling - achieving almost 2x p50 TPOT with through an online XGBoost model informing load balancing of LLM model servers on real world traffic: llm-d.ai/blog/predicted… Don’t leave perf on the table!
Predicted-Latency Based Scheduling for LLMs | llm-d
From llm-d.ai
Clayton Coleman
@smarterclayton
Feb 6, 2025
Excited to about the first release of the Kubernetes LLM inference gateway extension today github.com/kubernetes-sig… - improves tail latency and LoRA affinity on vLLM, adds standard operational tooling and metrics, and is a foundation for large scale LoRA serving platforms!
Release v0.1.0 · kubernetes-sigs/gateway-api-inference-extension
From github.com
3
Clayton Coleman
@smarterclayton
Jun 26, 2024
Thank you for having me on, Kaslin!
Kaslin Fields
@kaslinfields
Jun 26, 2024
This one's long, but so, SO worth it! @soltysh, @smarterclayton, and Dawn share insights not just from the beginning of Kubernetes, but from even before. Plus, they're all still actively leading efforts in the community to drive Kubernetes improvements for all users!
Quote
Kubernetes Podcast from Google
@KubernetesPod
Jun 26, 2024
In this 4th & final episode of our K8s 10th anniversary series, we interview @soltysh, @smarterclayton, & Dawn Chen. They share experiences from the earliest days of Kubernetes, & explore what the project is doing today to enable AI-style workloads. kubernetespodcast.com/episode/229-ai…
Clayton Coleman
@smarterclayton
Jun 4, 2024
Happy #kuberTENes! I can now put "10 years of kubernetes experience" on my resume without lying! #ContribCard
Kubernetes ContribCard
From contribcard.clotributor.dev
3
Clayton Coleman
@smarterclayton
Mar 16, 2024
Excellent! I also want to talk to folks who are scaling to hit specific per-token latency objectives while keeping optimal throughput - what sorts of pain points prevent that and what can we improve in the overall stack?
Age-restricted adult content. This content might not be appropriate for people under 18 years old. To view this media, you’ll need to log in to X. Learn more