Coder Legion - Recent posts tagged open source

How to Deploy an Open Source LLM Reliably (on Kubernetes)

Sun, 03 May 2026 07:06:11 +0000

Everyone is talking about LLMs, but very few people actually run one.

Most developers rely on APIs from OpenAI or Anthropic. It works great until you start thinking about cost, control, or what happens when that API isn’t available. That’s when a more interesting question shows up: can you run your own LLM reliably?

Not just something that works once on your laptop, but something that behaves like real infrastructure. Something observable, restartable, and usable through an actual interface. That’s what this guide is about. We’ll deploy an open-source model (TinyLlama) on Kubernetes using Kind, serve it using Ollama, build a simple chatbot UI, and monitor everything using Prometheus and Grafana. By the end, you won’t just have a working setup you’ll understand how LLMs behave as systems.

Understanding What We’re Actually Building

Before running commands, This setup is a small but complete system: a Kubernetes cluster running an LLM service, connected to a UI, with a monitoring layer observing everything. When you send a message in the chatbot, it flows through multiple layers before coming back as a response.

User → Chat UI → LLM API → Kubernetes → Metrics → Grafana

That flow is the entire point. Once you understand this, you stop seeing LLMs as magic models and start seeing them as workloads running on infrastructure.

Why Not Just Use Gemini or Any Commercial LLM?

At this point, a fair question comes up. Why go through all this effort setting up Kubernetes, deploying an LLM, adding monitoring when you can just open a browser, go to Google Gemini or use APIs from OpenAI, and get answers in seconds?And honestly, for many use cases… you should. Commercial LLMs are fast, powerful, and incredibly easy to use. They handle scaling, reliability, and updates for you. If your goal is to get high-quality answers quickly, they are the best choice.But that’s not the whole picture. Running your own LLM changes what you control.When it runs inside your infrastructure, there’s no external dependency, no per-request cost, and full control over data and behavior. The trade-off is simple: you gain control, but you take on responsibility.

This is where open-source LLMs make sense especially for internal tools, privacy-sensitive use cases, or cost-heavy workloads at scale. But smaller models like TinyLlama still lag behind commercial models in reasoning and depth. So this isn’t about replacing commercial LLMs. It’s about understanding when to own the system—and when not to.

Aspect	Open Source LLM (TinyLlama on K8s)	Commercial LLM (Gemini / OpenAI / Anthropic)
Setup	Complex (infra + deployment)	Instant (UI / API)
Cost	No per-request cost (infra only)	Pay-per-use
Latency	Very low (local)	Depends on network/API
Quality	Moderate	High
Control	Full control over system	Limited
Privacy	Fully local	Data leaves your system
Reliability	Depends on your setup	Managed by provider

The real decision isn’t which one is better it’s about which trade-offs you’re willing to make for your use case.

Clone the Project (Start Here)

Before we go deep into Kubernetes, LLMs, or observability, let’s get the system running locally. Everything in this guide is based on this repo: Clone the repository

git clone https://github.com/Ayushmore1214/llm-k8s-deployment.git
cd llm-k8s-deployment

Starting Clean

A lot of Kubernetes frustration comes from leftover state. Old configs, broken kubeconfigs, or previously installed clusters can silently interfere with your setup. So instead of debugging weird issues later, we start clean.

sudo rm -rf /etc/rancher
unset KUBECONFIG

Now we install and create a cluster using Kind. Kind is perfect here because it runs Kubernetes inside Docker, making it fast and reproducible without any cloud dependency.

kind create cluster --name drdroid-llm
kind export kubeconfig --name drdroid-llm

At this point, you already have something powerful: a working Kubernetes cluster on your machine. Everything after this builds on top of it.

Deploying the LLM

Now we deploy the actual LLM infrastructure using a Kubernetes manifest.

kubectl apply -f llm-stack.yaml

This single command creates a deployment to keep the model running, a service to expose it inside the cluster, and the container configuration that defines how the model behaves.

You can watch Kubernetes doing its job in real time:

kubectl get pods -w

Once the pod is running, we still need to load the model itself. Until this step, the system is alive—but empty.

kubectl exec -it $(kubectl get pods -l app=ollama -o name) -- ollama pull tinyllama

Now the model is actually usable. This is the moment where infrastructure turns into intelligence.

Observability Stack

Most guides stop after it works. That’s not enough. If you can’t see what your system is doing, you don’t really control it. That’s why we add observability using Prometheus and Grafana.

helm install obs prometheus-community/kube-prometheus-stack \
  --set prometheus.prometheusSpec.resources.requests.memory=300Mi \
  --set grafana.adminPassword=admin

This adds a full monitoring layer to your cluster. Prometheus collects metrics, and Grafana lets you visualize them. Instead of guessing what’s happening, you can now see CPU usage, memory consumption, and pod health in real time.

Accessing the System

By default, Kubernetes services are internal. To interact with them locally, we forward ports.

kubectl port-forward svc/ollama-service 11434:11434
kubectl port-forward svc/obs-grafana 3000:80

Now your LLM API is available on port 11434, and Grafana is accessible on port 3000. At this point, everything is running but it’s still not very interactive. That’s where the UI comes in.

Adding a Chatbot UI

Instead of calling APIs manually, we build a simple chatbot interface using Streamlit.

pip install streamlit requests
python3 -m streamlit run app.py

Open the UI in your browser and start asking questions. Things like:

“Which model are you?”
“What’s your latest knowledge?”
“Explain Kubernetes simply”

This is where the system finally feels complete. You’re interacting with your own LLM, running on your own infrastructure.

Watching the System Under Load

Now open Grafana at http://localhost:3000 and log in using admin/admin. Navigate to the Kubernetes dashboards and select the Ollama pod.

Go back to your chatbot and start sending prompts. You’ll notice something interesting immediately. Every request creates visible activity—CPU spikes, memory usage increases, and resource patterns start forming. The model isn’t just answering questions. It’s consuming compute. This is the shift in understanding that matters. LLMs are not just software. They are workloads.

What This Demo Actually Teaches You ?

This project might look simple, but it introduces some core ideas that show up in real systems:

Kubernetes ensures your service stays alive even if it crashes
LLM inference consumes real, measurable resources
Observability is what turns debugging into something practical

Once you see these pieces working together, the abstraction breaks in a good way. You stop treating LLMs as black boxes and start understanding how they behave under the hood.

Conclusion

Running an LLM locally is cool. Running it on Kubernetes with a full observability stack? That’s how you actually learn how AI behaves in the real world. This setup is just the beginning. It’s containerized, it’s orchestrated, and it’s observable this is the exact pattern that powers production AI at scale. Once you see the CPU spikes in Grafana and manage your own cluster, you stop looking at AI as a magic API and start seeing it as workload engineering.

The best way to learn isn’t by reading it’s by breaking stuff. I built this project to be a playground. Go ahead: delete a pod, change things in llm-stack.yaml, or try to crash the model with a massive prompt. See if your Grafana dashboard catches it.

If this guide helped you make sense of the chaos, give the repo a ⭐, and let’s connect on LinkedIn. I’m always looking for people who want to build, ship, and break things. Thank You for Reading!!!

The Story of Python Togo Building a Strong Python Community in Togo

Mon, 20 Apr 2026 12:24:12 +0000

Our Story – How Python Togo Began

At the heart of Lomé, Togo, a spark was lit not with fire, but with code.
This is the story of Python Togo, a community born from passion, fueled by resilience, and driven by the belief that Python can change lives and transform an entire tech ecosystem.

Humble Beginnings

It all started back in 2022 when @Wasiu Ibrahim (Wachiou BOURAIMA) and @Geoffrey Logovi joined the Microsoft Ambassadors program. Through this experience, they discovered the power of learning in community, mentorship, networking, and the transformative impact of volunteering.

Attending both local and international events online and in-person opened eyes to what a strong developer community could achieve. While some may say volunteering doesn’t pay, we believe it builds something far more valuable: growth, knowledge, and connection.

By 2023, Togo’s tech scene was growing rapidly. Developer communities were emerging across campuses, sparking friendships and collaborations that would lay the groundwork for what was to come. Events like DevFest Lomé, TDevFest, Tech Communities Day, MLSA Togo Meetups became meeting points for leaders who shared a common dream: building something bigger together.

Then came 2024 the year everything changed.

The Turning Point: PyCon Africa 2024

When financial aid applications for PyCon Africa 2024 opened, we saw an opportunity. We shared it widely, and many from Togo applied. Only five received support but we refused to let that limit us.

So we did what any strong community would do: we stepped up together. Some members contributed financially, others reached out through networks. Thanks to Julius Boakey from PyClubs UG / Python Ghana, several of our members received vouchers. For transport and accommodation, we pooled resources so more people could attend.

The result? A life-changing experience in Accra. We met PSF Directors Deb Nicholson, Ex Director David Mertz, Python community founders Many and Michael Young Python Ghana, Kafui Alordo Python Ho, Steve Yonkeu and organizers from around the world Samson Gody, Ditiro Rampate. And amid the conversations, workshops, and late-night chats, one idea took root:

Why not bring this energy back home?

And just like that, Python Togo was born.

PyDay Togo 2024 – Our First Big Step

Inspired by PyCon Africa, we organized PyDay Togo 2024, also known as PyCon Africa Extended Togo 2024. We expected about 50 attendees.

We got over 200 registration, and 150 attendees.

PyCon Togo 2025 – Building Something Greater

Now, we’re preparing for PyCon Togo 2025 our most ambitious project yet. This time, we want to do things right: better equipment, stronger organization, and inclusive access for participants from all corners of Togo and beyond.

This isn’t just another tech event. It’s the beginning of a movement one that empowers developers, educates students, and creates opportunities where they matter most.

Our Vision & Mission

Who We Are

Python Togo is a vibrant, grassroots community dedicated to growing the Python ecosystem in Togo. Founded by passionate developers and educators, we are united by a shared love for Python and a deep commitment to lifting up our local tech community.

We believe that open source, collaboration, and education are the keys to unlocking potential especially in places where opportunity has been scarce.

Our Vision

To create a thriving, innovative Python ecosystem in Togo that is recognized regionally and globally. We envision a future where Togo becomes a hub of Python excellence in West Africa, where developers use Python to solve real-world problems and build scalable solutions.

Our Mission

We are committed to:

Promoting Python as a tool for innovation and education
Training the next generation of developers
Fostering collaboration and knowledge-sharing
Creating pathways to professional growth
Launching impactful open-source projects
Building partnerships that strengthen the tech landscape

Our Values

Collaboration: We grow together.
Innovation: We challenge the status quo.
Excellence: We strive for quality in everything we do.
Inclusivity: Everyone belongs here.

Our Strategic Goals

Skills Development: Regular training sessions, workshops, and hackathons
Community Growth: Build a community of over 1,000 active members by 2027
Strategic Partnerships: Collaborate with schools, companies, and global communities
Open Source Projects: Solve challenges with Python-driven tools
International Visibility: Represent Togo at major Python events worldwide
Diversity & Inclusion: Promote equal participation, especially among women

Join the Movement

Whether you're a developer, student, educator, company, or supporter of open source, you have a place in our story.

You can help us:

✅ Attend and support our events
✅ Mentor and teach others
✅ Sponsor our initiatives
✅ Partner with us for long-term growth
✅ Contribute to open-source projects
✅ Share our mission with your network

Get Involved Today

Join our Discord community
View our sponsorship opportunities
Email us at sponsorship@pytogo.org

Let’s build the future of Python in Togo together.

Thank You

If you’ve made it this far, thank you. Whether you’re reading this as a curious learner, a potential sponsor, or someone looking to give back, you’re part of what makes this possible.

Python isn’t just a language.
It’s a community.
It’s a movement.
And today, it’s alive and growing in Togo.

Welcome to Python Togo.
Let’s code the future together.

Connect & Follow Us

Kindly like and share your thoughts in the comments below. See you in a future article!