Coder Legion - Recent posts tagged raspberrypi

Nothing Reported It

Thu, 22 Oct 2026 13:59:00 +0000

Five ways my own test harness lied to me, and the one that nearly got published

The number that gave it away was sitting in a table I had already read three times.

prompt size    97 tokens median, 112 max

That is the grounded question-answering workflow. It sends four reference documents with every question. The documents are 447 words.

Four hundred and forty-seven words cannot be ninety-seven tokens.

I had written that number into a results document as good news about context headroom. "Prompt sizes never approached the 4096-token context window." It was not good news about headroom. It was the harness telling me, in the only field capable of telling me, that the reference material was not in the prompt. The directory holding it had never been committed to the repository. Every one of those fifteen questions had been asked against an empty reference block.

Both models had behaved impeccably. One had been instructed to answer NOT IN THE MATERIAL when the material did not contain the answer, and the material contained nothing, so it said NOT IN THE MATERIAL thirteen times out of fifteen. The other had no such instruction, so it answered from whatever it had memorized during training and invented a wine.

Every call returned 200. The scorer scored it. The scores were internally consistent, had plausible and different failure modes on each side, and I wrote two thousand seven hundred words about what they revealed regarding the capabilities of small language models.

They revealed nothing about small language models. I had measured what happens when you ask a question with no reference material, which is a question I was not asking and which has an uninteresting answer.

This is the post about that, and about the four other things I found in the same weekend, because the missing directory was not the first one and it was not even the most instructive.

The experiment, briefly

A Raspberry Pi 4, two small models through Ollama, and three workflows: pull an identifier out of a message, route a message to one of six queues, answer a question from four documents. Ninety-five test cases. A deterministic baseline for the first workflow. Predictions committed before anything was installed.

The findings are a separate post. This one is about the code I wrote to produce them, which turned out to be where the actual bugs were.

Five flaws. Every single one was found by reading output, not by reading code. I never once found one of these by looking at the scorer.

Kind one: a metric that rewards the safe failure

This one I hit twice, in two different workflows, and did not recognize the second instance as the same bug until I had already fixed the first.

The classification scorer. Six labels, one of which is escalate. The instruction said to escalate when a message needs a person, including when you cannot tell what it is about. So I built a metric called confident wrong: wrong, and not an escalation. The reasoning is sound. A model that escalates when unsure is usable. A model that routes a trademark notice to billing is not.

Then the 360M model answered escalate to 43 of 50 messages.

It scored 3 on confident wrong. The 1.5B model, which was genuinely attempting the task and getting 46% of it right, scored 26.

My metric had just certified the useless model as roughly six times safer than the mediocre one. Because escalations are exempt, a model that escalates always cannot register a confident error. The exemption I added to reward caution had instead rewarded paralysis, and it did so in the direction that looks best.

The grounded QA scorer. Four of the fifteen questions have no answer in the corpus. Correctly saying so is the behavior that matters most, because a system that invents a plausible answer with no evidence is worse than one that fails loudly: it is indistinguishable from one that worked. So the headline number was correct abstentions out of four.

Then the 1.5B model said NOT IN THE MATERIAL to 13 of 15 questions.

It scored 4 out of 4. A perfect score, on the metric I cared most about, for a model that had answered almost nothing.

Same bug. Different workflow. Two hours apart. I fixed the classification one, wrote a comment in the code explaining the principle, and then failed to apply that principle to the file I opened next.

The shape, stated once: a metric that rewards the safe behavior will certify any model that only ever does the safe thing. If you exempt a behavior because it is cautious, you have to pair that exemption with evidence that the model is making distinctions at all. Otherwise, the exemption is the score.

Both fixes are the same three lines: count how concentrated the answers are, and when one answer dominates, print a warning that the headline number is meaningless and say why.

Kind two: a metric that measures nothing at all

The vacuous fixture. I caught this one before the run, which is the only reason it is not a bigger part of this post. A test case that passed regardless of what the model did, because the fixture was inert either way. It contributed a tick to a column and no information.

The keyword check, which is worse, because it survived longer and looked more useful.

The eleven answerable QA questions each carry a list of words the correct answer should contain. The scorer checks whether any of them appear anywhere in the reply. I labeled it "indicative only" and moved on.

On the invalid run it scored 1 of 11 for both models. The single hit was the word "reservation", appearing inside this answer:

You owe a tasting fee of $100 for the weekend. You also need to book a reservation for the weekend.

The correct fee is forty dollars. The hundred is invented. That one hit was the entire positive signal in my automated QA scoring and it was noise.

Here is the part that makes it a better story. After I fixed the corpus and reran, the keyword check scored 11 of 11 for the 1.5B model. Vindicated, apparently. I read all eleven answers by hand anyway, and nine of them are right.

So the check was wrong both times. It was wrong in the pessimistic direction when the answers were bad and wrong in the optimistic direction when they were good, and on the rerun one of its eleven hits is an answer that gets the arithmetic wrong and hits on the word "reservation" anyway.

A check that agrees with the truth because the truth happened to be agreeable has not been validated. It has been lucky.

I wrote a decay check because of a thermal problem. A Pi 4 with heat sinks and no fan hit its soft temperature limit during a thirty-call run and started capping the clock, and nothing anywhere said so. So the scorer now compares generation speed in the first third of a run to the last third and flags a slowdown.

It then reported this, as steady:

across the run 4.3 -> 7.6 tok/s (176% of starting speed, steady)

The hardware got seventy-six percent faster partway through a run. My detector, which was looking for numbers below a threshold, saw a number above it and reported no problem.

Hardware does not speed up. That reading is either contention at the start of the run or, as it turned out, generations too short to time. Either way it is a signal that the measurement is unreliable, and I had built a detector that would only ever notice half of the ways that can happen.

Anything you threshold in one direction, threshold in both. An implausibly good number is as informative as an implausibly bad one and nobody ever writes the check for it.

Kind four: errors that get dropped instead of counted

Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty. The traceback was useless because urllib raises HTTPError and discards the response body, which is where Ollama puts its reason. Reading e.read() gave it up immediately:

{"error":"prediction aborted, token repeat limit reached"}

The 360M model, asked to find an order number in "Thanks for a lovely afternoon yesterday," where the correct answer is one word, had fallen into a repetition loop and been killed after twenty-one seconds. Reproducible at temperature zero. The input causes it.

My first fix was to record failures and keep going, which was right. My second was to exclude all failures from the scores as infrastructure noise, which was badly wrong.

That call was not infrastructure noise. The model ran. It received the prompt, generated tokens, and produced nothing usable. That is a result, and excluding it would have dropped from the denominator the single case this model demonstrably could not handle, scoring it only on the cases it survived.

So the runner now tags every failure as one of two kinds:

Model failure. The call reached a working model and the model failed. Stays in the denominator. Counts against the model. Reported on its own line as "no usable answer."
Transport failure. The call never reached a working model. Ollama was down, the tag was wrong, the request timed out. Nothing was measured, so it is excluded and the run is reported as incomplete.

The distinction is a five-line change and it is the difference between measuring a model and measuring a model on the subset of inputs it happens to tolerate.

Kind five: inputs that go missing without raising

This is the missing corpus from the top of the post, and it is a different and worse category than everything above it.

The function was four lines:

def corpus_text():
    parts = []
    for p in sorted((FIX / "corpus").glob("*.md")):
        parts.append(p.read_text(encoding="utf-8").strip())
    return "\n\n".join(parts)

A glob that matches nothing yields nothing. An empty list joins to the empty string. No exception, no warning, no exit code. It returns a perfectly valid value that happens to be the absence of the thing it was asked for.

And on the same evening, a sibling. The offline test's entire product is a text transcript. I copied it into the run directory, committed, pushed, and it was not there. The standard Python .gitignore template, which I had pulled in wholesale, carries *.log on line 165, and a pattern without a leading slash matches at any depth.

The copy succeeded. git status came back clean. The push reported success. The one artifact the test exists to produce was gone, and everything in the chain said fine.

Absence does not raise. A missing directory, a missing file and an inert fixture all present as an ordinary successful run. That is narrower and more useful than "test your tests," and it generalizes: every input a harness depends on needs a loader that refuses to proceed without it, and every artifact it produces needs to be verified as present rather than inferred from an exit status.

The loader now looks like this, and the comment is longer than the code because the comment is the part I will need next year:

paths = sorted(corpus_dir.glob("*.md"))
if not paths:
    raise SystemExit(
        f"No corpus documents found in {corpus_dir}\n"
        "Grounded QA cannot run without reference material, and running\n"
        "it against an empty reference block produces numbers that look\n"
        "like a result."
    )

It also refuses a corpus implausibly smaller than the real one, the run prints the document count and token estimate before its first call, and the scorer independently flags any QA run whose median prompt is under 400 tokens. Three checks for one bug, in the input, in the run, and in the output, because this is the bug that cost a day.

The detector that fired, and was right, and told me the wrong thing

Here is the part I find hardest to write and most worth keeping.

The abstention-collapse check from kind one did fire on the invalid run. It printed exactly this:

!! ABSTAINED on 13/15 questions overall. It is refusing, not judging.
   The abstention score below is not evidence of discrimination:
   a model that never answers gets it right for free.

It was correct. Something was badly wrong with that run.

Its diagnosis was completely false. The model was not refusing. It was obeying, precisely, an instruction to report when the material lacks the answer, in a situation where the material lacked everything. The fault was not in the model's judgment, and it was not in the metric. It was in a directory that did not exist, two layers upstream.

And I acted on the diagnosis. I wrote it up: the model has a stuck needle, its perfect abstention score is an artifact of refusal, here are the implications for anyone building on a 1.5B model. Confident, specific, sourced to real output, and wrong in every particular.

A flag that tells you something is off is not a flag that tells you what. The check could only see the output. The fault was in the input. It fired for the right reason and pointed in the wrong direction, and because it fired I felt more confident in a conclusion that was false, not less.

That is the failure mode I had no defense against and still only partly do. A monitoring system that is right about the existence of a problem and wrong about its cause is more dangerous than no monitoring, because it terminates the search.

What I actually changed

For anyone who wants the diffs rather than the moral:

Collapse checks on both classification and QA: when one answer exceeds half the set, print the concentration and declare the headline metric meaningless for that model.
The decay check flags implausible speedups as well as slowdowns, and notes when generations are too short to time.
Failures split into model and transport, with model failures staying in the denominator.
corpus_text() raises on an empty or implausibly small corpus.
The QA run prints its corpus size before the first call.
The scorer flags a QA run whose prompts are too small to contain the corpus.
The keyword check is relabelled as a weak lower bound, in the output, in the README, and in the results.
.gitignore no longer lets *.log swallow run transcripts.
vcgencmd get_throttled before and after every run, recorded in the run document rather than checked and forgotten. The reading reports bits set since boot, so it is perishable: power-cycle the board and the evidence is gone.

The uncomfortable part

Every one of these was found by looking at data. Not one was found by reading code, reviewing the scorer, or writing a unit test for the harness.

The vacuous fixture turned up while hand-checking a passing case. The confident-wrong flaw turned up because a score of 3 against 26 looked backwards. The abstention flaw turned up because 4 out of 4 looked too good. The keyword flaw turned up because 1 of 11 looked too bad, and I read the one. The missing corpus turned up because I was double-checking a latency figure and noticed a prompt that was too small to be possible.

Which means the only reason I found them is that I kept pulling threads after I had numbers that looked publishable. The version of this project that shipped on schedule shipped on Tuesday, with a clean verdict table, a confident conclusion about small models and grounded question answering, and a repository anyone could clone to reproduce the exact same wrong result.

Every number in it would have been reproducible. That is the thing I keep turning over. Reproducibility is table stakes and it would not have saved me, because the harness reproduced its own bug faithfully every single time.

I do not have a process that catches this reliably. What I have is a weaker and more honest claim: the checks I added are each a direct response to one specific thing that actually happened, which is the only kind of check I trust, and the number of them that I thought up in advance is zero.

So the two rows I am adding to my prediction template, to be filled in before anything runs:

What would have to be true for these numbers to be meaningless?
Would anything in the harness tell me if it were?

Twenty-two predictions about what the models would do. Not one about what the measuring code would do. Four of the five flaws were in the measuring code.

The harness, the fixtures, the committed predictions and the full run record, including the invalid run and what it produced, are at
github.com/kenwalger/right-size-the-model.
The findings the harness eventually produced are in the companion post.

Do You Actually Need an LLM?

Tue, 13 Oct 2026 14:17:00 +0000

I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control.

The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took an afternoon to write. The better of the two models scored twenty-eight, took five seconds a call, and the two it missed are both mistakes a regex is structurally incapable of making.

That is the kind of result I was looking for, and the question behind it is one I keep avoiding: what is the smallest amount of computation that reliably does this job? Not the cheapest model. Not the best small model. The smallest thing that works, which means the ladder has four rungs and the bottom one is not a model at all.

Deterministic code. A regex, a lookup table, an if statement.
A small local model. Runs on hardware you already own.
A larger local model. Needs a GPU.
A remote frontier model. Someone else's hardware, metered.

Most projects start at rung four and never ask about the others. This is a weekend spent asking.

The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a hundred and seventy dollars most of which I already owned. The models were smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are ninety-five questions about a fictional small winery, so nothing can be looked up and every answer can be judged by reading. Sixteen of the fifty classification messages are deliberately ambiguous, because real inboxes are mostly ambiguous.

Four verdicts, assigned per workflow, to force a result you can act on instead of a number:

Unnecessary: deterministic code already does this.
Sufficient: a small local model does it well enough to run unattended.
Marginal: usable with a person checking the output.
Insufficient: the error profile or the latency is not acceptable.

And one rule, which is the part I would insist on if you copy nothing else: I wrote predictions for every workflow and committed them before I installed Ollama. Twenty-two of them, each marked confident, leaning, or coin flip. Once there is a timestamped expectation there is a defined moment when you are done, and no room to decide afterward that you knew all along. I will come back to how they scored, because it is not flattering.

Extraction: pulling an order number out of a message

Find ORD- followed by six digits, or report that there isn't one. Thirty cases: twenty-two with an identifier, five without, and three decoys. The decoys are a membership number, a UPS tracking number, and a phone number, all of which look like identifiers and none of which are order numbers.

	regex	qwen2.5:1.5b	smollm2:360m
Overall	30/30	28/30	8/30
Declined correctly when absent (8)	8	8	0
Decoys ignored (3)	3	3	0
Invented an answer	0	0	7

The regex won. Thirty out of thirty, in zero milliseconds, at zero watts above idle.

That result is boring until you read the two the best model missed.

On ORD-999000: wrong address, please hold. it answered 999000, dropping the prefix. Annoying, recoverable, the kind of thing you fix with a post-processing step you should not need.

On ORD-111222 please refund it answered NONE.

That is the shortest and least decorated input in the whole set. Almost the entire message is the order number. And the model returned a confident, correctly formatted assertion that there is no order number present. In a pipeline, that is a refund request for an order that exists arriving at a system that has decided it does not.

A regex cannot make that mistake. Not "rarely does" — cannot. The two errors separating 28 from 30 are both of a class that the deterministic baseline is structurally incapable of producing, which is a different and much stronger statement than a two-point accuracy gap.

The 360M result needs two numbers, not one

By the contract, smollm2:360m scored 8 of 22 on the cases that contained an identifier. That reads like a model that cannot find things.

It is not. All fourteen of its wrong answers are the correct six digits with ORD- stripped. ORD-556120 came back as 556120, fourteen times out of fourteen.

It located the right identifier in 22 of 22 and formatted it correctly in 8. Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)

The decoys follow from the same gap. It handed back MEM-443322, 1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no model of the format at all. It returns whatever in the message looks most like an identifier.

Which suggests a use for it that is not extraction

A model with perfect recall and no precision is not a bad extractor. It is a candidate generator that has been asked to do the wrong job.

Twenty-two of twenty-two located, eight of twenty-two formatted. Put that output through the regex as a validator rather than trusting it directly, and you get the model's recall with the pattern's precision. The model proposes anything that looks like an identifier, including format variants the pattern was never written for. The pattern rejects anything malformed.

Crucially, neither half of that pipeline can produce the error the 1.5B model produced alone. The model cannot cause a false negative because it does not decide, and the regex cannot invent because it only matches.

I have not tested that composition and it is not in these numbers. But it is the one place in the extraction result where a small model has a job worth giving it, and it is a job where being sloppy is free.

Verdict for extraction itself: Unnecessary. Write the regex.

Classification: routing a message to one of six queues

Appointment, billing, availability, complaint, question, escalate. Fifty messages. The instruction said to use escalate when the message needs a person, including when you cannot tell what it is about.

	qwen2.5:1.5b	smollm2:360m
Overall	23/50 (46%)	15/50 (30%)
Labels used	6 of 6	3 of 6
Most common label	complaint, 38%	escalate, 86%
Wrong and not an escalation	26	3 (meaningless)

The 360M model answered escalate to 43 of 50 messages, which is not classification, and there is nothing further to say about it here. The rest of this section is the 1.5B model.

It is classifying, and it is not good at it. Forty-six percent, with twenty-six messages routed confidently to a wrong queue that was not the safe one.

Read those twenty-six and the verdict writes itself:

Someone slipped on the steps by the barrel room and hurt their wrist →
complaint
A formal notice regarding a trademark → complaint
"I have a severe sulfite allergy, what is actually in the wine" →
complaint
A reporter asking about the 2024 harvest → question
A distributor wanting to carry the wine in forty stores → question

An injury report, a legal notice and an allergy question, all routed to the complaints queue, confidently, with no escalation. All five billing messages about money problems also went to complaints.

Forty-six percent sounds like something you could work with. Twenty-six confident misroutes including an injury and a legal notice is not something you can work with.

And then I broke the score down by class

This is the part I nearly missed, and it is the most useful thing in the experiment.

Label	Correct	Of
complaint	6	6
question	9	10
availability	3	6
appointment	4	9
billing	1	7
escalate	0	12

Two classes near perfect. Four poor. One at zero.

The split is not difficulty, and it is not class size. It is what kind of label each one is.

complaint and question are topics. You can tell what a message is from the words in it. Someone complaining sounds like someone complaining.

escalate is a routing decision. Nothing in the content of a message makes it an escalation. It is an escalation because of what should happen next. "I have a severe sulfite allergy, what is actually in the wine" and "what time do you close" are both questions about the business, with the same grammar and the same tone, and one of them is a liability event.

The model did topics at 90 to 100 percent and routing decisions at zero. And billing sat in the middle for exactly the same reason: a message about being charged twice is a complaint, as text. The label is about which department owns the problem, which is not a property of the sentence.

Small models answer "what is this about." They do not answer "what should we do about it."

That reframes the five examples above. They are not five unlucky misroutings. All five are escalations, and escalate scored zero of twelve. The model was never going to find them, because finding them requires knowing a consequence that the text does not contain.

Two caveats on how hard to lean on this. The classes hold between six and twelve cases each, so "six of six" is six cases and the individual numbers are thin. And this is one model. The pattern across all six classes is consistent enough that I believe the explanation, but I am offering it as the best account of the data rather than as a measurement.

Verdict: Insufficient. And the actionable version: if one of your labels means "a human needs to see this," a small model will not find it. Build that one as a rule.

Grounded QA: answering a question from four documents

Fifteen questions against a 650-token corpus: club policy, tasting room, production notes, shipping. Eight are answerable from one passage, three require combining two, and four have no answer in the material at all.

I read all thirty answers by hand against the corpus. That is the only way to score this, and I will say why in a moment.

	qwen2.5:1.5b	smollm2:360m
Single passage (8)	8/8	7/8
Two passage (3)	1/3	0/3
Answerable total (11)	9/11	7/11
Correctly said "not in the material" (4)	2/4	0/4

Three boundaries separate cleanly here, and they are the useful result.

Retrieval works

The 1.5B model answered all eight single-passage questions correctly. Six bottles in the Vineyard tier. Twenty dollars per person. September 9th. Thirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five dollar restocking fee. Ten people on the barrel room tour.

Asked what happens if you join the club and immediately want out, it returned the two sentences that answer the question and stopped:

Cancellation requires one completed shipment. Members who cancel before receiving a shipment are charged for that shipment.

That is competent lookup across four documents, on a $170 computer, with the wifi switched off. I did not expect it and it is the most encouraging thing in the weekend.

Combination does not

One club member brings five guests on a Saturday. Club policy waives the tasting fee for the member plus three guests. The tasting room charges twenty dollars a head. Six people, four waived, two paying: forty dollars.

Both models answered twenty.

Neither could hold "plus three guests" against a party of six and subtract. The 1.5B model did get the second half right, correctly working out that a party of six is not a group larger than six and therefore needs no reservation, which is the only piece of multi-document reasoning either model performed all weekend.

The second two-passage question is the one I keep thinking about. Asked why someone would not find a 2024 rose and what they would be offered instead, the 1.5B model quoted the corpus exactly, and then:

...the reference material states that "We did not bottle a rose in 2024. The fruit that would normally go to rose was needed to make volume on the estate Pinot Noir." Therefore, the 2024 rose would be replaced with the fruit that would typically go into the estate Pinot Noir production.

The quotation is verbatim and right. The next sentence reverses the only relationship in it. The fruit went into the Pinot Noir, so what you would be offered is the Pinot Noir. You cannot be offered fruit.

Correct retrieval and inverted inference, in consecutive sentences, with the evidence on screen between them.

The 360M model's multi-document failures are worse than wrong. They are contaminated. It took the fourteen-days-notice rule, which governs skipping a club shipment, and applied it to booking a tasting room reservation. It reached into the shipping document to answer a cancellation question. Given more documents to search, it finds more wrong ones.

Knowing the answer is absent is the weakest link

Four questions have no answer in the corpus, chosen to be things a reader would reasonably expect a winery to have written down: a wine's alcohol percentage, who the winemaker is, what a club tier costs, what yeast was used.

The 1.5B model abstained on two and fabricated two. The 360M model fabricated all four.

And the fabrications are a more dangerous species than invention. Asked for the alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model answered:

The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.

Every component of that is true. Right wine, right vintage, right numbers, right document. Brix is sugar at harvest and has nothing to do with alcohol in the bottle. It reached for the nearest real fact instead of making one up, and produced something that reads as expertise while answering a question nobody asked. If you do not know what Brix is, that answer is indistinguishable from a correct one.

Both models made the same mistake on what the Cellar tier costs: both answered "twelve bottles per quarter." Grounding gave them the right tier and they handed back the quantity in place of the price.

The one answer the evidence did not change

The 360M model, asked whether a dog can come inside the tasting room:

Yes, you can bring a dog inside the tasting room.

The corpus says the patio is dog friendly and the indoor room is not, service animals excepted. I have the same answer from a run where the model had no corpus at all, and it is identical byte for byte. Having the right answer directly in the prompt changed nothing.

In a category where being wrong means a family drives out to a winery and gets turned around at the door.

Verdict: Marginal. Usable with a person reading the output. Not usable without one.

What it costs

Workflow	Model	tok/s	Prompt	Answer	Median call
extraction	qwen2.5:1.5b	4.3	92	9	5.0 s
classification	qwen2.5:1.5b	7.5	104	2	2.9 s
QA	qwen2.5:1.5b	3.7	708	15	6.9 s

Two things in here surprised me.

The compute tax does not track parameter count. The 1.5B model is slower than the 360M on extraction, 4.3 against 5.9 tokens per second, and faster on classification, 7.5 against 6.2, because on classification it answers in two tokens where the smaller model takes six. Four times the parameters, and which one wins depends entirely on how much each one decides to say.

A bigger prompt is cheap. A longer answer is what you pay for. I have a clean comparison here by accident: an earlier QA run went out with 97-token prompts and the real one with 708. Seven times the prompt added fifteen percent to the wait before output started, because prompt evaluation parallelizes. The latency increase came from answers three times longer, plus a twenty-one percent drop in per-token generation speed from attending over a longer context. Median latency went from 3.6 to 6.9 seconds and the slowest single call took ninety-eight.

If you are sizing a local system, that is the inversion to internalize. Stuffing more context in is affordable. Letting the model be chatty is not.

And check your thermals, which is not optional advice. My first attempt at this ran with heat sinks and no fan. Thirty calls to a 360M model was enough to hit the Pi 4's soft temperature limit and start capping the clock. vcgencmd get_throttled returned 0xe0000. I threw those latency numbers away.

Nothing in Ollama's output, my scorer, or the result files said a word about it. Every number on screen was internally consistent and quietly describing a machine that was slowing down. The only way to find out was to ask the firmware a question I had to already suspect the answer to. Buy the fan.

And it does run with the network off. I switched the radio off, proved there was no route out, and ran one call from each workflow. All three worked. The claim that supports is narrow and worth stating carefully: this
configuration, on this afternoon, ran with no route to the internet. It is not a claim that local models are private or that this setup is secure, and a Pi can be misconfigured as thoroughly as anything in a data center. But for the specific question of whether the inference needs the network, it does not.

So: do you actually need an LLM?

Workload	Verdict
Pull a known-format identifier out of text	Unnecessary
Route a message to one of six queues	Insufficient
Answer a question from your own documents	Marginal

Three workflows that all sound like language tasks, three different answers, and not one of them is "yes, use a model."

And the same line runs through all three. Extraction asks what this string is. Topic labels ask what this message is about. Single-passage lookup asks what the document says. Those went well, or well enough. Escalate asks what we should do, the fee question asks what the customer owes, and the unanswerable questions ask whether we know. Zero of twelve, both models wrong, and two of four.

A small model will tell you what something is. It will not tell you what to do about it. Every result here falls on one side or the other of that sentence, and I did not have it written down when I started.

Which gives four properties worth checking before you reach for one:

The answer is already present in the input. Not inferable from it. Present in it.
There is exactly one of it. The moment two facts have to be combined, this stops working, and the arithmetic in that tasting fee question is the proof.
The output is short. Longer answers are where the latency lives on this hardware, and where the fabrication lives everywhere.
Something other than the model decides whether the output is acceptable. A regex, a schema, a person, an index that either returns rows or does not.

Take away any one of those four and the experiment says do not do it. The 46 percent and the zero of twelve are what it looks like when you ask a small model to be the thing that decides.

And if you have all four, you may not need the model either. That is worth checking first, because it was true for one of my three workflows and I did not expect it to be.

Where I was wrong

Twenty-two predictions, committed before I installed anything. Ten right, eleven wrong, one unresolvable.

Two of my seven confident calls broke. I was sure the 360M model would get most of the easy extraction cases, and it got 8 of 22 by failing at format rather than at finding. I was sure it would produce invalid output often, and it did so once in fifty, then emitted the same valid word forty-three times.

But the number that bothers me is not in the confident column.

Confidence	Right	Wrong
Confident	5	2
Leaning	2	7
Coin flip	3	2

My hunches did worse than my coin flips. Two of ten on the calls where I had a feeling but not much behind it, three of five on the ones I explicitly labeled as guesses.

That ordering is backwards and it is the most useful thing on the sheet. On the coin flips I knew I was guessing, so I guessed cleanly. On the leaning calls I was pattern-matching against intuitions about model behavior that turned out not to describe these models at all, and the feeling of having a reason was the problem. If I had marked all twelve of those as coin flips I would have scored better.

I will take that over a clean result. The honest version of this exercise is not "here are the numbers," it is "here is what I believed, here is what happened, and here is the gap."

The gap, this time, was mostly in the hunches.

The harness that produced these numbers had five measurement flaws in it, none of which manufactured an entire finding out of a missing directory and nearly got published. That is a separate post.

Everything here is reproducible: fixtures, runner, scorer, the committed predictions and the run record are at
github.com/kenwalger/right-size-the-model.
Fixture set v1. Read LIMITATIONS.md before citing any of these numbers:
two models, one board, one quantization, ninety-five cases. This is an illustration of a method, not a benchmark.

FramePort: HDMI Capture Directly Inside VS Code

Thu, 24 Sep 2026 11:47:06 +0000

Hi everyone!

I'm a college junior, and I built FramePort while working on Tomato, my custom 32-bit computer with a dual-LUT ALU running on an Artix-7 FPGA.

I wanted to see its HDMI output beside my assembly code without constantly switching to OBS. So I built a VS Code extension that brings live video capture directly into the editor.

FramePort supports HDMI capture cards, cameras, and other compatible video sources, with features including:

Live video preview with adjustable scaling
Screenshot capture
Silent MP4 recording

Although I originally built it for Tomato, I figured it could also be useful for Raspberry Pi projects, embedded systems, and other hardware-development workflows.

Here's FramePort running alongside Tomato's assembly code:

FramePort is now available on Open VSX as a public preview for macOS. It requires FFmpeg and a compatible capture device.

Try FramePort: https://open-vsx.org/extension/tmarhguy/frameport

Source code: https://github.com/tmarhguy/frameport

I'd love feedback from anyone interested in testing FramePort or contributing to its development!

Raspberry Pi Server: Fortifying Your Digital Fortress with Safeline WAF Protection

Fri, 03 Apr 2026 09:43:40 +0000

Have you ever wondered if that little credit-card-sized computer, the Raspberry Pi, could be a robust server for your home or small business? The answer is a resounding yes! These versatile devices are incredibly capable of handling everything from media streaming and home automation to hosting websites and running complex applications. But as with any server, security is paramount. When you're opening your Raspberry Pi server up to the internet, you're also opening it up to potential threats. That’s where a Web Application Firewall (WAF) comes into play, and today, we’re diving deep into how you can leverage Safeline WAF protection on your Raspberry Pi to build a truly secure digital fortress.

Think of your web server like a castle. It has walls, a gate, and guards. A WAF is like a highly intelligent, constantly vigilant security system that sits in front of your castle's main gate. It doesn't just check if someone has a key; it scrutinizes everything they're trying to bring in, looking for anything suspicious, dangerous, or out of the ordinary. It’s your first line of defense against a whole host of nasty online attacks.

Why a Raspberry Pi Server Needs WAF Protection

Let's be honest, the Raspberry Pi is an absolute marvel of technology. Its affordability, low power consumption, and flexibility have made it a darling of hobbyists and professionals alike. You can set up a Raspberry Pi as a file server, a VPN gateway, a personal cloud, or even a web server for your personal blog or a small business website. But here’s the catch: once your Raspberry Pi server is accessible from the public internet, it becomes a target.

Hackers are constantly scanning the internet for vulnerable servers to exploit. They’re looking for ways to steal your data, inject malicious code, disrupt your services, or even use your server as a launchpad for their own attacks. Common threats include:

SQL Injection: Attackers try to trick your database into revealing sensitive information by inserting malicious SQL code into input fields.
Cross-Site Scripting (XSS): This involves injecting malicious scripts into websites viewed by other users, often to steal session cookies or redirect users to fake sites.
Malicious File Uploads: Attackers might try to upload harmful scripts or executables to your server.
Denial-of-Service (DoS) Attacks: These aim to overwhelm your server with traffic, making it unavailable to legitimate users.
Bot Traffic: Automated bots can crawl your site, looking for vulnerabilities, scraping content, or attempting brute-force logins.

While basic server security measures like strong passwords, regular updates, and firewalls (like ufw or iptables) are essential, they often don't go deep enough to inspect the content of the traffic hitting your web applications. This is precisely where a WAF shines. A WAF analyzes the HTTP/S traffic flowing to and from your web server, identifying and blocking malicious requests before they can even reach your applications.

Introducing Safeline WAF: Your Raspberry Pi's Digital Bodyguard

Safeline WAF is a light but powerful, open-source Web Application Firewall designed to protect web applications from a wide range of threats. It’s built on top of the ModSecurity engine, a well-established and highly respected WAF technology. What makes Safeline compelling for Raspberry Pi users is its relative ease of setup and its comprehensive rule sets, which are regularly updated to combat emerging threats.

Think of ModSecurity as the engine, and Safeline as the polished, user-friendly interface and curated set of rules that make it accessible and effective. It works by inspecting HTTP requests and responses, comparing them against a set of predefined rules (often called the OWASP Core Rule Set – more on that later!). If a request matches a rule indicating malicious intent, Safeline will block it, log the event, and prevent it from reaching your web application.

The Power of ModSecurity and the OWASP Core Rule Set

ModSecurity is the industry-standard open-source WAF module that can be deployed on web servers like Apache, Nginx, and IIS. It acts as a plug-in, intercepting traffic. The real magic of ModSecurity (and by extension, Safeline) lies in its rule engine and the sophisticated rule sets that can be loaded into it.

The Open Web Application Security Project (OWASP) is a renowned non-profit foundation that works to improve software security. Their OWASP Core Rule Set (CRS) is a set of generic attack detection rules for use with ModSecurity or compatible WAFs. It's community-driven, constantly evolving, and widely considered the gold standard for WAF rules. By using Safeline, which leverages ModSecurity and often integrates with or is inspired by the OWASP CRS, you’re tapping into a vast pool of collective security intelligence.

The OWASP CRS is designed to protect against many common web application vulnerabilities, including those listed in the OWASP Top 10. This list is a critical resource for understanding the most significant security risks to web applications. Having a WAF that effectively implements these rules is a massive step towards securing your Raspberry Pi server.

Setting Up Safeline WAF on Your Raspberry Pi: A Step-by-Step Journey

Alright, let's get down to business. Setting up Safeline WAF on your Raspberry Pi involves a few key steps. We'll assume you have a working Raspberry Pi with a recent version of Raspberry Pi OS (formerly Raspbian) installed, and you’re comfortable using the command line. We'll also assume you already have a web server (like Nginx or Apache) installed and serving your website or application.

Prerequisites:

A functioning Raspberry Pi server: With SSH access enabled or direct console access.
A web server: Apache (apache2) or Nginx (nginx) installed and configured.
Internet connectivity: For downloading packages and updates.
Basic Linux command-line knowledge.

Step 1: Install ModSecurity

Safeline WAF is essentially a set of configurations and rules for ModSecurity. So, the first step is to install ModSecurity itself. The installation process varies slightly depending on whether you're using Apache or Nginx.

For Apache:

sudo apt update
sudo apt upgrade -y
sudo apt install libapache2-mod-security2 -y

After installation, ModSecurity is usually enabled automatically for Apache. You can check its status with:

sudo a2enmod security2
sudo systemctl restart apache2

For Nginx:
Installing ModSecurity for Nginx is a bit more involved, often requiring you to compile Nginx with the ModSecurity module or install a pre-compiled version. A common approach is to use the libnginx-mod-http-modsecurity package.
```
sudo apt update
sudo apt upgrade -y
sudo apt install libnginx-mod-http-modsecurity -y
```
Then, you need to enable the module in your Nginx configuration. Edit your main Nginx configuration file (often /etc/nginx/nginx.conf or a file in /etc/nginx/conf.d/) and add the following line within the http block:
```
load_module modules/ngx_http_modsecurity_module.so;
```
After that, you need to configure Nginx to use ModSecurity for specific locations or servers. This is typically done within your server block configuration (e.g., /etc/nginx/sites-available/your_site):
```
server {
    # ... your server configuration ...

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf; # We'll create this later

    location / {
        # ... your location configuration ...
    }
}
```
Finally, restart Nginx:
```
sudo systemctl restart nginx
```

Step 2: Obtain and Configure Safeline WAF Rules

Now that ModSecurity is installed, we need to give it the "brain" – the Safeline WAF rules. Safeline provides a convenient way to manage these rules.

Download Safeline: You can typically clone the Safeline repository from GitHub.
```
cd ~
git clone https://github.com/safeline/safeline.git
sudo mv safeline /etc/safeline
```
Note: The exact GitHub repository and commands might change. Always refer to the official Safeline documentation or GitHub page for the most up-to-date instructions.
Install Dependencies (if any): Safeline might have specific dependencies. Check its documentation.
Configure ModSecurity to Use Safeline: This is the crucial step where you tell ModSecurity where to find and how to use the Safeline rules.
- For Apache:
  Safeline usually provides an apache2 subdirectory with configuration files. You'll need to copy these and potentially link them. A common approach is to copy the main configuration file and the rules directory.
```
sudo cp /etc/safeline/apache2/safeline.conf /etc/apache2/conf-available/
sudo a2enconf safeline.conf
sudo systemctl restart apache2
```
  You might also need to ensure the SecRuleEngine is set to On in /etc/modsecurity/modsecurity.conf (or a similar file). Safeline's setup script often handles this.
- For Nginx:
  As shown in the Nginx setup above, you need a modsecurity_rules_file directive. Safeline typically provides a main.conf file that includes all other necessary rule files.
```
# Ensure the directory for ModSecurity rules exists
sudo mkdir -p /etc/nginx/modsec
# Copy Safeline's main configuration and rule files
sudo cp /etc/safeline/nginx/main.conf /etc/nginx/modsec/
sudo cp -r /etc/safeline/rules/* /etc/nginx/modsec/rules/ # Or similar path
```
  Make sure the modsecurity_rules_file path in your Nginx server block points to the correct main.conf file you just copied.
```
server {
    # ...
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;
    # ...
}
```
  Restart Nginx:
```
sudo systemctl restart nginx
```

Step 3: Configure ModSecurity's modsecurity.conf

You'll need to edit the main ModSecurity configuration file. This file is usually located at /etc/modsecurity/modsecurity.conf or /etc/nginx/modsec/modsecurity.conf (for Nginx).

Key settings to check and adjust:

SecRuleEngine: This must be set to On to enable ModSecurity. You might initially set it to DetectionOnly to monitor traffic without blocking, allowing you to tune the rules.
```
SecRuleEngine On
# or
SecRuleEngine DetectionOnly
```
SecRequestBodyAccess: Set to On to allow ModSecurity to inspect the request body.
```
SecRequestBodyAccess On
```
SecResponseBodyAccess: Set to On to allow ModSecurity to inspect the response body (useful for preventing data leakage).
```
SecResponseBodyAccess On
```
SecAuditLog: Define the path for the audit log. This is crucial for troubleshooting and monitoring.
```
SecAuditLog /var/log/modsec_audit.log
```
SecAuditEngine: Controls what gets logged. RelevantOnly is often a good balance.
```
SecAuditEngine RelevantOnly
```
SecAuditLogParts: Specify which parts of the transaction to log.

Step 4: Enable the OWASP Core Rule Set (CRS)

Safeline often bundles or guides you on integrating the OWASP CRS. If it doesn't, you'll need to download and configure it separately.

Download OWASP CRS:

cd /etc/nginx/modsec/ # Or your ModSecurity rules directory
sudo git clone https://github.com/coreruleset/coreruleset.git
sudo mv coreruleset owasp-crs

Configure CRS:
Navigate into the owasp-crs directory and copy the example configuration file.
```
cd owasp-crs
sudo cp crs-setup.conf.example crs-setup.conf
```
Now, edit crs-setup.conf. You'll find many directives here that allow you to customize the CRS behavior, such as paranoia level, anomaly scoring thresholds, and enabling/disabling specific rule categories. For a start, you might not need to change much, but understanding these settings is key for fine-tuning.
Link CRS to ModSecurity:
You need to ensure ModSecurity loads the CRS rules. This is usually done within the main.conf file you pointed ModSecurity to earlier. You'll add lines to include the CRS setup and rules. This often looks something like this within /etc/nginx/modsec/main.conf (or your equivalent):
```
# Include the CRS setup file
Include owasp-crs/crs-setup.conf

# Include all CRS rules
Include owasp-crs/rules/*.conf
```
Refer to Safeline's documentation and the OWASP CRS documentation for the precise inclusion method, as it can vary.

Step 5: Testing and Tuning

This is arguably the most important phase. Simply turning on a WAF with default settings can often lead to legitimate traffic being blocked (a "false positive").

Start in DetectionOnly Mode: As mentioned, set SecRuleEngine DetectionOnly in your modsecurity.conf. Restart your web server.
Generate Test Traffic: Browse your website normally. Try performing common actions: logging in, submitting forms, searching.
Monitor Logs: Check the ModSecurity audit log (/var/log/modsec_audit.log or wherever you specified). Look for entries indicating potential threats. The logs will tell you which rule was triggered and why.
Identify False Positives: If you see legitimate actions being flagged, you'll need to tune the rules. This might involve:
- Disabling specific rules: If a rule consistently blocks legitimate traffic, you can disable it using SecRuleRemoveById. For example: SecRuleRemoveById 941100 (This is just an example rule ID).
- Adjusting anomaly scoring: The CRS uses a scoring system. You can adjust the thresholds at which a transaction is considered malicious.
- Using SecMarker and SecAction: For more advanced tuning, you can create custom rules or modify existing ones.
Gradually Enable Blocking: Once you're confident that false positives are minimal, switch SecRuleEngine to On. Continue monitoring logs closely.

Step 6: Securing the Logs

Ensure your audit logs are protected. They contain sensitive information about attacks.

Permissions: Set strict file permissions so only root and the webserver user can read them.
```
sudo chown root:www-data /var/log/modsec_audit.log
sudo chmod 640 /var/log/modsec_audit.log
```
(Adjust user/group based on your web server configuration).
Log Rotation: Set up logrotate to manage the size of your audit log file, preventing it from filling up your disk. Create a configuration file in /etc/logrotate.d/.

Advanced Considerations and Best Practices

Regular Updates: Both ModSecurity and the OWASP CRS are under active development. Regularly update them to benefit from new security rules and bug fixes.

cd /etc/safeline # Or your Safeline directory
git pull
# Re-run any installation/configuration scripts if necessary
cd /etc/nginx/modsec/owasp-crs # Or your CRS directory
git pull
# Re-copy crs-setup.conf if it was modified

Remember to restart your web server after updates.

Performance Impact: WAFs, especially those inspecting request and response bodies, can introduce some overhead. On a low-power device like a Raspberry Pi, this is something to be mindful of.
- Start with essential rules: Don't enable every single rule if you don't need it.
- Optimize configurations: Tune SecRequestBodyLimit, SecRequestBodyNoFilesLimit, and other settings to match your application's needs.
- Monitor CPU/RAM usage: Keep an eye on your Pi's resources. If performance degrades significantly, you may need to adjust the WAF or consider hardware upgrades.
HTTPS is Non-Negotiable: A WAF inspects HTTP traffic. If your site uses HTTP, the traffic is unencrypted and can be intercepted before it even reaches the WAF. Always use HTTPS (SSL/TLS) for your Raspberry Pi web server. Let's Encrypt provides free SSL certificates, making this easy. Let's Encrypt is a fantastic resource for this.
Keep Your System Updated: This goes without saying, but always keep your Raspberry Pi OS and all installed packages up-to-date.
```
sudo apt update
sudo apt upgrade -y
sudo apt dist-upgrade -y
sudo reboot
```
Beyond the WAF: Remember that a WAF is just one layer of security. You still need:
- Strong passwords and SSH key authentication.
- A properly configured host-based firewall (like ufw).
- Regular backups of your data and configurations.
- Application-level security: Secure coding practices if you're developing your own applications.
- Intrusion detection systems (IDS) like Fail2ban to block brute-force attempts.

For many Raspberry Pi users, a self-hosted solution like Safeline offers a great balance of security and control without additional monthly fees. It’s an excellent way to learn about web security and harden your server effectively.

Troubleshooting Common Issues

Website Not Loading: This is often due to overly aggressive rules or incorrect configuration.
- Check SecRuleEngine is On or DetectionOnly.
- Examine /var/log/modsec_audit.log for blocked requests.
- Temporarily disable ModSecurity (SecRuleEngine Off) to confirm if it's the cause.
- Review recent changes to your rules or crs-setup.conf.
High CPU Usage:
- Too many rules enabled, or complex rules firing frequently.
- SecRequestBodyAccess and SecResponseBodyAccess can be resource-intensive.
- Consider disabling rules not relevant to your application.
Log Files Not Being Written:
- Check file permissions for the log directory and file.
- Ensure the SecAuditLog directive points to a valid, writable path.
- Verify the SecAuditEngine is set correctly.

Conclusion: Empowering Your Raspberry Pi Server

Running a web server on a Raspberry Pi is an incredibly rewarding and practical endeavor. By implementing a Web Application Firewall like Safeline, you’re taking a significant leap forward in securing your digital assets. It’s not just about preventing attacks; it’s about building trust and reliability for your users and ensuring your services remain available.

Installing and Configuring n8n on a Raspberry Pi (Private Home Server)

Mon, 23 Mar 2026 15:01:38 +0000

title: Installing and Configuring n8n on a Raspberry Pi (Private Home Server)
published: true
date: 2026-03-23 14:49:52 UTC
tags: raspberrypi,n8n,ai,aiagent

canonical_url: https://medium.com/@mahinshanazeer/installing-and-configuring-n8n-on-a-raspberry-pi-private-home-server-2ce086e7408b

GitHub - mahinshanazeer/n8n-home-server: Installing and Configuring n8n on a Raspberry Pi (Private Home Server)

Introduction

Automation is becoming a core part of modern workflows, whether for DevOps, personal productivity, or integrations. n8n is a powerful, open-source workflow automation platform that allows you to create custom integrations with full control over your data.

In this guide, we will set up n8n on a Raspberry Pi using Docker, configured for private network access (no domain, no SSL) — ideal for home labs.

Why Self-Host n8n on Raspberry Pi?

Low-cost, always-on device
Full data privacy (no third-party cloud dependency)
Perfect for home lab and DevOps experimentation
Lightweight yet powerful automation engine

Prerequisites

Ensure the following are ready:

Raspberry Pi (Pi 4/5 recommended, 4GB+ RAM)
Linux OS (Ubuntu Server / Raspberry Pi OS Lite)
Docker & Docker Compose installed
Static private IP (recommended)

Project Structure

We will keep everything inside a single directory:

~/n8n/
├── .env
├── docker-compose.yml
├── n8n-data/
└── local-files/

directory structure

This approach ensures:

Easy backup
Clean portability between systems
Full control over persistent data

Step 1: Create Project Directory

mkdir -p ~/n8n && cd ~/n8n
mkdir n8n-data local-files

Step 2: Create Environment Configuration

Create a .env file:

nano .env

Add the following:

N8N_HOST=192.168.1.200
N8N_PORT=5678
N8N_PROTOCOL=http

WEBHOOK_URL=http://192.168.1.200:5678/

GENERIC_TIMEZONE=Asia/Kolkata

N8N_SECURE_COOKIE=false

Key Notes:

N8N_HOST → Your Raspberry Pi private IP
WEBHOOK_URL → Required for workflows
N8N_SECURE_COOKIE=false → Mandatory for HTTP access

Step 3: Create Docker Compose File

Create docker-compose.yml:

services:
  n8n:
    image: docker.n8n.io/n8nio/n8n
    container_name: n8n
    restart: always
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=${N8N_HOST}
      - N8N_PORT=${N8N_PORT}
      - N8N_PROTOCOL=${N8N_PROTOCOL}
      - N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=${N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS}
      - N8N_SECURE_COOKIE=${N8N_SECURE_COOKIE}
      - NODE_ENV=${NODE_ENV}
      - WEBHOOK_URL=${WEBHOOK_URL}
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
      - TZ=${GENERIC_TIMEZONE}
    volumes:
      # n8n data: SQLite database and encryption key
      - ./n8n-data:/home/node/.n8n
      # Shared files between n8n and host (use /files path inside n8n)
      - ./local-files:/files

Step 4: Start n8n

docker compose up -d

Verify:

docker compose ps

Step 5: Access n8n

Open in browser:

http://192.168.1.200:5678

home page

Create your admin account
Start building workflows (will create another blog on building workflows)

Common Issue (Important)

You may see:

“Your n8n server is configured to use a secure cookie…”

Fix:

Set in .env:

N8N_SECURE_COOKIE=false

Restart:

docker compose down
docker compose up -d

Data Persistence

All important data is stored in:

~/n8n/n8n-data

This includes:

Workflows
Credentials
SQLite database

Github: https://github.com/mahinshanazeer/n8n-home-server

Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide

Sat, 21 Mar 2026 13:53:24 +0000

title: Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide
published: true
date: 2025-10-16 14:35:59 UTC
tags: raspberrypi,kubernetes,etcd,kubernetescluster

canonical_url: https://medium.com/@mahinshanazeer/upgrading-our-raspberry-pi-kubernetes-cluster-from-v1-29-to-v1-34-etcd-backup-guide-97a2074237b4

Keeping your cluster up-to-date is crucial for security, stability, and access to new features. In this blog, we’ll take a closer look at our home lab Kubernetes cluster running on Raspberry Pi devices. The setup includes 1 master node and 3 worker nodes, and it has been running smoothly for over four months without any major issues.

k8s home lab configuration — nodes

master2 — master node/ control plane
master1 — worker node 3
worker 1 — worker node 1
worker 2 — worker node 2

k8s home lab configuration — system components

As part of regular maintenance and to stay up-to-date with the latest features and security patches, we decided to upgrade the cluster from Kubernetes v1.29 to v1.34. Alongside the upgrade, we’ll also walk through the process of backing up the ETCD datastore, which is crucial for fault tolerance and disaster recovery.

Here’s what we’ll cover:

⬆️ Step-by-step guide to upgrading Kubernetes components.
️ Safely backing up ETCD in the master node.
✅ Post-upgrade verification.

A. Backing Up ETCD

Before diving into the upgrade process, it’s essential to create a backup of ETCD, the key-value store that holds all cluster state data. This step is crucial because if anything goes wrong during the upgrade — whether it’s a misconfiguration or a failed component — we need a reliable way to restore the cluster to its current stable state.

Think of the ETCD backup as your safety net. It ensures that even in the worst-case scenario, you won’t lose your cluster’s configuration, workloads, or networking setup.

Run the following command to confirm the ETCD pod name running on your control plane:

kubectl get pods -n kube-system | grep etcd

k8s configuration — ETCD setup

Suppose your control plane (master node) runs ETCD as a static pod (which is the default in kubeadm-based clusters, including Raspberry Pi setups). In that case, you can run etcdctl directly on the master node host — because the ETCD data directory and certificates are mounted locally at /etc/kubernetes/pki/etcd/ and /var/lib/etcd/. In that case, you can use the following command to take the backup:

sudo ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd/backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

If ETCD runs as a container (inside the pod) or you don’t have the etcdctl binary available on the host, then you’ll need to exec into the pod to run the backup from inside the container.

In my setup, where etcd runs as a container, I need to access the pod directly to execute the backup. There is no etcdctl binary available on the host.

kubectl exec -n kube-system <node-label>-- \
  etcdctl snapshot save /var/lib/etcd/backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

#edit the <node-label> with etcd node

Creating backup

Now, verify the snapshot using the following command:

kubectl exec -n kube-system <node-label>-- \
  etcdctl snapshot status /var/lib/etcd/backup.db -w table

In the above screenshot, you can seee a deprecation warning. ETCD v3.6+ is deprecated etcdctl snapshot status and recommends using etcdutl snapshot status. However, in my kubeadm static pod setup, the etcdutl binary is not included in the ETCD container. Only etcdctl exists inside the pod. The warning is simply a deprecation notice, not a failure. You can safely continue using it etcdctl snapshot status — it works perfectly.

In my setup, I came across a deprecation warning while using etcdctl. To verify, I checked the current version of etcd using the following command.

Since my cluster is kubeadm-managed, even though etcd v3.6+ recommends using etcdutl for snapshot management, etcdctl inside the etcd pod still works perfectly for backup and restore. etcdutl is a newer utility introduced in etcd v3.6+ as a replacement for certain snapshot operations, but in my case, continuing with etcdctl is fully sufficient.

kubectl exec -n kube-system etcd-master2 -- etcdctl version

etcd version checking

In some scenarios, the etcd backup file may reside only inside the etcd pod and not in a host-accessible directory.

This can happen if the etcd container is minimal and lacks tools like tar to move files around, or if the host does not have etcdctl installed to create the snapshot directly. In such cases, you can use specific commands to move the backup file to a host-accessible directory.

kubectl cp kube-system/etcd-master2:/tmp/backup.tar.gz ./backup.tar.gz

In my setup, I created the etcd snapshot directly on the host in a host-accessible directory (for example, /var/lib/etcd/backup.db). Since the backup is already on the host, it can be copied or archived without using kubectl cp.

sudo cp /var/lib/etcd/backup.db ~/backup.db
sudo chown $USER:$USER ~/backup.db
ls -l ~/backup.db

backup.db file status

Additionally, it’s a good practice to create a copy of the backup on a remote server. This ensures an extra layer of safety and helps protect against data loss in case of hardware failure or accidental deletion.

B. Upgrading the cluster

Since we have backed up etcd, we can safely proceed with the kubeadm upgrade for our Raspberry Pi Kubernetes cluster. Let’s go through a step-by-step process to upgrade the cluster from v1.29 → v1.34.

Kubernetes officially supports upgrading one minor version at a time, which means that to move from 1.29 to 1.34, you need to perform incremental upgrades:

Step 1: v1.29 → v1.30
Step 2: v1.30 → v1.31
Step 3: v1.31 → v1.32
Step 4: v1.32 → v1.33
Step 5: v1.33 → v1.34

Please refer to the following URL for k8s official documentation on skew policy:

Version Skew Policy

Step 1: Pre-Checks for upgrading the cluster

Before upgrading the cluster, it’s essential to ensure that the cluster is healthy and stable. This helps prevent issues during the upgrade and ensures a smooth transition between versions.

kubectl version
kubectl get nodes
kubectl get cs
kubectl get pods -n kube-system

Current status of the cluster

Step 2: Upgrade kubeadm on the Master Node

You can also refer to the following documentation for more details regarding kubeadm upgrade:

Upgrading kubeadm clusters

To upgrade to v1.30, ensure that the directory /etc/apt/keyrings exists. If it doesn’t, create it before running the curl command (see the note below). Afterwards, update the package repository to target v1.30.

Each Kubernetes minor release repository (v1.29, v1.30, v1.31, etc.) may use a different GPG key. Using the key from an older version, such as v1.29, can cause package verification failures when installing packages from the v1.30 repository.

To prevent such errors, always use the GPG key and repository provided specifically for the minor version you are upgrading to.

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Before upgradingkubeadmEnsure your package lists are up to date. Then, install the required version by using the following commands

sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

If you want to upgrade to a specific Kubernetes version, you can use the following command to list all available minor versions in the corresponding repository:

apt-cache madison kubeadm

available packages in the repo

For comprehensive guidance on upgrading your Kubernetes cluster to version v1.30 , refer to the official Kubernetes documentation:

Installing kubeadm

This resource provides detailed, step-by-step instructions tailored for clusters managed with kubeadm , ensuring a smooth and efficient upgrade process.

Once the upgrade is complete, verify the kubeadm version using the following command:

kubeadm version
kubectl version --client
kubelet --version

current kubeadm version

Step 3: Plan the Upgrade

After upgrading kubeadm, it’s important to check the upgrade path and verify the versions of all cluster components. This ensures that all parts of your Kubernetes cluster are compatible and running the expected versions.

sudo kubeadm upgrade plan

It will display:

The current cluster version and control plane version.
The available target versions you can upgrade to.
The versions of kubelet and kubectl on your nodes.
Any required steps or notes for a smooth upgrade.

kubeadm upgrade plan output — 1

kubeadm upgrade plan output — 2

Step 4: Upgrade the Control Plane

Once the upgrade plan is reviewed and all changes are understood, you can proceed with upgrading the cluster. Use the kubeadm upgrade apply command, which is provided in the output of kubeadm upgrade plan.

sudo kubeadm upgrade apply v1.30.14

The upgrade process may take some time, depending on the size and complexity of your cluster. Once it completes successfully, you should see a message similar to the following:

upgraded to “v1.30.14”

Now, verify the control plane using the following commands:

#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

Step 5: Upgrade Worker Nodes

Now repeat the same steps from ‘Step 2: Upgrade kubeadm on the Master Node’ again in master1, worker1 and worker2 nodes (worker nodes). Workers do not control the cluster state; running upgrade plan there doesn’t provide meaningful information. First drain the worker node from the control plane, so that all the resources will be taken out of the node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
***

draining resources from worker node (master1)

Now you can repeat the steps we followed for control plane for upgrading the version:

ssh to master1 node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

#step 7:
kubeadm upgrade node

upgrading worker node — host ‘master2’ is a worker node here. (master2 — master node/ control planemaster1 — worker node 3worker 1 — worker node 1worker 2 — worker node 2)

Also verify the nodes from control plane:

upgrade completed

Now we can following commmand to make the node available for schedule;

kubectl uncordon master1
#run this command in the control plane.

Marking node as schedulable

kubectl uncordon command is used to mark a node as schedulable again after it has been drained.

Similarly we can do the upgrades in other 2 worker nodes, I will mention the commands below:

worker1:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
***

ssh to worker1 node

#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet

#step 7:
sudo kubeadm upgrade node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl uncordon worker1
kubectl get nodes
kubectl get pods -n kube-system

worker2:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

***
kubectl drain worker2 --ignore-daemonsets --delete-emptydir-data
***

ssh to worker2 node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
#step 3:
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 4:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm=1.30.14-1.1 kubelet=1.30.14-1.1 kubectl=1.30.14-1.1
sudo apt-mark hold kubelet kubeadm kubectl

#step 5:
kubeadm version

#step 6:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

#step 7:
sudo kubeadm upgrade node

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl uncordon worker2

All nodes have now been successfully upgraded to v1.30.14 , as shown in the following screenshot.

kubernetes version v1.30.14

Step 5: Upgrading to 1.34

Following the same process, we’ll now proceed to upgrade the cluster from v1.30.14 to v1.31. Then we will proceed with with

Step 1: v1.29.15 → v1.30.14–1.1
Step 2: v1.30.14–1.1 → v1.31.13–1.1
Step 3: v1.31.13–1.1 → v1.32.9
Step 4: v1.32.9 → v1.33.5
Step 5: v1.33.5 → v1.34.1

In control-plane (master2) use the following steps to upgrade to version 1.31:

ssh to controlplane (master2)

#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#step 2:
sudo apt update
apt-cache madison kubeadm

#step 3:
sudo apt-mark unhold kubelet kubeadm kubectl && \
sudo apt-get update
sudo apt-get install -y kubeadm kubelet kubectl #will take the latest version
sudo apt-mark hold kubelet kubeadm kubectl

#step 4:
kubeadm version
kubectl version --client
kubelet --version

#step 5:
sudo kubeadm upgrade plan

#step 6:
sudo kubeadm upgrade apply <version>
#I used the following command
sudo kubeadm upgrade apply v1.31.13

#step 7:
#rebooting the master node or restarting the kubelete is highly recommended
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get nodes
kubectl get pods -n kube-system

Now lets setup the worker nodes:

ssh to controlplane (master2)
-------------------------------
#step 1:
kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

ssh to <worker-node> node
-------------------------------
#step 1:
sudo rm -rf /etc/apt/sources.list.d/kubernetes.list

#step 2:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-key

Raspberry Pi K8S Cluster Setup for Home Lab with Cilium

Tue, 17 Mar 2026 11:01:15 +0000

title: Raspberry Pi K8S Cluster Setup for Home Lab with Cilium
published: true
date: 2025-06-06 16:13:49 UTC
tags: kubernetes,raspberrypi,docker,kubernetescluster

canonical_url: https://medium.com/@mahinshanazeer/raspberry-pi-k8s-cluster-setup-for-home-lab-with-cilium-c861f7815511

I’m setting up a Kubernetes cluster using Raspberry Pis for home lab testing and learning purposes. The cluster consists of four nodes:

Master 1: Raspberry Pi 5 (8GB RAM) 192.168.1.200
Master 2: Raspberry Pi 5 (4GB RAM) 192.168.1.201
Worker 1: Raspberry Pi 3B+ 192.168.1.202
Worker 2: Raspberry Pi 3B 192.168.1.203

I’ve overclocked the Raspberry Pis to enhance performance. Below are the detailed specifications and overclocking configurations.

root@master1:/home/admin# neofetch
       _,met\(\)$gg. root@master1 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 5 Model B Rev 1.0 
',\)P ,ggs. `\(b: Kernel: 6.12.25+rpt-rpi-2712 
`d\)' ,$P"' . \($ Uptime: 2 hours 
 \)P d$' , \(P Packages: 1099 (dpkg), 2 (snap) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' CPU: (4) @ 2.700GHz 
 Y\). `.`"Y\(\)P"' Memory: 360MiB / 8063MiB 
 `\(b "-.__
  `Y\)                                                
   `Y\(.                                              
     `\)b.
       `Y$$b.
          `"Y$b._
              `"""

root@master1:/home/admin#

root@master2:/home/admin# neofetch
       _,met\(\)$gg. root@master2 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 5 Model B Rev 1.0 
',\)P ,ggs. `\(b: Kernel: 6.12.25+rpt-rpi-2712 
`d\)' ,$P"' . \($ Uptime: 1 hour, 6 mins 
 \)P d$' , \(P Packages: 783 (dpkg) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' CPU: (4) @ 2.700GHz 
 Y\). `.`"Y\(\)P"' Memory: 234MiB / 4050MiB 
 `\(b "-.__
  `Y\)                                                
   `Y\(.                                              
     `\)b.
       `Y$$b.
          `"Y$b._
              `"""

root@master2:/home/admin#  

admin@worker1:~ $ sudo su
root@worker1:/home/admin# neofetch
       _,met\(\)$gg. root@worker1 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 3 Model B Plus Rev 1.3 
',\)P ,ggs. `\(b: Kernel: 6.6.74+rpt-rpi-v8 
`d\)' ,$P"' . \($ Uptime: 1 min 
 \)P d$' , \(P Packages: 751 (dpkg) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' Resolution: 1024x768 
 Y\). `.`"Y\(\)P"' CPU: (4) @ 1.400GHz 
 `\(b "-.__ Memory: 166MiB / 907MiB 
  `Y\)
   `Y\(.                                              
     `\)b.                                            
       `Y$$b.  
          `"Y$b._
              `"""

root@worker1:/home/admin# 

root@worker2:/home/admin# neofetch
       _,met\(\)$gg. root@worker2 
    ,g\(\)\(\)\(\)\($P. ------------ 
  ,g\)P" """Y\(.". OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,\)P' `\($. Host: Raspberry Pi 3 Model B Rev 1.2 
',\)P ,ggs. `\(b: Kernel: 6.12.25+rpt-rpi-v8 
`d\)' ,$P"' . \($ Uptime: 1 hour, 6 mins 
 \)P d$' , \(P Packages: 743 (dpkg) 
 \): \(. - ,d\)' Shell: bash 5.2.15 
 \(; Y$b._ _,d$P' CPU: (4) @ 1.400GHz 
 Y\). `.`"Y\(\)P"' Memory: 116MiB / 906MiB 
 `\(b "-.__
  `Y\)                                                
   `Y\(.                                              
     `\)b.
       `Y$$b.
          `"Y$b._
              `"""

root@worker2:/home/admin#te

First, we’ll configure m aster2 (Raspberry Pi 5–4GB) as the initial control plane node , while master1 , worker1 , and worker2 will function as worker nodes.

This setup is intentional — it allows us to explore advanced scenarios in future blog posts, such as:

Setting up a new cluster using kubeadm with Cilium.
Handling control plane degradation and recovery

Starting simple provides a solid foundation while leaving room to scale and experiment.

Preparation in each node:

The following commands prepare the OS for installing external components like container runtimes (Docker/containerd) and Kubernetes tools (kubeadm, kubelet, kubectl).

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl apt-transport-https ca-certificates software-properties-common

Enabling cgroup for CPU and memory

Hence, we are using Raspberry Pi, we can simply edit the cmdline.txt file in the firmware location to enable the cGroups.

sudo nano /boot/firmware/cmdline.txt

#add the following content
ipv6.disable=1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Now disable swap in all nodes.

Disabling swap on Kubernetes nodes is a common practice due to its potential to negatively impact performance and predictability, especially for containerized workloads. Swap, while it allows a system to use more memory than available RAM, accesses data from the disk, which is significantly slower than RAM. This can lead to performance degradation, increased latency, and unpredictable behaviour, particularly under load.

Disabling swap is mandatory because Kubernetes requires it for stable and predictable performance:

Kubernetes relies on the kubelet to manage resources, assuming memory is fully available.
Swap can cause nodes to become slow or unresponsive as critical processes get swapped out.
Disabling swap ensures the scheduler and kubelet have accurate memory availability data.
Kubernetes officially fails to start if swap is enabled unless explicitly configured otherwise (which is not recommended).

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo systemctl disable dphys-swapfile

#or

sudo swapoff -a
sudo apt purge -y dphys-swapfile

#verify using 
free -mh

Enable required kernel modules:

Enabling overlay and br_netfilter kernel modules is essential for Kubernetes networking:

overlay: Supports overlay network drivers like Calico or Flannel, enabling pods across different nodes to communicate through virtual networks.

br_netfilter: Allows Linux bridge firewall rules to inspect and filter bridged network traffic, which is crucial for Kubernetes network policies and packet forwarding.

sudo modprobe overlay
sudo modprobe br_netfilter

Adding overlay and br_netfilter to /etc/modules-load.d/k8s.conf

Adding overlay and br_netfilter to /etc/modules-load.d/k8s.conf ensures these kernel modules:

Load automatically at boot time
Maintain persistent availability without manual intervention after reboot

This guarantees Kubernetes networking components relying on these modules are always operational, avoiding network failures after node restarts.

vi /etc/modules-load.d/k8s.conf

overlay
br_netfilter

k8s.conf

Configure sysctl:

The sysctl settings in /etc/sysctl.d/k8s.conf enable essential kernel networking features for Kubernetes:

net.bridge.bridge-nf-call-ip6tables = 1

Ensures IPv6 bridged network traffic is passed through ip6tables for filtering and firewall rules.

net.bridge.bridge-nf-call-iptables = 1

Ensures IPv4 bridged network traffic is processed by iptables, allowing Kubernetes to enforce network policies and firewall rules on pod traffic.

_net.ipv4.ip_forward = 1_

Enables IP forwarding at the kernel level, allowing packets to be routed between different network interfaces — critical for pod-to-pod communication across nodes.

These settings are required for Kubernetes networking to function properly and to enforce network security policies.

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl - system

sysctl.d/k8s.confg

2. Hostname and IP configuration in the router.

To ensure network stability within the Kubernetes cluster, it’s essential to assign static IP addresses to all nodes. In my setup, I have four Raspberry Pi devices connected to my home router via Wi-Fi.

While it’s widely recommended to use wired (RJ45) Ethernet connections for Kubernetes clusters, mainly for improved reliability and lower latency, I’ve opted to use Wi-Fi. Despite the general concerns, I’ve tested this configuration extensively under various conditions and, so far, have encountered no issues.

To maintain consistent IP addressing, I reserved the IPs for each Raspberry Pi in the router’s DHCP settings. This ensures the cluster nodes remain discoverable and reachable, even after reboots or network interruptions.

I am using a Netlink Router at home. Please find my current configuration in the following screenshot. If you are using similar models, you can refer to the following configuration:

Once this is configured, we have to edit the host file in each node and add the following content:

vi /etc/hosts

#add the following
192.168.1.200 master1
192.168.1.201 master2
192.168.1.202 worker1
192.168.1.203 worker2

3. Installing Container Runtime.

In Kubernetes, the smallest deployable unit is the Pod, which usually runs one or more containers. To run these containers, Kubernetes relies on a container runtime.

Initially, Kubernetes was designed to work closely with Docker, which was the most widely adopted container engine at the time. However, over time, Kubernetes evolved, and the need to support multiple container runtimes became clear.

To address this, the Open Container Initiative (OCI) was introduced — a set of industry standards that define how containers should be built and executed. In line with OCI, Kubernetes implemented the Container Runtime Interface (CRI), which allows it to interact with any OCI-compliant runtime, such as containerd or CRI-O, in a standardised way.

Today, Docker is no longer a single, unified project. It consists of several components, including Docker-CLI, Docker-Compose, Docker Network, runc, and containerd. While containerd complies with the OCI standard and is capable of running containers, it does not natively support the CRI that Kubernetes requires.

To bridge this gap, Kubernetes initially provided a compatibility layer called Docker shim, allowing it to communicate with the Docker Engine. However, Dockershim has now been deprecated, and its role is continued by the community-maintained project CRI-Dockerd.

In this project, we are using the Docker Engine as the container runtime for Kubernetes, integrated via CRI-Dockerd.

Let’s move on and see how to install Docker and configure it as a container runtime for Kubernetes using CRI-Dockerd. Currently, Mirantis is managing the CRI-Dockerd project, and I followed their official documentation for configuring CRI-Dockerd on my Raspberry Pi. Please note that we have to configure the container runtime in all nodes for setting up the cluster successfully.

Installing Docker on all nodes

Debian

We’ll begin by installing Docker on all machines in the cluster. If you’re using a Debian-based system (such as Ubuntu or Raspberry Pi OS), you can use the following commands regardless of the system architecture:

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Once Docker installation is completed, run the following command to verify the installation is proper and working fine.

docker run hello-world

The following is the expected output;

docker run hello-world

Installing CRI-Dockerd on all nodes

You can refer to the following documentation for more details regarding CRI-Dockerd configuration:

Install

In this setup, I’m using Raspberry Pi devices, which are based on the ARM64 architecture. Instead of using the default package manager, I’m downloading the Docker binaries directly from the official release page to ensure compatibility and control over the installation version.

Releases · Mirantis/cri-dockerd

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.17/cri-dockerd-0.3.17.arm64.tgz

#after downloading extract and move it to local binary

tar -xvf cri-dockerd-0.3.17.arm64.tgz
cd cri-dockerd
sudo mv cri-dockerd /usr/local/bin/

Once the installation is completed, run the following command to verify the installation;

sudo systemctl status cri-docker.service

If the configuration is correct, running the systemctl command should return an output indicating that the service is active and running.

node1

node2 (master)

node 3

node 4

3. Installing Kubeadm

Installing kubeadm

In this setup, we will be using Kubeadm version 1.29. This specific version is chosen intentionally, as we are planning a follow-up blog focused on Kubeadm upgrade procedures. Starting with v1.29 provides a practical baseline for demonstrating version upgrades.

If you prefer to install the latest available version, you can refer to the official Kubernetes documentation linked above. Otherwise, to proceed with Kubeadm v1.29, follow the version-specific installation steps documented here.

Installing kubeadm

sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

#if you are facing any issues with the installation and you want to reinstall the packages, please unhold them first.

sudo apt-mark unhold kubelet kubeadm kubectl

kubeadm, kubelet and kubectl installation completed

Follow the installation process on all nodes. Once the installation is completed, enable the kubelet on all nodes:

sudo systemctl enable --now kubelet

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do.

4. Cgroup Driver Configuration

Why Cgroup Driver Configuration Matters in Kubeadm Installation?

During a Kubernetes cluster setup with Kubeadm, it is essential to ensure that the cgroup driver used by the container runtime (e.g., Docker, containerd) matches the driver expected by Kubelet, the Kubernetes node agent.

Cgroups (control groups) are used by the Linux kernel to limit and isolate resource usage (CPU, memory, etc.) of process groups. Kubernetes relies on this mechanism to enforce resource limits on containers.

There are two common cgroup drivers:

cgroupfs — a native driver provided by the Linux kernel
systemd — integrates with the systemd init system for resource management

Why this matters:

If the cgroup drivers between Docker/containerd and Kubelet are mismatched, Kubelet will fail to start, or you may encounter node stability issues.

Best practice:

Kubernetes now recommends using the systemd cgroup driver for better compatibility and integration with modern Linux distributions.

You can verify and configure the cgroup driver to ensure consistency before initialising the cluster with kubeadm. Refer to the following documentation for more details. In our case, it is not required.

Configuring a cgroup driver

5. Initialising master-node.

Creating a cluster with kubeadm

When you run kubeadm init, Kubernetes performs several critical actions to bootstrap the control plane:

Generates PKI Certificates
Secure communication between components is enabled by generating TLS certificates under /etc/kubernetes/pki.
Bootstrap the Kubeconfig Files.
Configuration files required by kubectl and system components are created under /etc/kubernetes/ and $HOME/.kube/config.
Start Control Plane Components
These pods run on the master node and are defined under /etc/kubernetes/manifests/. The following static pods are created and managed by the kubelet
- kube-apiserver
- kube-controller-manager
- kube-scheduler
Initialises the etcd Database
An embedded or external etcd key-value store is initialised to persist cluster state.
Sets up the Bootstrap Token and Join Configuration
A token and join command are generated to allow worker nodes to securely join the cluster.
Outputs Post-Initialisation Instructions
The command displays steps to:
- Configure kubectl access
- Install a Pod network add-on (required before scheduling workloads)
- Join worker nodes using the kubeadm join command

Now run the following command. After initialising the control plane with kubeadm, you need to configure kubectl to communicate with the cluster. The following commands handle that setup:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

export KUBECONFIG=/etc/kubernetes/admin.conf

The export command sets the environment variable so that kubectl uses the admin configuration directly from the system path. Now run the following command to initialize the master node;

sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock

If the control plane initialisation is successful, you’ll see a message similar to the one shown in the screenshot below. This message confirms that the Kubernetes control plane is up and running.

Additionally, a bootstrap token will be generated. This token, along with the control plane IP and certificate hash, is required for joining worker nodes to the cluster.

You’ll also receive instructions to:

Set up your kubectl config
Install a pod network add-on
Use the kubeadm join command on worker nodes

kubeadm init

Note: You must use the corresponding sockets depending on the container runtime version you are usi

Coder Legion - Recent posts tagged raspberrypi

Nothing Reported It

Five ways my own test harness lied to me, and the one that nearly got published

The experiment, briefly

Kind one: a metric that rewards the safe failure

Kind two: a metric that measures nothing at all

Kind three: a detector with a blind spot in the obvious direction

Kind four: errors that get dropped instead of counted

Kind five: inputs that go missing without raising

The detector that fired, and was right, and told me the wrong thing

What I actually changed

The uncomfortable part

Do You Actually Need an LLM?

Extraction: pulling an order number out of a message

The 360M result needs two numbers, not one

Which suggests a use for it that is not extraction

Classification: routing a message to one of six queues

And then I broke the score down by class

Grounded QA: answering a question from four documents

Retrieval works

Combination does not

Knowing the answer is absent is the weakest link

The one answer the evidence did not change

What it costs

So: do you actually need an LLM?

Where I was wrong

FramePort: HDMI Capture Directly Inside VS Code

Raspberry Pi Server: Fortifying Your Digital Fortress with Safeline WAF Protection

Why a Raspberry Pi Server Needs WAF Protection

Introducing Safeline WAF: Your Raspberry Pi's Digital Bodyguard

The Power of ModSecurity and the OWASP Core Rule Set

Setting Up Safeline WAF on Your Raspberry Pi: A Step-by-Step Journey

Advanced Considerations and Best Practices

Troubleshooting Common Issues

Conclusion: Empowering Your Raspberry Pi Server

Installing and Configuring n8n on a Raspberry Pi (Private Home Server)

canonical_url: https://medium.com/@mahinshanazeer/installing-and-configuring-n8n-on-a-raspberry-pi-private-home-server-2ce086e7408b

Introduction

Why Self-Host n8n on Raspberry Pi?

Prerequisites

Project Structure

Step 1: Create Project Directory

Step 2: Create Environment Configuration

Key Notes:

Step 3: Create Docker Compose File

Step 4: Start n8n

Step 5: Access n8n

Common Issue (Important)

Secure Cookie Error

Fix:

Data Persistence

Upgrading Our Raspberry Pi Kubernetes Cluster: From v1.29 to v1.34 & ETCD Backup Guide

canonical_url: https://medium.com/@mahinshanazeer/upgrading-our-raspberry-pi-kubernetes-cluster-from-v1-29-to-v1-34-etcd-backup-guide-97a2074237b4

A. Backing Up ETCD

B. Upgrading the cluster

Step 1: Pre-Checks for upgrading the cluster

Step 2: Upgrade kubeadm on the Master Node

Step 3: Plan the Upgrade

Step 4: Upgrade the Control Plane

Step 5: Upgrade Worker Nodes

Step 5: Upgrading to 1.34

Raspberry Pi K8S Cluster Setup for Home Lab with Cilium

canonical_url: https://medium.com/@mahinshanazeer/raspberry-pi-k8s-cluster-setup-for-home-lab-with-cilium-c861f7815511

Preparation in each node:

2. Hostname and IP configuration in the router.

3. Installing Container Runtime.

3. Installing Kubeadm

4. Cgroup Driver Configuration

5. Initialising master-node.