Coder Legion - Recent posts tagged computervision

From Lab to Code #1: Exploring Classical Computer Vision with Python and OpenCV

Thu, 22 Oct 2026 12:53:42 +0000

Most conversations about computer vision today revolve around deep learning, neural networks, and increasingly sophisticated AI models.

But not every image processing problem requires a trained model.

Sometimes, a Gaussian filter is enough to reduce noise. A threshold can separate objects from their background. A Fourier transform can reveal interference that is difficult to understand in the spatial domain.

These techniques are relatively simple, mathematically interpretable, and still useful in practical applications.

Recently, while exploring image and signal processing concepts in my computer engineering studies, I decided to start a small programming series called From Lab to Code.

The idea is simple: take engineering concepts I encounter, build independent implementations, visualize the results, and share what I learn.

For the first project, I built a small interactive image processing toolkit using Python, OpenCV, NumPy, and Streamlit.

GitHub repository: https://github.com/Ertugrulmutlu/from-lab-to-code

Project 01 — Image Processing: https://github.com/Ertugrulmutlu/from-lab-to-code/tree/main/01-image-processing

What Are We Building?

The application explores four classical computer vision techniques:

Image denoising using Gaussian and median filters.
Object counting using illumination correction, thresholding, and connected components.
Periodic noise removal using the Fast Fourier Transform (FFT).
Color clustering using K-means.

The application provides interactive controls, generated example images, and visual comparisons.

No pretrained models, API calls, or external datasets are required.

All bundled demonstration images are generated programmatically.

1. Image Denoising: Gaussian vs. Median Filtering

Digital images are often affected by noise.

Noise can originate from camera sensors, poor lighting conditions, signal transmission, or image acquisition hardware.

Two common types are:

Gaussian noise: Random variations in pixel intensity, often modeled using a normal distribution.
Salt-and-pepper noise: Individual pixels unexpectedly taking very dark or very bright values.

Different noise distributions require different filtering strategies.

Gaussian Filtering

A Gaussian filter performs a weighted average of neighboring pixels.

Pixels closer to the center receive higher weights, following a Gaussian distribution.

Mathematically, the two-dimensional Gaussian kernel can be written as:

G(x, y) = (1 / 2πσ²) × exp(-(x² + y²) / 2σ²)

The parameter σ controls the amount of smoothing.

A larger σ produces stronger smoothing but can also remove fine image details.

In OpenCV, Gaussian filtering is straightforward:

import cv2

gaussian = cv2.GaussianBlur(
    image, (5, 5), 0
)

Median Filtering

Unlike Gaussian filtering, a median filter replaces the center pixel with the median value of its neighborhood.

Consider these pixel values:

[10, 11, 12, 13, 255]

Their average is 60.2, but their median is 12.

If the value 255 represents an isolated noise spike, the median is much closer to the surrounding pixel values.

This makes median filtering particularly useful for salt-and-pepper noise.

median = cv2.medianBlur(image, 5)

Comparing the Results

The application applies both filters to the same noisy image.

It also calculates the Mean Squared Error (MSE) against the original image when a reference is available.

MSE = (1/N) × Σ(original - processed)²

A lower MSE indicates a smaller average squared difference from the reference image.

However, MSE alone does not fully describe perceptual image quality.

Key takeaway: Gaussian filtering is useful for smoothing distributed noise, while median filtering is often more effective against isolated intensity spikes.

Neither method is universally superior.

2. Object Counting: From Pixels to Objects

Suppose we want to count small objects placed on a table.

At first, this sounds like a simple thresholding problem.

If the objects are darker than the background, we could classify pixels based on their intensity.

But what happens when the background illumination is uneven?

One side of the image might be noticeably darker than the other, making a single global threshold unreliable.

Step 1: Estimate the Background

We can approximate slowly changing illumination using a large Gaussian blur.

background = cv2.GaussianBlur(
    gray, (0, 0), sigmaX=30
)

Subtracting the original image from the estimated background helps emphasize darker foreground objects.

corrected = cv2.subtract(
    background, gray
)

This reduces the influence of gradual illumination changes.

Step 2: Apply Otsu Thresholding

Otsu's method automatically selects a threshold based on the image's intensity distribution.

Instead of manually choosing a number such as 100 or 150, it searches for a threshold that separates the intensity classes according to their statistical variance.

_, mask = cv2.threshold(
    corrected,
    0,
    255,
    cv2.THRESH_BINARY + cv2.THRESH_OTSU
)

The output is a binary image containing foreground and background pixels.

Step 3: Identify Connected Components

After thresholding, neighboring foreground pixels can be grouped into connected regions.

OpenCV provides a useful function for this:

count, labels, stats, centroids = (
    cv2.connectedComponentsWithStats(
        mask, connectivity=8
    )
)

The function returns component labels and information such as bounding boxes and pixel areas.

By excluding the background and removing regions below a minimum area, we can estimate the number of objects.

The project also applies a small morphological opening operation before counting to remove isolated foreground noise.

Where Can This Be Useful?

This approach can be applied to controlled visual inspection tasks involving separate objects, such as small manufactured components.

However, it comes with important assumptions.

Objects should be sufficiently separated, and the contrast between objects and background should be reasonably clear.

Touching objects may merge into a single connected component.

Key takeaway: Reliable object counting often requires preprocessing, not just thresholding.

3. Fourier Transform: Removing Periodic Noise

This was one of the most interesting parts of the project.

Images are normally represented in the spatial domain, where each pixel corresponds to a position.

But some image artifacts are easier to identify in the frequency domain.

For example, imagine an image affected by regularly repeating stripes.

Those stripes represent periodic variations in pixel intensity.

Moving to the Frequency Domain

The two-dimensional Discrete Fourier Transform decomposes an image into spatial frequency components.

Each component represents a particular frequency and orientation.

Using NumPy:

import numpy as np

spectrum = np.fft.fft2(gray)

shifted = np.fft.fftshift(spectrum)

magnitude = np.log1p(
    np.abs(shifted)
)

Here:

fft2() computes the two-dimensional Fourier transform.
fftshift() moves the zero-frequency component to the center.
abs() obtains the magnitude.
log1p() makes the magnitude spectrum easier to visualize.

Periodic interference may appear as distinct peaks in the Fourier spectrum.

Removing Specific Frequencies

If we know which frequencies correspond to interference, we can suppress them using a notch filter.

A notch filter removes selected frequency regions while preserving much of the remaining spectrum.

For real-valued images, the Fourier spectrum has conjugate symmetry.

This means a sinusoidal interference component generally produces a symmetric pair of frequency peaks.

Both should be handled to preserve the symmetry of the filtered spectrum.

The basic operation is:

filtered_spectrum = shifted * mask

Here, mask contains zeros at the frequencies we want to suppress and ones elsewhere.

We then reconstruct the image using the inverse Fourier transform.

restored = np.fft.ifft2(
    np.fft.ifftshift(filtered_spectrum)
).real

An Important Limitation

In this project, the periodic noise is generated with known frequencies.

Therefore, the notch filter already knows where the unwanted frequency components are located.

This is intentional: it lets us isolate and study the effect of frequency-domain filtering.

Automatically identifying unknown interference frequencies in arbitrary photographs is a separate problem and is not implemented here.

A notch filter can also remove useful image information if the rejected frequencies overlap with actual image content.

Key takeaway: The Fourier transform provides another perspective on image processing, allowing us to manipulate structures according to their spatial frequencies rather than their pixel positions.

4. K-Means: Unsupervised Color Clustering

The final technique explores unsupervised learning.

Unlike classification models, K-means does not require predefined class labels.

Instead, it groups observations based on similarity.

For an RGB image, each pixel can be represented as a three-dimensional vector:

Pixel = [Red, Green, Blue]

K-means attempts to divide these vectors into K clusters by minimizing the squared distances between pixels and their assigned cluster centers.

The objective function is:

J = Σ ||xᵢ - μ(clusterᵢ)||²

where xᵢ is a pixel vector and μ is the corresponding cluster center.

Implementing K-Means with OpenCV

First, reshape the image into a collection of RGB vectors:

pixels = image.reshape(
    -1, 3
).astype(np.float32)

Then run K-means:

criteria = (
    cv2.TERM_CRITERIA_EPS +
    cv2.TERM_CRITERIA_MAX_ITER,
    25,
    0.2
)

_, labels, centers = cv2.kmeans(
    pixels,
    4,
    None,
    criteria,
    5,
    cv2.KMEANS_PP_CENTERS
)

We can reconstruct an image by replacing each pixel with the color of its assigned cluster center.

This produces a simplified image containing a small number of representative colors.

Color Clustering Is Not Semantic Segmentation

This distinction is important.

K-means only sees RGB values in this implementation.

It does not understand objects, scenes, or semantic categories.

Two unrelated objects with similar colors may belong to the same cluster.

Likewise, one object with multiple colors may be split across several clusters.

Still, color clustering is useful for exploring image structure, reducing color complexity, and understanding unsupervised learning.

Running the Project Locally

The project is available on GitHub:

https://github.com/Ertugrulmutlu/from-lab-to-code/tree/main/01-image-processing

You will need Python 3.10 or newer.

Clone the repository:

git clone https://github.com/Ertugrulmutlu/from-lab-to-code.git
cd from-lab-to-code/01-image-processing

Create a virtual environment:

python -m venv .venv

Activate it on Windows PowerShell:

.\.venv\Scripts\Activate.ps1

Install the dependencies:

python -m pip install -r requirements.txt

Start the application:

python -m streamlit run app.py

Streamlit will open an interactive application in your browser.

You can switch between the four techniques, adjust processing parameters, and compare the results.

Each mode includes a programmatically generated example image. You can also upload an image of your own.

To regenerate the demonstration gallery:

python demo.py

To run the included tests:

python -m unittest discover -s tests -v

The processing algorithms are implemented in processing.py, while synthetic.py generates the example images and app.py contains the Streamlit interface.

What I Learned

Building this project reinforced several important ideas.

First, understanding the characteristics of a problem matters as much as selecting an algorithm.

Gaussian filtering and median filtering may look similar at first, but they behave differently depending on the noise distribution.

Second, preprocessing can determine whether a simple computer vision algorithm succeeds or fails.

In the object counting example, background illumination correction can be just as important as the thresholding operation itself.

Third, the frequency domain provides a different way to reason about images.

Some patterns that are difficult to isolate spatially become much easier to analyze after a Fourier transform.

Finally, unsupervised clustering demonstrates that useful structure can be extracted from images without labeled training data, even though grouping colors is not the same as understanding a scene.

Why I Started From Lab to Code

I wanted a lightweight way to turn engineering concepts into practical software projects.

Not every project needs to become a startup, a large framework, or a long-term product.

Sometimes, a small, well-explained implementation is enough to explore an idea and demonstrate what you have learned.

That is the philosophy behind From Lab to Code.

Each entry will focus on an engineering topic, build a small independent implementation, and document the underlying concepts and results.

The projects are independently implemented, with original demonstration materials rather than redistributed university coursework.

This first project focuses on classical image processing. Future entries may explore other engineering topics as I encounter them.

If you are learning computer vision or signal processing, I hope the code and explanations help you experiment with these concepts yourself.

Explore the series: https://github.com/Ertugrulmutlu/from-lab-to-code

Explore the image processing project: https://github.com/Ertugrulmutlu/from-lab-to-code/tree/main/01-image-processing

Thanks for reading!

RIVET — Environmental Incident Investigator

Wed, 21 Oct 2026 09:04:48 +0000

RIVET uses computer vision and water-quality sensors to investigate suspicious changes in waterways, verify evidence, and generate traceable environmental incident reports.

Inspiration

Environmental changes in waterways can go unnoticed when monitoring depends on isolated measurements or occasional inspections. We wanted to explore how computer vision and intelligent investigation could help identify suspicious changes and direct attention toward situations that need further examination.

What it does

RIVET (Environmental Incident Investigator) is a proposed system that combines OpenCV-based video analysis with water-quality sensor readings. It aims to detect unusual changes, compare visual evidence with sensor measurements, investigate uncertainty through additional analysis, and generate traceable incident reports.

How we built it

The planned prototype uses Python and OpenCV to analyse video over time. Water-quality sensors will provide independent measurements, while an AI agent will select follow-up analyses when observations are incomplete or conflicting. AWS will support the data-processing and storage workflow.

Challenges we expect

Important challenges include distinguishing genuine environmental changes from lighting variations, synchronising video with sensor readings, collecting reliable test data, and reducing false alarms.

Accomplishments we're aiming for

Our goal is to build a working prototype that demonstrates evidence-driven environmental investigation, combines computer vision with sensor measurements, and evaluates its performance against simpler monitoring methods.

What we aim to learn

We aim to investigate whether adaptive visual analysis and sensor fusion can improve environmental anomaly detection, how uncertainty should influence an AI agent's next action, and how to make automated environmental monitoring more transparent and reliable.

How to Build a Deep learning model to detect Ships

Mon, 19 Oct 2026 16:30:01 +0000

For anyone studying practical computer vision and deep learning deployment:
A frequent challenge in applied vision projects is balancing spatial accuracy against computational overhead during training. Selecting an architecture variant like YOLOv8l provides the parameter capacity necessary to resolve multi-scale objects in clustered environments, but success depends entirely on proper configuration and validation monitoring. This tutorial breaks down the mechanics of training a custom detector from scratch, detailing how loss convergence curves and validation checkpoints guide the selection of deployable weights.

The progression moves methodically from environment initialization to final model inference. It covers dataset re-indexing, parsing raw coordinate tags into standardized label files, executing supervised training runs while monitoring loss metrics, and implementing an OpenCV inference script to handle bounding box coordinates, class labels, and confidence thresholds on unseen imagery.

The Resources:

Reading on Medium: https://medium.com/object-detection-tutorials/yolov8-custom-object-detection-full-code-walkthrough-23fa476f834e

Detailed written explanation and source code: https://eranfeit.net/yolov8-custom-object-detection-full-code-walkthrough/

Join my Newsletter : https://eranfeit.net/advance-your-skills-in-computer-vision-ai/

Deep-dive video walkthrough: https://youtu.be/zAyPboTjcAA
This material is published solely for educational purposes. Feel free to leave constructive feedback or questions about threshold tuning and batch performance below.

The adaptive threshold that deleted the thing we were looking for

Mon, 28 Sep 2026 05:03:07 +0000

Finding the four corners of a page in a phone photo is the step that turns a picture of a document into a copy of one. Everything downstream — the perspective warp, the flattening, the PDF — is easy once you know where the page is.

We do that client-side, in a browser tab, on a grayscale copy of the photo scaled to a long edge of 640 px. The pipeline is the textbook one:

grayscale → blur → Sobel → adaptive edge map → Hough lines → best convex quad

Two near-horizontal lines, two near-vertical lines, score every candidate quadrilateral, keep the best. Disclosure up front: this is from LensUp, which is ours. What follows is the part where the textbook pipeline quietly failed, and the one-line change that nearly doubled how often it works.

The failure: the page fills the frame and detection gets worse

The adaptive edge threshold is the standard trick. Take the histogram of Sobel magnitudes, pick the 90th percentile, call everything above it an edge. It adapts to lighting, exposure and camera noise for free.

It also has a property nobody mentions: it is set by whatever has the strongest gradients in the image. On a photo of a document, that is not the page border. It is the letter strokes. Printed text is black-on-white at high spatial frequency — it produces the sharpest gradients in the frame by a wide margin.

The paper-to-desk step, meanwhile, is weak. Measured on SmartDoc 2015, a page lying on light wood clears the surrounding surface by roughly 14 luma levels. That is the signal you actually need, and it is an order of magnitude quieter than the text sitting in the middle of it.

So the adaptive threshold, doing exactly what it was designed to do, sets itself by the text and prices the page border out of the edge map. And it does this worst in the case you most want to work: when the user fills the frame with the page, so text dominates the histogram and there is barely any desk left to argue for a lower threshold.

The fix is not clever. Cap it:

edgePercentile: 0.9,     // gradient magnitudes above this percentile are edges …
minEdgeMagnitude: 14,    // … but never below this …
maxEdgeThreshold: 30,    // … and never above this: print must not price the page edge out

Measured on 300 real frames, with every other guard unchanged:

threshold cap	pages located
220	23.3 %
30	41.0 %

An adaptive threshold with a hard ceiling is no longer fully adaptive, which felt wrong when we wrote it. It is also the single highest-value line in the file.

Telling paper from a block of printed text

Capping the threshold gets the page border back into the edge map. It also lets a lot of rectangles in that are not pages — and the most dangerous one is a dark block of printed text, because it is genuinely a bright-surrounded rectangle with strong edges.

The first instinct is a contrast magnitude: require the inside of the quad to be meaningfully brighter than the outside. We had that set at 14 gray levels, which is the median paper-on-light-wood step from above.

That was the wrong knob. A threshold at the median of true pages rejects about half of the true pages. What actually discriminates is not how big the step is but which way it points:

minContrast: 8,   // 40th percentile of (inside − outside) across each side, in gray levels.
                  // A page on light wood measures 14 at the median, so 14 rejected half of them;
                  // the polarity rule is what keeps text blocks out, not this magnitude.

The polarity rule is the actual test: a quad is a page only if it is brighter inside than outside, sampled at two depths, on all four sides. A text block fails that on the sides where more text continues past it. A page on a desk passes it everywhere.

There is still a case that beats local evidence: a large dark region of print can look like "the desk" if you only sample a few pixels out. So before applying a detection automatically — as opposed to just pre-positioning the handles — we probe farther outside, looking for the page continuing past that edge. If the page continues, the edge was not the page edge.

The guards that are product decisions wearing algorithm clothes

Three of the constants in that file are not computer vision. They are decisions about what to do when we are unsure, and they are the ones I would port to any similar project:

borderMargin: 0.02,        // a side within 2% of the frame edge is "on the border"
applyMinAreaRatio: 0.25,   // a smaller page is only proposed, never applied
minSideSupport: 0.42,      // every side needs this much edge support to be applied

A quad with a side lying on the image border is rejected. If the "page edge" is the edge of the photo, the page is not fully inside the frame, and the honest default is the whole frame rather than a crop that silently cuts off whatever was outside it.

The largest well-supported quad wins. This sounds like a tie-breaker and is actually a correctness rule: it makes an inner rectangle — a photo printed on the page, a bordered table — lose to the page around it.

Below a quarter of the frame, a detection is only a proposal. It pre-positions the four draggable handles and waits. Auto-applying a small quad is how you produce the single worst outcome in this whole category: a confident crop that removes two thirds of somebody's passport.

That last one generalises. The result of this module is a proposal and never the truth; every handle stays draggable, and the caller decides from a confidence value whether to apply it silently. When detection fails outright it returns null, and null means keep the entire original image. A scanner that crops wrong is worse than a scanner that does not crop.

Two features that ship turned off

weakEdgeRecovery: 0,   // Experimental: desktop cost is not yet justified by coverage gains.
contourRecovery: 0,    // Correct proposals improved, but the measured latency gate failed.

Both of these work. Both improve the numbers. Both are disabled in the shipped defaults because they did not clear a latency budget on the hardware people actually hold. Leaving them in the file at 0, with the reason written next to them, has been more useful than deleting them — the next person to look at this does not have to rediscover that the idea was tried and why it is not on.

What I would take away from this

The general shape of the bug is worth more than the specific fix: an adaptive parameter adapts to the strongest thing in your input, and the strongest thing is often not your signal. Text beats page borders. Specular highlights beat document edges. The loudest object in the frame sets your threshold, and if your target is quiet, it gets deleted.

When an adaptive method underperforms, it is worth checking whether it is adapting to something you did not intend before reaching for a better algorithm. In our case the better algorithm was a ceiling of 30.

If you want to watch this run on your own photo, it is at a browser-based document scanner — the detection happens in the tab and your files are never uploaded, so the Network panel is a fair way to check that claim.

I Built a Real-Time Perception Stack on CARLA. Here Is What Actually Happened.

Sun, 30 Aug 2026 08:57:09 +0000

I want to tell you about the three weeks I spent making YOLO and ROS 2 talk to each other inside a simulator, and why it was nothing like the tutorials made it look.

This is not a clean architecture post. This is a "here is what broke, here is what I changed, here is what I actually learned" post.

If you are building robotics perception systems, or you are just curious what it looks like to wire YOLOv11, ByteTrack, StrongSORT, CARLA, and ROS 2 into something that runs in real time, this one is for you.

What I was building and why

The goal was a real-time robotic perception system that could:

detect objects in a moving vehicle simulation
track those objects across frames with stable identities
feed structured perception data into a ROS 2 navigation stack
run fast enough to be genuinely useful for autonomous decision making

CARLA was the simulator. YOLOv11 was the detector. ByteTrack and StrongSORT handled multi-object tracking. ROS 2 was the middleware. ONNX and TensorRT handled edge optimisation.

On paper this is a well-defined problem. In practice, every one of those components has opinions about data formats, timing, and threading that you only discover by running them together.

Problem one: getting camera data out of CARLA cleanly

CARLA has its own Python API and its own event-driven callback system. ROS 2 has its own message types, publisher model, and executor. Getting them to cooperate without dropping frames or blocking the main thread is not automatic.

Here is a simplified version of how I handled the camera subscription and published frames into ROS 2:

import carla
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import numpy as np

class CARLACameraNode(Node):
    def __init__(self):
        super().__init__('carla_camera_node')
        self.publisher = self.create_publisher(Image, '/carla/camera/rgb', 10)
        self.bridge = CvBridge()

    def camera_callback(self, image):
        array = np.frombuffer(image.raw_data, dtype=np.uint8)
        array = array.reshape((image.height, image.width, 4))
        bgr = array[:, :, :3]
        msg = self.bridge.cv2_to_imgmsg(bgr, encoding='bgr8')
        msg.header.stamp = self.get_clock().now().to_msg()
        self.publisher.publish(msg)

The key thing I got wrong initially: I was not stamping the header with the ROS 2 clock.

That one missing line caused timestamp mismatches downstream that made the tracker behave erratically. ByteTrack and StrongSORT both calculate displacement and velocity across frames using time deltas. If those deltas are wrong or inconsistent, the association algorithm produces noisy, unstable tracks that look like a model problem but are actually a data problem.

That bug cost me two days.

Add the timestamp. Always. It is not optional in robotics.

Problem two: YOLO runs fast, ROS 2 callbacks do not wait

YOLOv11 at a reasonable resolution on a GPU is quick. But "quick" in model benchmarking terms and "quick" in robotics middleware terms are genuinely different things.

ROS 2 uses a callback-based execution model. If your inference call is blocking the subscription callback thread, you are missing messages while you compute. The subscriber queue fills. You start processing frames that are already stale by the time inference finishes. Your tracker is now seeing the world as it was, not as it is.

The fix was decoupling ingestion from inference using a bounded queue and a dedicated inference thread:

import threading
import queue
from rclpy.node import Node
from sensor_msgs.msg import Image

class PerceptionNode(Node):
    def __init__(self):
        super().__init__('perception_node')
        self.frame_queue = queue.Queue(maxsize=5)
        self.subscription = self.create_subscription(
            Image,
            '/carla/camera/rgb',
            self.image_callback,
            10
        )
        self.inference_thread = threading.Thread(
            target=self.run_inference,
            daemon=True
        )
        self.inference_thread.start()

    def image_callback(self, msg):
        if not self.frame_queue.full():
            self.frame_queue.put(msg)

    def run_inference(self):
        while rclpy.ok():
            msg = self.frame_queue.get()
            # run YOLOv11 inference here

The maxsize=5 on the queue is deliberate. When inference cannot keep up, I drop old frames rather than accumulate a growing backlog of stale images.

This is a real-time systems principle worth internalising: a stale frame processed late is usually worse than a dropped frame. Timeliness matters as much as completeness in a live perception pipeline.

Problem three: tracking across frames is not free

ByteTrack and StrongSORT both work by associating new detections to existing tracks across frames using motion models and, in StrongSORT's case, appearance embeddings. This sounds clean until you hit occlusion, rapidly moving objects, or a detector that briefly misses something due to a lighting change or partial obstruction.

What I observed specifically in CARLA: when the ego vehicle turned sharply, vehicles and pedestrians would leave the camera frustum and re-enter within a few frames. The tracker sometimes assigned fresh IDs to those returning objects, breaking the continuity that downstream navigation reasoning depended on.

The parameters that mattered most here were:

tracker_config = {
    "max_age": 30,        # frames a track survives without a detection match
    "min_hits": 3,        # detections needed before a track is confirmed
    "iou_threshold": 0.3  # minimum overlap for detection-to-track association
}

Increase max_age too much and you get ghost tracks for objects that have genuinely left the scene. Decrease it too much and you lose real tracks during brief occlusions. Tune iou_threshold too high and the tracker drops tracks during fast motion. Too low and it starts merging distinct objects.

There is no universally correct set of values. They depend on your scene density, your detector confidence, your frame rate, and what your downstream system needs from the tracker. The only way to find the right values is to run scenarios, observe failures, and adjust deliberately.

Problem four: edge hardware changes the problem completely

Everything above ran well on my development GPU. Then I exported the model and tested closer to constrained hardware conditions.

The frame rate dropped. The tracker fell behind. The pipeline that felt smooth in development started introducing the kind of latency that makes real-time perception meaningless.

The fix was ONNX export followed by TensorRT optimisation:

import torch

model = load_yolov11("yolov11n.pt")
model.eval()

dummy_input = torch.randn(1, 3, 640, 640).cuda()

torch.onnx.export(
    model,
    dummy_input,
    "perception.onnx",
    opset_version=17,
    input_names=["images"],
    output_names=["output0"],
    dynamic_axes={"images": {0: "batch"}}
)

After TensorRT conversion and INT8 quantisation, inference time dropped enough to bring the full pipeline back into a usable range. But the deeper lesson is not about TensorRT specifically. It is that edge deployment is a different problem to development, and discovering that at the end of the project means redesigning under pressure.

Test on target hardware early. Not day one necessarily, but not month three either.

Problem five: your confidence threshold is a system design decision

This one is subtle and I did not fully appreciate it until I had the full pipeline running.

YOLO outputs detections with confidence scores. It is tempting to pick a threshold like 0.5 and move on. But in a live robotics system, that threshold is not a model hyperparameter. It is an architectural decision about when your robot is allowed to act on what it sees.

A threshold that is too low sends noisy, unreliable detections to the tracker. The tracker generates unstable tracks. The navigation stack gets bad inputs. The robot behaves erratically.

A threshold that is too high causes the detector to miss real objects in challenging conditions. Tracks are lost. The robot navigates as if obstacles do not exist.

What helped was treating the threshold not as a single number but as part of a validation layer:

def is_valid_detection(detection, track_age, platform_stable):
    if detection.confidence < CONFIDENCE_THRESHOLD:
        return False
    if track_age < MIN_TRACK_AGE:
        return False
    if not platform_stable:
        return False
    return True

Multiple signals need to agree before a detection influences navigation. This pattern, requiring consensus across confidence, temporal stability, and platform state, is something I now consider non-negotiable in any perception pipeline with real-world consequences.

Five things I know now that I did not know at the start

Timestamps are not optional. Misaligned timestamps produce bugs that look like model bugs and take days to diagnose.

Decouple your threads early. Blocking the callback thread is an easy mistake with expensive downstream consequences.

The tracker is part of your perception system. Not an add-on you configure at the end. Design around it from day one.

Test on target hardware before you think you are done. Edge deployment is a different engineering problem to development.

Log everything during development. You cannot replay failures you did not record, and in robotics you will need to replay failures.

Closing thought

Building this system taught me that robotics perception is not a model problem. It is a systems problem where the model is one moving part among many.

The interesting engineering is in how you connect those parts, how you handle failures at the boundaries between them, and how you make the full pipeline behave reliably when nothing goes quite according to plan.

Which is always.

If you are building something similar or want to dig into any part of this, drop a comment. I am always happy to talk robotics and perception systems.

Can AI Remember What It Sees?

Mon, 24 Aug 2026 23:26:09 +0000

So, I have been asking myself this question for quite a while, Can AI actually remember what it sees? and tell a user later that "I saw X at place Y last", or for instance when a person goes to a cafe and forgets something. e.g let's say they actually forgot their charger. The system should be able to say, "Hey, I think you came here with a charger and now you are leaving without one, or you forgot your charger".

I think that will be really helpful, and it can also help a person when it observes that they skipped a step, then the system proceed to say, "I think you skipped step 5 in this process".

So the vision is: can AI maintain enough continuity about the physical world to notice when something important has been forgotten, misplaced, changed, or left unfinished?

I am exploring these four questions:

Perception: what exactly is in the scene?
Identity: is this the same physical object as before?
Memory: what happened to it over time?
Reasoning: does the current situation look wrong or incomplete?

This is Day 1 for me to actually take a step towards creating this, btw I did my research and found some amazing projects around this, so I am actually going to be looking at them (I already started).

What I did today, I just created a simple Python program that calls a simple YOLO model, and then lets it detect objects, I focused on detecting bottles, cause I wanted to just see if the system can remember the last time it actually saw the object and it did, it actually remembered the last time it saw the object, like the exact time.

My bottle demo has basically answered the perception + primitive memory question.

It shows that the system can actually see what is in a scene and then remember it and when last it saw it. So now I will be working on identity and reasoning, and testing it, I will also be documenting it all.

Thank you, I hope you follow the build process.

How to build yolov8 dental object detection model

Fri, 21 Aug 2026 08:03:06 +0000

For anyone studying computer vision and medical image analysis, applying deep learning to dental radiography presents distinct challenges in feature extraction and anatomical localization.
Dental radiographs often feature dense, overlapping structures, low contrast margins, and significant variation across imaging hardware. Manual landmark identification remains labor-intensive and susceptible to inter-observer variability. This tutorial utilizes YOLOv8 as a single-stage detector because it delivers high mean Average Precision (mAP) on closely packed, low-contrast bounding boxes while maintaining real-time inference speeds suitable for practical clinical pipelines and batch processing.

Detailed written explanation and source code: https://eranfeit.net/how-to-build-yolov8-dental-object-detection-model/

The end-to-end workflow begins by establishing an isolated PyTorch environment with CUDA acceleration. The dataset is structured into normalized YOLO-format annotations paired with a custom dataset configuration file. Before initiating model training, normalized bounding box coordinates are converted to pixel dimensions and rendered using OpenCV to verify annotation alignment against raw X-ray data. Once validated, the YOLOv8 architecture is trained on GPU, and the resulting weights are evaluated by comparing predicted bounding boxes against ground-truth annotations in a side-by-side visual analysis.

Reading on Medium: https://medium.com/object-detection-tutorials/how-to-build-yolov8-dental-object-detection-model-07ee6ee36296

Join my Newsletter : https://eranfeit.net/advance-your-skills-in-computer-vision-ai/
Deep-dive video walkthrough: https://youtu.be/mfv1ps-tHDk

This content is intended for educational purposes only. Constructive feedback, technical questions, and discussions regarding custom dataset training or model optimization are welcome in the comments.

EigenCAM YOLOv5 Explained: Understanding What YOLOv5 Sees

Sat, 15 Aug 2026 18:12:04 +0000

For anyone studying Image Heat Maps...
Understanding model interpretability in deep learning object detection presents distinct challenges compared to standard image classification. Traditional attribution methods like Grad-CAM rely heavily on backpropagated class gradients, which can yield noisy or incomplete visualizations when applied to object detection architectures like YOLOv5 due to multi-scale feature maps and simultaneous localization-classification outputs. EigenCAM resolves this limitation by applying Principal Component Analysis (PCA) directly to the learned feature activations of the convolutional layers. By computing the primary eigenvector of the feature representations without requiring gradient calculations or specific target class selection, EigenCAM produces clean, class-agnostic heatmaps that clearly delineate spatial attention regions.
The implementation workflow begins by loading a pretrained YOLOv5 model using PyTorch and identifying key target layers—such as specific feature extraction bottlenecks—within the backbone or neck architecture. After preprocessing the input image to match the network requirements, forward inference is executed to capture feature maps at the target layer. EigenCAM then computes the first principal component across these activation channels, normalizes the resulting matrix, and overlays the heatmap onto the original image using OpenCV. This systematic process provides a deterministic view into which visual features most strongly influence the detector's focus.
Reading on Medium: https://medium.com/object-detection-tutorials/how-to-use-eigencam-for-yolov5-object-detection-07e4a386e567
Detailed written explanation and source code: https://eranfeit.net/how-to-use-eigencam-for-yolov5-object-detection/
Join my Newsletter : https://eranfeit.net/advance-your-skills-in-computer-vision-ai/
Deep-dive video walkthrough: https://youtu.be/pcgvcIJuKnI

This content is prepared strictly for educational purposes. Technical inquiries, implementation questions, and constructive feedback regarding visual explainability techniques are welcome in the comments.
Enjoy , Eran

YOLOv5 #HeatMaps #ObjectDetection

The "Physical NFT" is Dead. Long Live the Verifiable Digital Twin

Thu, 30 Apr 2026 16:06:00 +0000

We’ve spent years putting "Monkey JPEGs" on the blockchain. But as a system architect, I’m less interested in speculative art and more interested in the mathematical reality of physical objects. Can we bridge the gap between a raw river rock and a Hedera hashgraph?

The Reality: Photogrammetry (Museum-Grade Ingestion)

Before we talk about ledgers, we have to talk about the data. Photogrammetry is the process of extracting 3D information from 2D photographs. By taking hundreds of overlapping photos, we can reconstruct a dense point cloud and eventually a high-resolution 3D mesh.

This isn't just a "3D model"; it’s a forensic record. For example, check out this Granite Head of Amenemhat III on Sketchfab. That level of detail—the micro-textures of the stone—is what we are aiming to "anchor."

The Ledger: Why Hedera and the DAG?

For a project involving thousands of physical items (like my current Backyard Quarry experiment), a traditional, slow, expensive blockchain is a non-starter. I’ve been looking into Hedera specifically because of its Directed Acyclic Graph (DAG) structure.

Unlike a linear chain, a DAG allows transactions to happen in parallel via a "Gossip Protocol." This means finality in seconds for a fraction of a penny ($0.0001). If I want to anchor the provenance of a single rock, it has to be financially viable.

The Concept: The Decentralized Scribe

In my work, I define a Decentralized Scribe as a high-integrity agent—an automated system that sits between the physical world and the ledger. It:

Observes: Ingests the photogrammetry data.
Hashes: Creates a geometric "fingerprint" of the 3D mesh.
Signs: Anchors that fingerprint to the Hedera Consensus Service (HCS).

The Scribe is the bridge of trust. It proves that the 3D model on your screen matches the physical object in my hand at a specific point in history.

Connecting the Dots: The "Ask"

I’m currently exploring this intersection, conceptually in my Rock Quarry series, and I’d love to hear from this community:

Geometric Hashing: Is a hash of a 3D mesh stable enough? If I re-scan the object next month, will the "fingerprint" stay the same?
The Oracle Problem: How do we decentralize the "Trust" that the scan actually matches the item?
Utility: Beyond collectibles, where does "Verifiable Provenance" actually solve a pain point for you?

Is this the bridge we’ve been waiting for, or is the "Oracle Problem" too high of a wall to climb?