Coder Legion - Recent posts tagged computer-vision

Our AI staging model sometimes moves the camera. Here is how to spot it, and how we catch it

Sat, 24 Oct 2026 23:14:24 +0000

A virtual staging render is supposed to be the same photo with different furniture. Same walls, same windows, same door, and the same spot where the photographer stood. Most of the time the image model gives exactly that. Sometimes it does not.

We run an AI virtual staging tool, and one of the quieter failures we deal with is this one: the model moves the camera.

What it looks like

It is rarely dramatic. On its own the render looks great. The sofa is nice, the light is warm, nothing is melted or floating. Only when you put it next to the original do you notice that the room is being seen from a different place.

The versions we see most often:

A step back. A wardrobe that the photo cut off at the edge turns into a wall, and the view moves back behind it. Suddenly there is more room than the photographer ever captured.
Straightened up. A bedroom shot from the corner comes back facing the bed square on, like a catalogue page.
Zoomed in. The frame creeps toward the fireplace or the archway.

For a listing photo this matters. The picture is supposed to show the room a buyer will walk into. Some MLS rules go further and require an unstaged photo of the same view alongside the staged one. If the staged version was taken from somewhere else, the pair does not match, and anyone comparing them will see it.

How to spot it by eye

You do not need software for the obvious cases. Put the original and the render side by side and ignore the furniture. Look at the things furniture cannot change:

The line where the ceiling meets the walls. It should sit in the same place in both images. If it is higher, lower or tilted differently, the camera moved.
Window and door edges. Check how far each one is from the edge of the frame. If a door that touched the left edge now has a strip of wall beside it, the view shifted.
Whatever is cut off. Anything the original cropped at the frame edge should be cropped at the same place.

If all three line up, the camera almost certainly stayed put, whatever happened to the sofa.

Why we could not just ask an AI

Our first idea was the obvious one. We already use a vision model to compare every render with its original and answer a fixed list of yes-or-no questions: is the window still there, is the door still a door, and so on. So we added one more question: is this the same viewpoint?

It did not work. Renders shot from a visibly different spot kept passing, across three separate rounds of cases we had checked by hand. We tried laying the two images over each other at half strength and asking whether the walls looked doubled. Not reliable either. A stronger model later had the opposite habit and called good renders moved. Today its answers about the camera do not count for room photos at all.

What we built instead

So we measure it, much the way you would by eye, with two plain checks.

The first one traces the line where the walls meet the ceiling across the whole width of the photo and compares the two lines. Furniture rarely reaches the ceiling, so that line should not change. Its weak spot is decoration: a dark beam or a picture hung high can pull the traced line off course.

The second one estimates how far away the back wall is, column by column, in the upper part of the image, and compares the two results. Its weak spot is glass. Windows and mirrors are read differently every time.

The useful part is that the two weak spots rarely meet. A picture on the wall does not change depth, and glass does not move the ceiling line. So a render is flagged only when both checks agree that the camera moved. On our hand-labelled set of 24 pairs, that caught all 5 moved renders with no false alarms. The margin was thin, so borderline scores are only logged and never fail a render.

A flagged render is rejected the same way as one with a missing door, and the pipeline makes another attempt. If the check itself breaks or runs slow, the render is judged by the other checks instead of being held up.

What it still gets wrong

Three things, and we would rather name them than hide them.

Decoration and glass in the same room. A towel rail next to a glass shower. A picture shelf beside a full-height window. Both checks get fooled a little at the same time, and a render is flagged even though the camera held still.

A turn under a flat ceiling. If the camera only turns and the ceiling is flat, the ceiling line barely changes, the first check stays quiet, and the render slips through. A bedroom straightened to face the bed is the usual case.

A straight step back. This one showed up in October on close-up, cluttered photos. Both checks look at where things sit across the frame, not at how big they look, so a pure pull-back hardly registers. On 20 hard photos the check caught 2 of 5 bad frames and missed 3, all of them pull-backs. A simple scale comparison caught 2 of those 3, but it also raised false alarms of its own, so it is not live yet.

What we took from it

A model that describes a room well is not the same as a model that can tell where the camera stood. For that one question, two boring measurements did better. They are not perfect, but they fail in ways we can name and test.

If you are checking a staged photo yourself, the same habit works: skip the furniture and look at the ceiling line, the window edges and what is cut off at the frame. If you are buying, you can ask for the unedited original of every staged photo. And if you are the one taking the picture, remember that close-up, cluttered shots are where the step-back problem showed up, so shoot the room wide and level.

The full technical write-up, with the six approaches we tried, the calibration numbers and the bugs we hit in production, is on dev.to.

We build this at AI Flip Room.

How we check every AI-generated room render before a user sees it

Fri, 09 Oct 2026 18:55:25 +0000

Image models are very good at restyling a room and quietly bad at leaving the room alone. Ask for a Japandi living room and you get beautiful furniture, and sometimes also:

a window that moved, shrank, or appeared over a kitchen sink where the photo shows a solid wall;
a closet door that came back as a curtain, a sliding screen, or plain wall;
a sofa parked across the only archway;
a new doorway at the frame edge, or a niche carved into a flat wall;
a camera that slid sideways;
or the opposite: a picture that is almost exactly the photo you uploaded.

Our users put these images in property listings, where a staged photo may change the furniture but not the windows, walls or doors (why that line exists). So every render is checked automatically before anyone sees it. Here is how, and what building it taught us.

The pipeline at a glance

For an indoor room, each render goes through three gates in parallel:

A vision-model judge compares the original photo with the render and answers a fixed list of yes/no questions.
A mechanical camera check measures whether the viewpoint moved, with no language model involved.
A crop-level presence check looks at each door, window and staircase found in the original, at full resolution.

If all three pass, the render is served. If not, we retry. If nothing passes, the user gets the best attempt, clearly marked as a draft.

Gate 1: a judge that only answers booleans

The judge gets the original and the render, both downscaled, and must return a fixed JSON object. We use structured output with a schema, so it cannot reply in prose. An earlier version asked for "only JSON" and scraped it out with a regex; when the output got long or cut off, the parse failed and the gate silently did nothing.

Indoors there are 16 checks, outdoors 9. Indoors they cover:

Openings: every door is still a door you could open; every passage has clear floor to walk through.
Windows: same count, position, size and shape; none invented; not hidden behind a tall wardrobe; the same view outside.
The room shell: ceiling, fireplace, no carved niches, features on the same walls, solid frame edges, no new rooms or corridors.
Finish consistency: no half-repainted rooms.
Content rules: no people, and no photo or realistic portrait of a person in the wall art.
Camera: same viewpoint, same room depth.

Outdoors: buildings, fence line and gates, terrain (no new pool), mature trees, background, and a usable driveway.

A simplified, illustrative verdict:

{
  "doors_preserved": true,
  "openings_clear": false,
  "windows_ok": true,
  "window_unblocked": true,
  "walls_uniform": true,
  "frame_edges_solid": true,
  "no_faces_in_art": true,
  "fail_reasons": ["sofa end stands on the archway floor"]
}

A render passes only if every counted boolean is true; the reasons feed logs and the retry prompt. Most of the work went into the wording:

Say what must pass. Restyling is the product, so the instructions list what is expected (new floors, paint, wallpaper, fixtures) and say "unsure means pass". The exception is a short list where unsure means fail, led by a window that may not have been in the original. An invented window is the worst defect a listing photo can have.

One question per field. While the frame-edge rule was a sub-clause of a broader "no phantom architecture" check, new doorways at the frame edge got through. Its own boolean made the model actually look.

Naming things primes the judge. An early niche check listed which shelves were allowed, and the judge started flagging every shelf. The current wording is purely geometric: is this flat wall still one plane?

Allow "nothing to compare". The "same view through the window" check kept firing on originals with blinds drawn or glass blown out to white. That case is now an explicit pass.

Wording is measurable. The old wall-art wording wrongly failed 10 of 16 good frames (stylised line drawings counted as faces). The rewrite failed 0 of 16 and still caught all 5 real portraits.

Gate 2: measuring the camera instead of asking about it

Early on, the vision model was close to blind to viewpoint changes: renders shot from a visibly different spot passed every neural check. Later, a stronger judge had the opposite problem: on 43 renders where the camera had not moved, it said it had 13 times. So indoors the judge's camera answers no longer decide pass or fail. A measurement does, with two channels:

Ceiling line. We trace the wall/ceiling boundary as a curve (dynamic programming over the vertical brightness gradient) and compare the original's curve with the render's.
Depth profile. A monocular depth model estimates both images. Per column we take how far the far wall reads, then compare the profiles with a robust fit, a rank correlation and a search for a sideways shift.

Each channel is fooled by something different. Paint fools the ceiling line (dark beams, art on a mantel). Glass fools depth (windows and mirrors are estimated differently each time). Paint is flat in depth, and glass does not move the ceiling line, so we flag a shift only when both agree: the minimum of the two normalised scores.

On 24 hand-labelled pairs from two rooms, this caught 5 of 5 real shifts with 0 false flags. The margin was thin, so a grey zone around the threshold is logged, not failed. Outdoors there is no ceiling to trace, so the mechanical check is off and the judge's camera question stays in charge.

Gate 3: crops, because a door is 80 pixels wide

At the judge's resolution a front door is a thin strip. So doors, windows and stairs are detected at upload, and after each render a model is asked about each crop at full resolution: is it still there? A mechanical channel also measures whether a window drifted or widened. On 87 labelled pairs, the crop judge caught 7 of 10 broken renders with no false alarms on the 75 clean ones; the mechanical channel caught the other 3.

When a render fails

Retries depend on the kind of defect.

Placement defects (a blocked passage, a face in the art, half-painted walls) are retried on the same image model with the judge's reasons added to the prompt.
Geometry defects go to a second, pricier image model. We measured a same-model retry fixing broken geometry about one time in five. In an earlier bake-off the two models were clean on 18 and 17 of 20 renders, and 19 of 20 together: their failures barely overlap.

The geometry retry gets a fixed, count-based instruction ("same set of openings, not one more, not one less"), never the judge's description of the defect. When a retry prompt named an arched opening at the left edge, the retry model painted exactly that.

If nothing passes, the user still gets the best attempt, under a "rough draft" banner. On paid plans the first four rejected drafts in a rolling week don't use a render; the free plan gets one. The cap is deliberate: a draft is a real, downloadable image, so unlimited forgiveness would be a free image farm.

The checks fail open. If the judge itself errors or times out, the user is not blocked: the image is served flagged, and an unverified render is not charged as a draft.

The near-copy trap

At first, "best attempt" meant the one with the fewest failed checks. The attempt that breaks the least is often the one where the model barely did anything. In one real case the first attempt was a convincing industrial restyle that failed on geometry, and the user was served the second: their old room, old wallpaper and all, plus a pair of lamps. The ranking was paying the model for doing nothing.

The fix is a change meter: shrink both images to a tiny thumbnail and count how much of the frame visibly changed. No model call, and we calibrated it on 110 real renders and 10 synthetic near-copies (recompressed, blurred, brightened, shifted, a couple of lamp-sized patches). The ranking now:

// simplified
function draftRank(a: Attempt): number {
  if (a.checkerErrored) return LAST;
  let rank = a.failedChecks.length;
  if (a.hasPeopleOrRealFacesInArt) rank += HARD_BAN;
  if (a.isNearCopy && a.intensity !== "light") rank += NEAR_COPY;
  return rank; // lowest is served
}

A light restage is exempt, because there the user asked us to keep the room.

Picking the judge model with a calibration set

The judge runs on every attempt, so a cheaper model is tempting. We keep a calibration set of 31 hand-labelled frames and run every candidate on it twice. A cheaper model at under half the price per call missed 6 defects per run against our current judge's 3. On 46 live renders with 8 defects marked by eye, it caught 0 and 1 across two runs; the current judge caught 6. A newer model wasn't clearly better either: fewer misses (2 vs 3), twice the false alarms (4 vs 2). We kept the current judge.

Takeaways

Ask the judge narrow yes/no questions, not one "is this good?".
Where geometry can be measured, measure it, with channels whose failure modes don't overlap.
Calibrate every wording change and model swap on a labelled set, twice.
Design the failure path as carefully as the check: ranking, draft label, who pays.
"Least broken" can mean "least changed". Measure change directly.

We build this at AI Flip Room.

I Built a BSL Translation Extension That Nobody Asked For. Here Is Why I Did It Anyway.

Sun, 30 Aug 2026 08:58:02 +0000

Late last year I started building something that had no job description attached to it, no business case ready, and no guarantee that anyone would use it.

I built Signlytic, a Chrome extension that intercepts live captions from streaming platforms and translates them into British Sign Language signing animations in real time, rendered as a floating overlay panel directly on top of the video.

Not because someone commissioned it. Not because it was the safe, hireable thing to build. Because it felt like a problem worth solving and because I wanted to know if it was even technically possible.

This is the story of what I built, how I built it, what repeatedly broke, and what the whole experience taught me about the kind of engineering that actually matters.

The problem I kept thinking about

British Sign Language is the primary language of a significant deaf community in the UK. Most digital content is not natively accessible in BSL. Subtitles exist on most platforms, but reading subtitles while watching video is a fundamentally different cognitive experience from watching sign language, which is the natural language for many BSL users.

The question I kept sitting with was this: what would it look like if a browser could intercept live captions and render them as BSL signing in real time, without requiring any changes from the streaming platform, without cloud APIs, and without meaningful delay?

There was no off-the-shelf answer. So I started building one.

The architecture

Signlytic is a multi-model pipeline connecting several distinct systems:

Sign recognition: A Video-SWIN-T model trained to 100% Top-1 accuracy on 5,203 BSL signs, handling the recognition direction for BSL input.

Gloss translation: Groq-hosted Llama 3.3 70B converting English captions to BSL gloss, the written intermediate representation of sign language grammar that bridges English word order and BSL sentence structure.

Voice synthesis: Coqui XTTS v2 for speech output in the reverse direction of the pipeline.

The Chrome extension: Captures live captions across streaming platforms, renders BSL animations as a floating panel, and handles fallback when captions are unavailable.

The four components talk to each other through a local FastAPI backend that manages the pipeline and keeps the extension from needing cloud access for core functionality.

How caption detection actually works

The first engineering problem was fundamental: how do you reliably intercept captions across platforms that all structure their DOM differently?

The answer is a MutationObserver watching the document body for text node additions, with platform-specific filtering to reduce false positives:

const observer = new MutationObserver((mutations) => {
    for (const mutation of mutations) {
        for (const node of mutation.addedNodes) {
            if (node.nodeType === Node.TEXT_NODE) {
                const text = node.textContent.trim();
                if (text.length > 2) {
                    sendToTranslationPipeline(text);
                }
            }
        }
    }
});

observer.observe(document.body, {
    childList: true,
    subtree: true,
    characterData: true
});

The subtree: true flag is not optional. Caption nodes on most streaming platforms are deeply nested inside shadow DOM structures that change between platform updates. Targeting a specific element selector breaks the moment the platform updates their frontend. Observing the full document body and filtering by text content is more resilient, even though it means processing more mutation events.

When captions are not available at all, the extension falls back to the Web Speech API using the en-GB locale. This covers cases like live streams without auto-captions, or platforms the MutationObserver has not been configured for yet.

How the gloss translation works

Raw English caption text cannot be directly rendered as BSL. BSL has its own grammar, its own word order, and its own sentence structure that differs from English in significant ways. The intermediate representation is called gloss, a written form that maps English concepts to BSL grammar.

The Llama 3.3 70B model running on Groq handles this translation step. A simplified version of that API call looks like this:

from groq import Groq

client = Groq()

def translate_to_bsl_gloss(english_text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a British Sign Language linguist. "
                    "Convert the following English text to BSL gloss notation. "
                    "Preserve meaning. Adjust word order to BSL grammar. "
                    "Output gloss only, no explanation."
                )
            },
            {"role": "user", "content": english_text}
        ]
    )
    return response.choices[0].message.content.strip()

The system prompt is doing significant work here. Without explicit instruction to output gloss only, the model tends to explain its translation choices. In a real-time pipeline with caption text arriving every few seconds, you need clean output on every call.

What broke repeatedly

Platform inconsistency was relentless. YouTube structures its caption DOM differently from Netflix, which differs from BBC iPlayer, which differs from Amazon Prime. There is no standard. Each platform required individual testing and observer tuning. Some platforms dynamically recreate their caption containers on seek or quality change, which kills a targeted observer but survives a body-level one.

Timing and queue management were harder than expected. Live captions arrive in short, irregular bursts. BSL animation rendering takes time, especially for 3D Mixamo avatar playback via Three.js. If the animation queue grows faster than it can drain, the signing falls behind the audio and the experience breaks down completely.

The fix was a priority queue that drops the oldest pending animation segment when the queue depth exceeded a threshold, similar to the bounded queue pattern I used in my ROS 2 perception work. In real-time human-facing systems, falling behind and catching up is a worse experience than occasionally dropping a segment cleanly.

The gap between recognition accuracy and production performance. The Video-SWIN-T model achieves 100% Top-1 accuracy on the trained sign vocabulary. That number is real. It is also a controlled dataset metric.

Live, unconstrained video input introduces lighting variation, angle variation, background clutter, partial hand occlusion, and signer-to-signer variation that does not exist cleanly in training data. Every computer vision engineer knows this gap exists. Every product builder has to decide honestly how to communicate it. I chose to scope the extension's recognition capabilities explicitly rather than imply a performance that only holds under ideal conditions.

The rendering layer: 2D versus 3D

The extension supports two rendering modes for BSL animation output.

The 2D mode uses MediaPipe Holistic to render pose landmark skeletons, a lightweight representation that works well on lower-end hardware and loads fast. The 3D mode uses Mixamo avatars rendered through Three.js r128 for a more naturalistic signing appearance.

The tradeoff is real. 3D rendering is more legible for BSL users because it more closely resembles actual human signing. But it is heavier, slower to initialise, and can introduce rendering latency that compounds with translation latency if the system is already under load.

The draggable and resizable overlay panel is built using an iframe-based architecture inside Chrome MV3. This is the current Chrome extension manifest standard, and it changes how content scripts, service workers, and injected iframes communicate compared to MV2. That migration from MV2 patterns to MV3 constraints was its own engineering challenge that deserved more preparation time than I gave it initially.

What this project taught me about multi-model AI systems

Multi-model pipelines fail at the connections, not inside the models. The Video-SWIN-T, the Llama translation layer, and the XTTS synthesis are all individually strong. The hard engineering is connecting them without introducing unacceptable cumulative latency, handling failure at each handoff gracefully, and making the end-to-end system feel coherent to someone using it in real time.

Fallback design is product design. What happens when captions are unavailable? What happens when the translation is ambiguous? What happens when the animation queue falls behind? If you have not designed the failure states, you have not designed the product. You have designed the happy path and called it done.

Accessibility is a technical discipline. It is not enough to have good intentions and an interesting architecture. Accessible technology requires rigorous engineering around latency, consistency, reliability, and graceful degradation. The users who depend on it deserve systems that are actually dependable, not just impressively described.

174 offline signs versus 5,203 via backend. The extension ships 174 BSL signs bundled for completely offline use. The full 5,203-sign vocabulary requires the local FastAPI backend to be running. This two-tier architecture was a deliberate product decision: the extension should work in a degraded but functional state without any setup, and scale up when the backend is available. That kind of progressive capability is harder to build than a single-mode system but meaningfully better for real users.

What comes next

A Windows desktop application built in Electron is in development. The goal is system-wide caption capture across any application, not just browser-based streaming. This requires a fundamentally different approach to caption interception since there is no DOM to observe, which means working with Windows accessibility APIs and screen reader hooks.

The longer-term vision is a fully bidirectional BSL communication system: sign-to-speech for BSL users communicating outward and speech-to-sign for hearing people communicating inward, running locally without cloud dependency.

You can explore Signlytic and learn more about the project at signlytic-ai-website.vercel.app.

I am documenting the engineering on GitHub as I go. If you are working in accessibility technology, computer vision, browser extension development, or local AI deployment, I would genuinely like to connect.

Closing thought

There is a category of project nobody has a template for. No tutorial, no course, no job description points you toward it. You find it by sitting with a question long enough that building becomes the only honest answer.

Signlytic was that kind of project for me.

It is technically demanding, genuinely unfinished, and one of the most meaningful things I have built. Not because it is perfect. Because the problem is real, the technology is ready enough to try, and the engineering forces you to think at a level that standard work never demands.

Build the thing you cannot stop thinking about.

The engineering will follow.

How AI Sees Images: From Pixels to Understanding (Part 1)

Fri, 10 Jul 2026 16:48:25 +0000

When Machines Look at the World

Have you ever uploaded a photo to Google Lens, ChatGPT, or any AI-powered application and wondered how it instantly knows what's inside the image? Maybe you uploaded a picture of your cat, and within a second the AI replied, "This is a gray cat sitting on a wooden chair." Or perhaps you scanned a plant, and the application immediately identified its species. It almost feels like magic. The AI seems to look at the image just like we do. But here is the surprising truth: it doesn't. AI never actually sees the image the way humans do. It doesn't see a smiling face, a beautiful sunset, or a cute puppy. In fact, it doesn't see anything at all. Everything it understands begins with something much simpler—numbers.

To understand this, imagine that I hand you a family photo. You look at it for less than a second and immediately recognize your parents, your siblings, your friends, and even remember where the photo was taken. You don't stop to analyze the color of every shirt or measure the distance between people's eyes. Your brain simply understands the entire scene almost instantly. Now imagine giving that exact same photo to a computer. Would it immediately recognize your family? The answer is no. As far as the computer is concerned, that family photo is not a family photo. It is simply a huge collection of numbers stored inside its memory. There are no people, no smiles, no trees, and no memories. There are only numbers.

This idea sounds strange at first because we naturally think that computers see the world just like we do. After all, AI can recognize faces, identify diseases from medical scans, detect traffic signs for self-driving cars, and even describe photographs in great detail. If it can do all of that, surely it must be seeing the image. The reality, however, is very different. AI does not understand images because it has eyes. It understands images because it has learned how to find patterns hidden inside millions of numbers. The journey from numbers to understanding is what makes computer vision one of the most fascinating fields in artificial intelligence.

Before we learn how AI recognizes objects, we first need to understand how humans see the world. Think about the last time you saw a banana. You probably recognized it immediately. Nobody needed to tell you that it was yellow, curved, or edible. You already knew all of that because your brain has seen bananas thousands of times since you were a child. Every time you looked at one, your brain quietly learned something new. Over many years, it built an internal understanding of what a banana looks like. Now, whenever you see another banana, your brain compares it with everything it has learned before and instantly says, "That's a banana."

A computer has none of these experiences. It has never eaten a banana. It has never held one in its hand. It doesn't know that bananas are sweet or that monkeys like them. In fact, it doesn't even know what the word "banana" means. If you show a banana to a computer, it doesn't think, "This is a fruit." Instead, it receives millions of tiny pieces of information that look something like this:

255 248 120
252 240 118
249 236 110
...

Looking at those numbers, you might wonder how anyone could possibly recognize a banana. To us, they seem completely meaningless. But to a computer, those numbers are the image itself. Every single photograph, whether it shows a cat, a car, a mountain, or your family, is stored exactly like this. The challenge of computer vision is teaching machines how to find meaning hidden inside these numbers.

So where do these numbers come from? The answer begins with something called a pixel.

Every digital image is made of tiny squares called pixels. The word "pixel" simply means picture element, which is the smallest piece of a digital image. You can actually see pixels yourself. Open any photo on your phone and keep zooming in. At first, the picture becomes larger, but after enough zooming, it starts to look blurry. Eventually, you'll notice tiny colored squares. Those squares are the pixels that make up the entire image.

A simple way to think about pixels is to imagine building a picture using LEGO bricks. One LEGO brick doesn't tell you much. It could belong to a house, a spaceship, or a castle. But when thousands of LEGO bricks are connected together in the right way, they create something meaningful. Pixels work exactly the same way. One pixel by itself is almost useless. Millions of pixels working together create a photograph.

Each pixel stores color using numbers. Most digital images use something called the RGB color model, where RGB stands for Red, Green, and Blue. Every color you see on your screen is created by mixing different amounts of these three colors. Think about adjusting the volume on three different speakers. One controls red, one controls green, and one controls blue. By changing the level of each one, you can create almost any color imaginable.

For example, a pixel with the values (255, 0, 0) represents pure red because the red channel is at its maximum while the other two are turned off. A pixel with (0, 255, 0) is pure green, and (0, 0, 255) is pure blue. If all three channels are at their maximum value, (255, 255, 255), the pixel becomes white. If all three are zero, (0, 0, 0), the pixel becomes black. Every color you see on your phone, laptop, or television is simply a different combination of these three numbers.

Now imagine taking a selfie with your smartphone. A typical phone camera today captures images with a resolution of around 4000 × 3000 pixels, which is about twelve million pixels in a single photo. Since every pixel stores three numbers—one for red, one for green, and one for blue—that means your selfie actually contains more than thirty-six million numbers before any AI model even looks at it. While you see a smiling face, the computer sees a massive spreadsheet filled with millions of numerical values.

At this point, you might start seeing images in a completely different way. What we call a photograph is actually a mathematical object. It is a giant grid of numbers arranged in a specific order. The computer has no idea whether those numbers represent a person, a dog, or a sunset. All it knows is that certain patterns of numbers appear more often than others. The goal of artificial intelligence is to discover those patterns and connect them with meaningful objects.

Imagine trying to identify a song by looking only at its sound wave. At first, the wave looks like random lines moving up and down. But someone who understands audio processing can analyze those patterns and recognize the song. Computer vision works in a very similar way. AI doesn't recognize a cat because someone programmed it with a long list of rules about cats. Instead, it learns that certain patterns of numbers appear again and again in images labeled as cats. After seeing millions of examples, it slowly builds the ability to recognize similar patterns in new images it has never seen before.

This is one of the biggest differences between traditional programming and artificial intelligence. In traditional programming, developers explicitly tell the computer what to do by writing detailed rules. If you wanted to detect a square, for example, you might tell the computer to look for four equal sides and four right angles. But imagine trying to write rules for recognizing every possible dog in the world. Dogs come in different sizes, colors, breeds, poses, and lighting conditions. Writing enough rules would be nearly impossible. Instead of relying on fixed rules, AI learns directly from examples. The more images it studies, the better it becomes at recognizing new ones.

However, before an AI model can start learning these patterns, there is one important step that cannot be skipped. Images captured by different cameras are not all the same. Some are very large, others are very small. Some are bright because they were taken in sunlight, while others are dark because they were taken at night. Some are blurry, rotated, or noisy. If we feed these raw images directly into an AI model, the learning process becomes much more difficult.

Think about teaching children using books where every page has a different font size, different language, different page layout, and random ink stains. Learning would become much harder because the information is inconsistent. Before teaching begins, it would make sense to organize the books into a consistent format. AI works in exactly the same way. Before learning can happen, images are cleaned, resized, and prepared so that the model can focus on what really matters instead of being distracted by unnecessary differences.

This preparation process is called image preprocessing, and it is the first real step in teaching machines how to understand the visual world. Although it may sound like a simple cleaning process, preprocessing plays a huge role in the success of modern computer vision systems. It ensures that every image is presented in a way that makes learning easier, faster, and more accurate.

paring Images: The First Step Toward Machine Vision

Now that we know computers don't actually see images the way humans do, another interesting question comes to mind. If an image is already stored as numbers, why can't we simply give those numbers to an AI model and let it start learning immediately? The answer is simple: just because information exists doesn't mean it is ready to be used. Think about preparing for an important exam. If your notes are scattered across different notebooks, written in different handwriting, and mixed with unrelated information, studying becomes much harder. Before you even start learning, you probably organize your notes, highlight the important parts, and remove unnecessary distractions. Images are no different. Before an AI model can learn from them, they need to be prepared in a way that makes learning easier and more consistent. This preparation process is called image preprocessing, and although it happens before the AI starts learning, it has a huge impact on how well the final model performs.

Imagine that you are teaching a child to recognize different types of animals. You wouldn't give them one tiny picture, another giant poster, and a third image turned upside down while expecting them to learn equally well from all of them. Instead, you would probably make all the pictures roughly the same size so the child can focus on the animal itself rather than being distracted by differences in presentation. AI models work in exactly the same way. Images collected from the internet or taken with different cameras come in many shapes and sizes. One image might be only a few hundred pixels wide, while another could contain millions of pixels captured by a professional camera. Since neural networks expect every input to have the same dimensions, one of the very first preprocessing steps is resizing. Every image is transformed into a fixed size, such as 224 × 224 pixels, before it enters the model. Although making an image smaller removes some tiny details, the important visual patterns usually remain, allowing the AI to focus on learning instead of worrying about inconsistent image sizes.

Another important step is called normalization. At first, the word sounds technical, but the idea behind it is surprisingly simple. Earlier, we learned that every pixel stores numbers between 0 and 255. These numbers represent the intensity of red, green, and blue colors. While these values work perfectly for displaying images on a screen, they are not the most convenient values for a neural network to process. Imagine asking two students to compare their exam scores, but one exam is graded out of 100 while the other is graded out of 1,000. Comparing them directly would be confusing because they use different scales. A much better approach would be to convert both scores into percentages. AI does something very similar. Instead of working with numbers between 0 and 255, it usually divides every value by 255 so that every pixel falls between 0 and 1. This small mathematical change makes learning more stable, helps the model train faster, and allows it to find useful patterns more efficiently.

One of the smartest ideas in computer vision is something called data augmentation. Imagine you are trying to teach someone what a bicycle looks like. If you only show them one single photograph of one bicycle taken from one angle, they may struggle to recognize a different bicycle later. But if you show them bicycles from the front, the side, from above, during the day, at night, and in different colors, they quickly understand the general idea of what makes a bicycle a bicycle. AI learns in exactly the same way. Unfortunately, collecting millions of different images is expensive and time-consuming. Instead of taking new photographs, engineers create new training examples from the images they already have. A picture can be flipped horizontally, rotated slightly, zoomed in, cropped, or made brighter or darker. To us, these images still look almost identical, but to the AI, each one becomes a new learning experience. It's similar to letting someone practice the same skill in many different situations instead of always repeating the exact same exercise. This simple technique often helps AI models perform much better when they encounter images they have never seen before.

Sometimes images contain problems that have nothing to do with the objects inside them. A photo might be blurry because the camera moved while taking the picture. Another image might contain random dots caused by poor lighting or a low-quality camera sensor. Imagine trying to read a book through a dirty window. The words haven't changed, but the dirt makes them harder to see. If someone cleaned the window first, reading would become much easier. Image preprocessing works in the same way. Before the AI begins learning, engineers often remove unnecessary noise, improve the image quality, or adjust the brightness and contrast so that the important information becomes easier to recognize. The goal isn't to make the image look prettier for humans; it's to make the useful patterns clearer for the AI.

Another common preprocessing step involves changing the way colors are represented. Earlier, we learned that most images use the RGB color model, where every pixel stores separate values for red, green, and blue. However, not every computer vision task actually needs color information. Imagine building an AI system that reads handwritten numbers on a piece of paper. Whether the paper is white, yellow, or light blue usually doesn't matter. The important part is the shape of the handwritten numbers. In situations like this, engineers often convert the image into grayscale, where each pixel stores only a single brightness value instead of three different color values. This makes the image much smaller, reduces the amount of computation the AI needs to perform, and allows the model to focus entirely on shapes and patterns rather than colors. On the other hand, if the task is identifying different kinds of flowers or traffic lights, color becomes extremely important, so the original RGB image is usually kept. This shows that preprocessing isn't a fixed set of rules. Instead, it depends on the problem the AI is trying to solve.

By now, every image has been resized, cleaned, normalized, and prepared for learning. At this point, you might imagine that the AI is finally ready to recognize cats, cars, people, and buildings. Surprisingly, that wasn't how the earliest computer vision systems worked. Modern AI models can learn directly from images, but computers weren't always this intelligent. Long before deep learning became popular, engineers had to teach computers how to recognize visual patterns manually. Imagine asking someone who has never seen a house before to find one in a drawing. Instead of saying, "That's a house," you might tell them to look for straight lines, sharp corners, rectangles that look like walls, and triangles that could be roofs. Early computer vision systems worked exactly like this. Instead of learning automatically from millions of examples, they relied on carefully designed rules written by human engineers. These systems searched for simple visual clues such as edges, corners, and textures, hoping that combining enough of these small pieces would eventually allow the computer to recognize complete objects.

Although these early techniques may seem old-fashioned compared to today's powerful AI models, they completely changed the field of computer vision. They taught researchers one very important lesson: before a computer can understand an object, it first needs to understand its smallest visual building blocks. Many of the ideas developed during this period still influence modern computer vision today, even though deep learning has largely replaced handcrafted rules. Understanding this history helps us appreciate how far AI has come over the past few decades.

Before AI Could Recognize Objects, It Learned to Find Edges

After images have been cleaned, resized, and prepared, you might think the computer is finally ready to recognize a cat, a car, or a person's face. Today, that is almost true because modern AI models can learn directly from millions of images. But if we go back twenty or thirty years, computers were nowhere near that smart. They didn't know what a cat looked like. They didn't understand the difference between a person and a tree. They couldn't even recognize a simple cup sitting on a table. In fact, one of the biggest challenges in the early days of computer vision was teaching a computer where to start looking. Before a computer could identify an object, it first needed to understand the tiny visual details that make up that object. Just as children learn the alphabet before reading books, computers first had to learn the smallest building blocks of an image before they could understand the whole picture.

Imagine asking a young child to draw a house. Even if they have never taken an art class, they usually begin with a square for the walls, a triangle for the roof, a rectangle for the door, and a few smaller squares for the windows. They don't think about millions of tiny colors or pixels. Instead, they focus on the important shapes that define a house. Early computer vision worked in almost exactly the same way. Researchers believed that if a computer could detect simple shapes like lines, edges, and corners, it could combine those pieces together to recognize larger objects. This idea became the foundation of traditional computer vision.

To understand why edges are so important, look around the room you're sitting in right now. Every object you see is separated from its surroundings by edges. Your laptop has edges. Your phone has edges. The table you're sitting at has edges. Even your face is defined by edges where your hair meets your forehead or where your eyes meet your skin. Without edges, the world would look like one giant blur with no clear boundaries between objects.

Imagine taking a black marker and tracing the outline of every object in a photograph. Even after removing all the colors, you could probably still recognize most of the objects because their outlines remain. That's because our brains rely heavily on edges to understand shapes. Researchers realized that if humans naturally depend on edges, perhaps computers should begin there as well.

This idea led to one of the earliest techniques in computer vision called edge detection. Instead of trying to understand the entire image, edge detection simply asks one question: Where does the image suddenly change? Remember that every image is made of pixels, and every pixel stores a number representing its color or brightness. If two neighboring pixels have almost the same value, they probably belong to the same object. But if one pixel is very bright and the next one is much darker, something important is happening. That sudden change usually marks the boundary between two different objects.

Imagine walking from a sandy beach into the ocean. As long as you're walking on sand, everything feels the same beneath your feet. But the moment your feet touch the water, you immediately notice a change. Edge detection works similarly. It looks for places where the image changes suddenly instead of staying the same. Those sudden changes become the edges of objects.

One of the first successful algorithms designed for this purpose was the Sobel Operator. Although the name sounds complicated, the basic idea is surprisingly simple. Imagine looking at only a tiny part of an image, perhaps just three pixels by three pixels. The Sobel Operator compares neighboring pixels to measure how quickly their brightness changes. If the difference is very small, there is probably no edge. But if the brightness changes dramatically within a short distance, the algorithm concludes that it has found an edge.

Think of driving along a perfectly flat road. The journey feels smooth because the road changes very little. Now imagine suddenly reaching the edge of a cliff. The change is dramatic and impossible to miss. The Sobel Operator searches for those "cliffs" in an image—not physical cliffs, but places where pixel values change sharply. These locations usually correspond to the outlines of real objects.

The result of applying the Sobel Operator is quite interesting. Instead of producing another colorful photograph, it creates an image where only the important edges remain visible. Large areas with similar colors almost disappear because they don't contain significant changes. What remains is a simple sketch showing the outlines of objects. Although this sketch contains far less information than the original photograph, it often captures the most important structural details.

However, the Sobel Operator had a weakness. Real-world images are rarely perfect. Photos often contain small lighting changes, camera noise, shadows, and tiny texture differences. Unfortunately, the Sobel Operator treats many of these small changes as edges even though they aren't meaningful. Imagine trying to hear someone speaking in a crowded café. Along with the person's voice, you also hear people talking at nearby tables, coffee machines making noise, and chairs moving across the floor. Separating the important sound from all the background noise becomes difficult. The Sobel Operator faced a similar problem. It detected real object boundaries, but it also detected many unnecessary edges caused by random variations in the image.

Researchers realized they needed a smarter approach, which led to one of the most famous algorithms in computer vision: Canny Edge Detection.

You can think of the Canny algorithm as a much more careful artist. Instead of immediately drawing every edge it finds, it first cleans the image to remove small amounts of noise. Next, it searches for places where brightness changes significantly. Then it removes weak edges that are probably unimportant while keeping only the strongest and most reliable ones. Finally, it connects broken edge segments together so that object boundaries appear complete rather than fragmented.

Imagine trying to draw the outline of a mountain from a photograph. A beginner might trace every tiny rock, shadow, and crack, creating a messy drawing full of unnecessary details. An experienced artist, however, ignores the small distractions and focuses only on the important outlines that define the mountain's shape. The Canny algorithm behaves like that experienced artist. It filters out distractions and keeps only the edges that truly matter.

This made Canny Edge Detection far more reliable than earlier methods. Even today, decades after it was introduced, many computer vision systems still use Canny whenever accurate edge detection is needed. It remains one of the classic algorithms that every computer vision student learns because it demonstrates how much information can be extracted from something as simple as changes in brightness.

Although edge detection was a huge breakthrough, researchers soon discovered another limitation. Imagine trying to recognize a human face using only its outline. You might see the shape of the head, but you would miss many important details like the eyes, eyebrows, nose, and mouth. Two completely different people could have similar head shapes but very different facial features. Edges alone simply weren't enough.

Researchers began asking a new question. If edges describe the boundaries of objects, what describes the important points inside those objects?

This question led to the idea of detecting corners and interest points. Corners are special locations where two or more edges meet, such as the corner of a window, the tip of a triangle, or the intersection of two roads in a satellite image. Unlike long straight edges, corners tend to remain recognizable even when the image changes slightly. They act like landmarks that help computers identify important parts of an object.

Imagine visiting a city you've never been to before. You probably won't remember every street or every building. Instead, you'll remember famous landmarks—a tall tower, a large bridge, or a unique monument. Later, when you return, those landmarks help you find your way around. Corners serve a similar purpose in computer vision. They become visual landmarks that help the computer recognize and compare different images.

This idea opened the door to even more advanced techniques that could describe not only where important points existed but also what made those points unique. Instead of simply detecting edges, researchers began designing algorithms capable of describing textures, patterns, and local image details in ways that remained stable even if the image was rotated, resized, or viewed from a different angle.

These breakthroughs eventually produced some of the most influential algorithms in the history of computer vision, including HOG (Histogram of Oriented Gradients) and SIFT (Scale-Invariant Feature Transform). These algorithms allowed computers to recognize objects much more reliably than edge detection alone and became the foundation of countless vision systems used throughout the early 2000s.

However, despite their remarkable success, they all shared one important limitation. Every rule, every feature, and every algorithm had been carefully designed by human engineers. The computer wasn't truly learning; it was simply following instructions written by people. As images became more complex and new problems appeared, researchers realized something revolutionary: instead of teaching computers what features to look for, what if computers could discover those features by themselves?

Teaching Computers to Learn Instead of Following Rules

By the early 2000s, computer vision had come a long way. Computers could detect edges, identify corners, describe textures, and even recognize some objects using clever algorithms like HOG and SIFT. At the time, these techniques were considered revolutionary because they allowed machines to understand images far better than before. However, researchers kept running into the same problem. Every time they wanted the computer to recognize something new, they had to design new rules by hand. If they wanted the computer to detect faces, they created one set of rules. If they wanted it to recognize cars, they wrote another set of rules. If they wanted to identify animals, they often had to start all over again. It was like teaching a student every single answer instead of teaching them how to think.

Imagine hiring a new employee at a company. Every morning, you tell them exactly what to do. You explain every small task, every decision, and every possible situation they might face. At first, this works. But as the company grows, the employee keeps asking more questions because they only know the rules you taught them. Now imagine another employee who learns by watching examples instead. After seeing hundreds of situations, they begin making good decisions on their own without waiting for instructions. Which employee would become more useful over time? Most people would choose the second one. Researchers wanted computers to become like that second employee. Instead of following thousands of handwritten rules, they wanted computers to learn directly from experience.

This idea gave birth to one of the biggest revolutions in artificial intelligence: Deep Learning.

Instead of telling a computer exactly what to look for, researchers began asking a completely different question. What if we simply showed the computer millions of images and allowed it to discover the important patterns by itself? Rather than programming features like edges, corners, and textures manually, the computer would learn those features automatically. This idea changed everything because it transformed computers from rule-following machines into learning machines.

To understand how this works, let's imagine teaching a child what a cat looks like. You probably wouldn't start by saying, "A cat has triangular ears exactly 7 centimeters apart, two eyes with a certain distance between them, and whiskers extending at a specific angle." That would be an impossible way to learn. Instead, you simply show the child many different cats. Some are black, some are white, some are fluffy, and others have short fur. After seeing enough examples, the child naturally begins recognizing cats without memorizing a long list of rules.

Deep learning works in exactly the same way.

Instead of writing instructions, we provide thousands—or even millions—of labeled images. One image says "cat." Another says "dog." Another says "car." At first, the computer has absolutely no idea what these labels mean. It simply makes random guesses. Most of those guesses are completely wrong. But after every mistake, the computer adjusts itself slightly. Then it tries again. And again. And again. After repeating this process millions of times, it slowly becomes better at recognizing patterns that separate one object from another.

The system responsible for this learning process is called an Artificial Neural Network.

The name comes from the human brain, although the two are very different. Your brain contains billions of neurons connected together. Each neuron receives information, processes it, and passes it to other neurons. Artificial neural networks use a similar idea. Instead of biological neurons, they use mathematical units called artificial neurons. Each one receives numbers, performs a simple calculation, and sends the result to the next layer. Individually, these neurons are incredibly simple. But when millions of them work together, they become capable of solving surprisingly complex problems.

Imagine a factory that builds cars. At the first station, workers install the wheels. At the next station, another group installs the engine. Later, another team paints the car, and finally, someone inspects the finished product. No single worker builds the entire car. Instead, each worker performs one small task before passing the product to the next station. Neural networks operate in much the same way. Information moves through several layers, and each layer performs a small piece of the overall job.

When an image enters a neural network, it doesn't magically become a prediction. Instead, it passes through many layers of calculations. The first layer receives the raw pixel values. The next layer processes those values and passes its results forward. Every new layer learns slightly more complicated patterns than the one before it. Eventually, after passing through many layers, the network produces its final prediction.

For example, imagine showing the network a picture of a cat. At the beginning, the first layer doesn't know it's looking at a cat. It simply notices tiny changes in brightness and color. The next layer begins combining those small changes into simple shapes like lines and curves. A deeper layer starts recognizing fur textures, eyes, and ears. Even deeper layers begin combining these features together until the network eventually realizes, "This collection of patterns most likely represents a cat."

This gradual learning process is one of the reasons deep learning became so successful. Instead of relying on engineers to design features manually, the network learns its own features directly from data.

But another challenge quickly appeared.

Remember that a modern smartphone photo contains more than thirty-six million numbers. If we connect every single pixel directly to every neuron in a neural network, the number of calculations becomes enormous. Training would require an unbelievable amount of memory and computing power. Even today's most powerful computers would struggle to process high-resolution images using this approach.

Researchers needed a smarter solution.

The answer came from a very simple observation about how humans look at the world.

Imagine walking into a room you've never seen before. Do you examine every single object at exactly the same moment?

Probably not.

Your eyes naturally focus on one small area at a time. You might first notice the door. Then you look at the table. Next, you notice a chair, a lamp, and finally a bookshelf. Your brain gradually combines these local observations into one complete understanding of the room.

Researchers realized that computers should do something similar.

Instead of analyzing the entire image all at once, what if the computer looked at small parts of the image one by one?

This idea led to one of the most important inventions in the history of artificial intelligence: the Convolutional Neural Network, or CNN.

A CNN doesn't begin by looking at the entire image. Instead, it uses a tiny window called a filter (sometimes called a kernel). You can imagine this filter as a small magnifying glass that slides across the image. At every position, it looks at only a tiny group of neighboring pixels before moving to the next location.

Think about reading a long book.

You don't read every page simultaneously.

Your eyes move across one sentence, then another, then another until you've finished the chapter.

A CNN processes images in almost the same way. It examines one small area, learns what it finds there, and then moves on to the next area.

At first, these filters don't know what they're looking for. They're simply collections of random numbers. But during training, the network slowly adjusts these numbers until each filter becomes useful. One filter might become excellent at detecting horizontal edges. Another might specialize in vertical edges. Another could recognize curves. Others may learn to identify textures, circles, or repeated patterns.

The remarkable part is that nobody tells the filters what to learn.

The network discovers these features completely on its own simply by trying to improve its predictions.

Imagine giving ten students the same box of puzzle pieces and asking them to build a picture. At first, everyone struggles because the pieces seem random. But after enough practice, each student starts noticing useful clues. One student learns to search for blue sky pieces first. Another focuses on corners. Someone else looks for pieces with straight edges. Nobody explicitly taught them these strategies—they developed them through experience.

CNNs learn in exactly the same way.

As the image passes through many convolutional layers, the network builds a deeper understanding of what it sees. The first layers recognize simple patterns like edges. The middle layers combine edges into textures and shapes. The deeper layers begin recognizing object parts like wheels, windows, noses, or eyes. Finally, the last layers combine everything together to identify complete objects.

This process is surprisingly similar to how humans learn.

Imagine you've never seen an elephant before. The first time someone shows you one, you notice its enormous ears. Then you notice its long trunk. Later, you recognize its thick legs and massive body. Eventually, your brain combines all these features into one complete concept called "elephant." The next time you see one, recognition becomes almost instant.

CNNs build knowledge in exactly this gradual way.

By learning simple patterns first and combining them into more complex ones, they become incredibly good at understanding images.

This breakthrough completely transformed computer vision. Tasks that once required thousands of carefully designed rules could now be solved simply by showing the network enough examples. Instead of engineers deciding what features mattered, the AI learned the most useful features automatically.

However, one question still remained.

Once the network has learned thousands of different visual features, how does it decide which ones are actually important? How does it ignore unnecessary details while focusing on the patterns that matter most? And how does it finally turn all these learned features into the single answer we see on the screen?

To answer these questions, we need to explore two more important ideas that make CNNs so powerful: activation functions and pooling layers. These components allow neural networks to focus on meaningful information, reduce unnecessary complexity, and eventually recognize objects with incredible accuracy.

How AI Finally Understands What It Is Looking At

By now, our AI model has learned something incredible. It no longer looks at an image as millions of random numbers. Instead, it has learned to recognize simple patterns like edges, curves, textures, and shapes. As the image moves through more convolutional layers, these simple patterns are combined into more meaningful ones. A few edges become an eye. Several textures become fur. Different shapes combine to form a face. Eventually, the network begins recognizing complete objects like cats, cars, people, and buildings. But there is still one important question left to answer. If the network has learned thousands of different features, how does it know which ones actually matter? After all, not every detail inside an image is equally important. Some information helps the AI make the correct decision, while other information only distracts it.

Imagine that you are trying to recognize one of your friends in a crowded shopping mall. There might be hundreds of people around you, colorful advertisements hanging on the walls, music playing in the background, and dozens of stores competing for your attention. Even though there is so much happening, your brain somehow ignores almost everything except the features that matter. You notice your friend's hairstyle, their face, the color of their jacket, or the way they walk. You don't waste time studying the pattern of the floor tiles or counting the lights on the ceiling. Your brain naturally filters out unnecessary information and focuses only on what is useful.

Artificial intelligence faces exactly the same challenge. Every image contains a huge amount of information, but only a small portion of it is actually useful for recognizing the object. If an AI model tried to remember every single pixel with the same importance, it would become extremely slow and inefficient. Instead, it needs a way to keep the important information while removing details that don't really matter. This is where one of the smartest ideas in Convolutional Neural Networks comes into play: pooling.

AI for Inclusion: Translating Makaton into English

Mon, 20 Oct 2025 23:16:03 +0000

In an age where artificial intelligence (AI) continues to transform how we communicate, learn, and live, one of the most meaningful applications of this technology lies in accessibility, particularly in helping individuals with communication challenges express themselves more easily.

Among these individuals are users of Makaton, a unique language programme that uses signs and symbols alongside speech to support communication. It is widely used by people with learning or communication difficulties, as well as by their families, teachers, and therapists.

While digital tools for language translation are increasingly advanced, there remains a noticeable gap when it comes to translating visual communication systems like Makaton into spoken or written English. This is where AI-driven translation systems can make a remarkable difference.

The Challenge: Translating Beyond Words

Traditional translation systems focus on text or speech. Makaton, however, is a multimodal language which involves visual signs, symbols, and gestures that convey meaning together. Translating such a system requires more than just mapping symbols to words; it demands understanding visual cues, context, and intent.

Many existing solutions focus only on sign languages such as British Sign Language (BSL), leaving Makaton which combines signs and symbols with speech, relatively underexplored in the AI domain. As a result, Makaton users often rely on human interpreters or educators to facilitate communication, limiting their independence and digital participation.

The Innovation: AI for Makaton-to-English Translation

My research focuses on developing an AI-based system that automatically translates Makaton into English using a combination of computer vision, natural language processing (NLP), and machine learning techniques.

The goal is to create a system capable of recognizing Makaton signs or symbols from images or videos and translating them into meaningful English sentences.

The process involves:

Computer Vision: Detecting and classifying Makaton gestures and
symbols.
NLP Models: Converting recognized signs into coherent English
sentences.
Context Learning: Understanding surrounding cues (e.g., gesture
sequence or emotion) for more accurate translation.

This combination moves beyond static recognition toward context-aware translation, making communication smoother and more natural.

Early Testing and Educational Impact

I have had the opportunity to test early prototypes with students at Derby Cathedral School, While these students are not Makaton users, their feedback helped refine the system’s user interface, accessibility, and accuracy.

The vision is to integrate such systems into inclusive classroom technologies, where Makaton users can communicate more freely with peers and teachers, promoting equal participation in learning activities.

AI for Inclusion: Why It Matters

This work aligns closely with the UN Sustainable Development Goal 4 (Quality Education) and Goal 10 (Reduced Inequalities). AI technologies should not only make systems faster or smarter, they should make them fairer, more inclusive, and human-centered.

By translating Makaton into English, AI can help reduce barriers to communication, support inclusive education, and empower individuals who rely on alternative communication methods.

The Future of AI and Accessibility

Looking ahead, the potential extends beyond Makaton translation. The same technology can be adapted for:

Translating other sign-based systems.
Supporting speech therapy and early language development.
Powering assistive communication apps for non-verbal users.

AI’s promise lies not just in automation but in amplifying human expression, especially for those who have long been underserved by traditional technology.

Conclusion

Bridging the communication gap for Makaton users is not just a technical challenge, it’s a human one. By leveraging AI thoughtfully, we can design systems that empower, include, and uplift.

Innovation in AI should go hand in hand with compassion, ensuring that as we build smarter technologies, we also build a more connected and inclusive world.