I Built My Own Personal AI Assistant — Meet IRIS

Wed, 07 Oct 2026 06:52:43 +0000

I Built My Own Personal AI Assistant — Meet IRIS

I've always wanted to build my own personal assistant.

Not just another chatbot that answers questions, but something that could actually live on my computer and help me get things done.

So I started building IRIS.

What started as an idea slowly turned into a real desktop AI assistant running locally on my Mac.

IRIS can now chat with me, use voice, work with files, search the web, remember information, understand screenshots, control applications, plan tasks, and interact with my computer.

And honestly, getting it to this point wasn't exactly straightforward.

There were plenty of bugs, broken implementations, slow models, permission issues, packaging problems, and a lot of moments where something simply refused to work. 😂

But that's also what made building it interesting.

What exactly is IRIS?

IRIS is my attempt at building a personal AI assistant, rather than just another conversational AI application.

The idea is simple:

Instead of me adapting to a bunch of different tools, I want one assistant that can work with those tools for me.

I can talk to IRIS normally, and depending on the request, it can decide what it needs to do.

For example, a request might involve:

User
   ↓
Understand request
   ↓
Determine what needs to happen
   ↓
Select the appropriate tool
   ↓
Check permissions
   ↓
Execute the action
   ↓
Observe the result
   ↓
Respond to the user

That's the direction I wanted IRIS to move toward from the beginning.

What can IRIS do?

💬 Natural conversations

At the core of IRIS is a local language model running through Ollama.

I'm currently using Qwen 2.5 3B for the conversational side.

This gives IRIS its ability to understand requests, answer questions, reason about tasks, and interact naturally.

The important part for me was keeping the core assistant local, rather than building everything around a cloud API.

🎙️ Voice interaction

I didn't want IRIS to feel like an application where I always have to type.

It can accept voice input through the Mac's microphone and convert speech into text.

The flow is:

My voice
   ↓
Microphone
   ↓
Speech recognition
   ↓
IRIS
   ↓
Response

IRIS can also speak its responses back using the system's voice output.

There's even an audio toggle in the interface, so I can choose whether IRIS responds verbally.

📁 Working with files

IRIS can interact with files on the computer.

For example, it can:

Search for files
Read files
Create files
Edit files

For actions that can modify the filesystem, IRIS has a permission/confirmation layer rather than blindly executing everything.

That's important because giving an AI access to your computer without any safeguards is obviously not something I wanted to do.

🌐 Web search and browsing

IRIS can also search the web and retrieve information.

The system can:

Search
   ↓
Find relevant results
   ↓
Retrieve useful content
   ↓
Process the information
   ↓
Generate an answer

This allows IRIS to go beyond the information available inside its local model.

🖥️ Application control

One of the things I really wanted was the ability to tell my assistant to interact with applications.

IRIS can launch supported applications on my Mac.

For example:

"Open Calculator."

And IRIS can actually open it.

This might sound simple, but it's one of the steps toward making an AI assistant something that can do things, rather than simply tell you how to do them.

🧠 Memory

Another major part of IRIS is memory.

I didn't want every conversation to feel like starting from zero.

IRIS has different layers for things such as:

Working memory
Long-term memory
Personal profile
Conversation state
Memory retrieval
Memory updates
Memory deletion

The idea is that information can be stored and later retrieved when it's relevant.

For example:

"Remember that this project is called IRIS."

Later:

"What was the project I told you about?"

The assistant should be able to connect those two conversations.

👁️ Vision

IRIS can also understand screenshots.

For this, I integrated a local vision model:

Granite 3.2 Vision 2B

The workflow is:

Screenshot
    ↓
Image processing
    ↓
Vision model
    ↓
IRIS understands the screen
    ↓
Response

This opens up a completely different direction for the project.

Instead of only understanding text, IRIS can start understanding what is actually happening on the screen.

🤖 Planning and autonomous tasks

Another major part of IRIS is task planning.

For more complicated requests, the assistant can break a goal into smaller steps.

For example:

Goal
 ↓
Task 1
 ↓
Task 2
 ↓
Task 3
 ↓
Observe results
 ↓
Recover from failures
 ↓
Complete task

I built components for:

Planning
Task decomposition
Multi-step execution
Observation
Recovery
Task state
Goal tracking
Agent memory
Evaluation

The goal is to eventually move from:

"Tell me how to do this."

to:

"Do this for me."

🔐 Permissions and safety

This was something I didn't want to ignore.

An AI assistant with access to your computer needs boundaries.

IRIS has a permission system that distinguishes between different types of actions.

Some actions can happen directly, while potentially risky actions can require confirmation.

For example, filesystem and terminal operations can require permission before execution.

The idea is simple:

IRIS should be capable, but it shouldn't blindly do everything it can.

🧩 Plugins

I also added a plugin system so IRIS doesn't have to remain limited to the tools I originally built.

Plugins can provide additional capabilities while following a common interface.

That means the architecture can eventually grow from:

IRIS
 ├── Files
 ├── Web
 ├── Applications
 ├── System
 └── Memory

into something more extensible:

IRIS
 ├── Core tools
 ├── Plugins
 ├── External capabilities
 └── Future integrations

🏗️ How IRIS is structured

The project has grown into multiple layers rather than being one giant Python file.

At a high level:

                    IRIS
                     │
             ┌───────┴───────┐
             │               │
          Interface         Brain
             │               │
       ┌─────┴─────┐    ┌────┴─────┐
       │           │    │          │
      Chat       Voice  Memory   Planning
                             
                     │
                Tool Layer
                     │
       ┌─────────────┼─────────────┐
       │             │             │
    Files          Web          System
       │             │             │
 Applications     Search       Computer
                     │
                   Vision

The exact implementation is more detailed, but this is the basic idea.

The interface talks to the assistant engine, the engine can use memory and planning, and the tool layer gives IRIS the ability to interact with the computer and the outside world.

🛠️ Tech stack

Some of the main technologies behind IRIS are:

Python
PySide6 — desktop interface
Ollama — local model runtime
Qwen 2.5 3B — local language model
Granite 3.2 Vision 2B — vision
Nomic Embed — embeddings
PyTorch
PyInstaller — macOS application packaging

The project currently runs as a proper macOS application, so I don't need to open VS Code and run a Python command every time I want to use it.

I can simply launch IRIS like a normal application.

😅 The difficult part wasn't always the AI

One thing I learned while building IRIS is that making an AI assistant isn't just about connecting an LLM to a chat box.

A lot of the work was actually around the systems surrounding the model.

Things like:

Voice input

Getting the microphone, speech detection, silence detection and packaged application permissions working correctly.

Memory

Making sure memories can actually be stored, retrieved, updated and forgotten.

Tool execution

Making sure the assistant can select and execute the correct tool without breaking everything around it.

Vision

Getting screenshot analysis fast enough to actually be useful.

Permissions

Allowing IRIS to interact with the computer without giving it unlimited freedom.

Packaging

Turning a Python project with all these dependencies into an actual Mac application was its own challenge.

Those problems taught me something important:

Building an AI product is much more than building the model.

The model is only one part of the system.

🚀 What's next?

I'm not considering IRIS finished.

There are still a lot of things I want to improve.

The next direction is focused less on simply adding more features and more on making IRIS actually smarter and more useful.

Some of the things I want to work on:

Better tool selection
Better understanding of natural language
Stronger conversational context
More reliable memory
Better autonomous task execution
Improved computer control
Better screen understanding
More robust error recovery
Faster responses
More integrations

Eventually, I want to be able to give IRIS a goal and have it figure out the steps itself while keeping me in control of important actions.

Final thoughts

IRIS started because I've always wanted to build my own personal assistant.

It could have stayed as an idea.

Instead, I decided to actually build it.

There were plenty of times when things broke, features had to be rebuilt, and the project became much more complicated than I initially expected.

But that's probably my favorite part of the project.

I'm not just experimenting with an AI model anymore.

I'm building a system around it.

Something that can see, listen, remember, reason, use tools and interact with my computer.

It's still far from the assistant I imagine in my head.

But now I can actually open my Mac, launch IRIS, talk to it and watch that idea come to life.

And honestly, I'm just getting started. 🚀

Coder Legion - Recent posts tagged personal-ai

I Built My Own Personal AI Assistant — Meet IRIS