Home control: Sonos, Hue, Apple TV and more, working together

R&D notes

Doesn't Siri or Alexa already do this?

Partly. Siri, Alexa and Google Assistant can switch Hue lights on and play music on Sonos, and a smart speaker hears you from across the room better than a phone does. They're good at single commands and routines set up in advance. This does whole jobs, and checks them:

• It finishes the job. "Put episode 4 of Bluey on" switches the TV on, moves the HDMI switch to the Apple TV, opens Disney+, steps through the menu to the episode, and dims the lights if it's dark.
• It checks the result. It reads back what the Apple TV is playing, which episode Disney+ is on and what sound the soundbar is receiving, and says plainly when something didn't work. "The TV sound's gone" works out what's wrong and tries the fixes, rather than getting an apology.
• It includes the awkward equipment. A 2015 TV with no remote, an HDMI switch, three games consoles and the router's parental controls answer to the same requests as the lights and speakers.
• It stays in the house. The equipment is controlled over the home network rather than through each maker's cloud, and simple requests are worked out on the phone itself.

How it started

It began when the Apple TV remote went missing. The Apple TV can be controlled over the home network, so we put a remote in a web page. We added the Sonos speakers next, then the Hue lights, then the router's parental controls, which were already running on the same Mac mini. The apps came once the web page had settled into a set of controls worth having in a pocket.

The apps

Everything goes through one set of controls on the Mac mini, so each new way in can do the same things.

The iPhone app, here showing its built-in sample house.

A phone is paired by scanning a code on the web app, and each phone or Watch gets its own key, which can be cancelled on its own if a device is lost. Requests from outside the house are authenticated, and the router controls can't be reached from outside at all.

What it controls

The web app: music playing on the living-room Sonos, with the remote and each room's volume.

The app talks to each device directly on the home network rather than through the manufacturers' cloud services. Responses are quicker, and it carries on working if one of those services is down.

The TV and games consoles

The living room has a 2015 TV whose remote went missing, an HDMI switch, and an Apple TV, a PlayStation 5, an Xbox, a Steam Machine and a Mac all plugged into it. None of the TV, the switch or the Xbox can be controlled over the network, so a small Wi-Fi infrared transmitter does it instead. It learned the switch's codes from its own remote, and the TV's and Xbox's come from published code sets. The PlayStation is woken, put to rest and given controller presses over the home network, and the Steam Machine, which runs Linux, is controlled directly.

Together they make a universal remote, in the web app and the iPhone app. There's a tab for each device. Choosing one switches the TV to it and shows a pad laid out like that device's controller, with the TV's power, input and settings menu underneath. For the Steam Machine it shows the installed games instead: tapping one turns the TV on, switches to it, wakes the Steam Machine over Wi-Fi if it's asleep, and starts the game. Requests work too: "switch to the PlayStation", "play Firewatch" or "turn the TV off". "Goodnight" switches the TV off.

The web app's remote for the Xbox, with the TV's settings menu open, and for the PlayStation 5.

The same remotes in the iPhone app, in light and dark mode, here with the app's sample house.

How the Steam Machine got 5.1 sound through a 2015 TV

The Steam Machine needed a different fix. Games on it play 5.1 surround sound, but the 2015 TV only passes surround on to the soundbar as Dolby Digital, which the Steam Machine wouldn't send, so everything arrived in stereo. It now encodes the game's sound into Dolby Digital as it plays, with under a fifth of a second of delay. Each time it starts it checks what the TV accepts, so with a newer TV it would step aside and send surround directly.

Moods

A mood is a saved set of lights, music and TV. "Washing up" turns the kitchen lights to a colour scene and starts a radio station on the nearest speaker, then pauses the music once the kitchen's motion sensor has seen nobody for ten minutes. "Goodnight" turns every light off, pauses every speaker, sleeps the TV, and says which rooms were left on and how warm the house is. Moods are edited in the web app and appear in the iPhone and Watch apps and in Siri straight away.

Getting someone's attention

This one came from being locked out while someone inside had headphones on. One button plays a message on chosen speakers, flashes the lights in a chosen room, and can pause the children's internet for five minutes. Each part also works on its own. A message can be typed, and read out by a computer voice, or recorded on the phone or Watch. Whatever the speaker was playing carries on afterwards.

House status

The motion sensors also measure temperature and light, so the house can say which rooms have had movement recently, which are cold, and what's on. A note appears once when a sensor's battery is nearly flat, when a sensor stops responding, or when a room drops below a chosen temperature, and not again until something changes.

Typed and spoken requests

A box at the top of the web app takes requests such as "dim the living room to 30% and put the film on" or "play the new Bicep album in spatial in the living room". A language model turns each one into a list of steps, using the actual names of the rooms, scenes and apps in the house. Music is looked up in Apple Music's catalogue, and when spatial audio is asked for it picks the Dolby Atmos version if there is one, or says there isn't.

Other requests it handles include "put Bluey on Disney+", "play Orbital in the back room at 20%", "film mode for two hours", "snooze all the sensors until morning", "make a playlist of 90s house" (saved to Apple Music) and "set the mood for washing up". It keeps the last few requests, so "try again" (after logging in to an app, for example) and "a bit louder" work. If part of a request can't be done, such as a room with no lights in it, the reply says so. In the iPhone app, the screen then moves to what changed: the player for that speaker, the remote, or that room's lights.

How a request is handled

Every way in, from the apps and the web app to Siri, ends up at the same set of controls on a Mac mini in the house, and only that talks to the equipment. Where a language model is involved, it only plans: it returns a list of steps from a fixed set, and ordinary code carries them out and checks what happened.

For example, "put episode 4 of Bluey on":

1. The phone turns the speech into text. It isn't a simple request, so it goes to the Mac mini.
2. The cloud model gets the request with the names of the apps and rooms, and returns one step: play Bluey, episode 4, in Disney+, with the show's link.
3. The Mac mini switches the TV on and the HDMI switch to the Apple TV by infrared, then opens the link.
4. Disney+ opens the show, not the episode, so it moves through the menu with remote presses, then reads which episode Disney+ says is playing and corrects it if it's wrong.
5. After dark, film mode dims the living room. The reply says what was done, and anything that couldn't be.

More on how it's built

• On the Mac mini, two small services: one serves the web app and handles logins and each device's key, the other talks to the equipment. They're written in TypeScript and Python.
• The equipment: Sonos and Hue through their own interfaces on the home network; the Apple TV, router and PlayStation through interfaces their makers don't publish, using open-source libraries; the TV, HDMI switch and Xbox by infrared; and the Steam Machine directly, as it runs Linux.
• The apps are native iPhone and Apple Watch apps that share their code. Siri, the Action button and the Lock Screen controls all use the same actions, so each one is written once.
• Checking: each step has to name a real room, scene, app or device, and a step that fails is reported while the others still run. Where the equipment can say what it's doing (what the Apple TV is playing, which episode Disney+ is on, what format the soundbar is receiving), the result is read back rather than assumed.
• Keeping screens current: the apps keep a request open to the Mac mini, which answers as soon as it sees a change, rather than the app asking every few seconds.
• From outside the house, requests come in through an authenticated tunnel, so no ports are open on the router, and each phone or Watch has its own key.

Playlists from listening data

Playlists were the weakest part at first. Asked for something in the feel of a particular song, the language model picked from memory: the same few well-known tracks, grouped artist by artist. Now its choice is only a first draft. The system looks up the songs people actually listen to alongside those tracks on Last.fm, and a larger model chooses 20 to 25 from around 160 suggestions, leaving out what's been played a lot recently. The songs are then ordered so the same artist doesn't come round twice in a row, and each one is checked against Apple Music's catalogue before it's saved.

A playlist in the feel of UNKLE's "Looking for the Rain" went from two artists to fourteen. It takes a minute or so longer, which is a fair trade for a playlist. Only artist and song names go to Last.fm.

Local first: what runs where, and why

Where a model that runs on the device can do the job, we prefer it, and this project is a case in point. Most of the system doesn't use a language model at all: the lights, speakers, TV and router are controlled directly on the home network. In the iPhone app, speech is turned into text on the phone, and simple requests, such as "kitchen lights to 40%", "a bit louder" or a mood by name, are worked out by the small language model built into the iPhone and carried out straight away, without the request leaving the house.

What the cloud model sees, and how the local model is tested

Requests that need wider knowledge, such as finding where a film is streaming or choosing tracks for a playlist, go to a cloud model (Claude, from Anthropic). It receives the request, the names of the rooms, scenes, apps and moods, and the last few requests. Which room someone is in, worked out from the motion sensors, stays on the Mac mini; only a request like "play something in here" sends that room's name. The cloud model doesn't receive any audio and can't reach the devices: it returns a list of steps from a fixed set of actions, which the Mac mini carries out.

Recent Macs include a similar on-device model, so the Mac mini also plans each of those requests locally in the background and the two plans are compared. So far the local model gets simple requests right in a second or two, but makes mistakes on longer ones that mix lights and music, for example mixing up the Sonos and Hue names for the same room. Both plans are kept for every request, and each type of request moves over to the local model once that record shows it can be trusted with it. The on-device models also improve with each Apple update, so the share of requests going to the cloud should keep shrinking.

This matters more in a home than almost anywhere. Which room someone is in, when the children's devices are online and what people ask for are all things a family might reasonably want kept in the house. Local models also answer faster, carry on working when the internet is down, and cost nothing per request. The cloud model is there for what the local ones can't do yet.

Film mode, and making the Apple TV behave

When a film or show starts on the Apple TV after dark, the living room lights go off for a film or down to their lowest for a TV episode, and the motion sensors stop switching lights on. When it's paused or stopped, the lights go back to how they were. If film mode has been started by hand with a timer, the automatic version leaves it alone.

Streaming apps on the Apple TV don't all behave the same way, and the HDMI sound link to the soundbar sometimes drops. Two of the fixes:

Disney+ links open the show, not the episode (and iPlayer won't take links at all)

Disney+ and Apple Music open straight to a title when the Apple TV is sent a link. BBC iPlayer on Apple TV doesn't accept links at all, even the ones BBC registers for its iPhone app. Disney+ links go to a show rather than an episode, so asking for episode 3 would play whichever episode Disney+ thinks you're on.

For episodes, the app opens the show and moves through the menu with remote presses to the episode asked for. The starting position in the menu varies, so it sometimes landed on the wrong one. Disney+ reports the episode that's playing (for example "S1:E4 Malware"), so the app now checks it and, if it's wrong, goes back and moves up or down by the difference.

When the TV keeps its picture but loses its sound

Twice in one evening the TV kept its picture but stopped sending sound to the soundbar, a known fault with the HDMI audio (eARC) link. The app can see this happening, because the soundbar reports no audio while the Apple TV reports playing. A "fix sound" request re-selects the soundbar's TV input and, if that doesn't work, switches the TV off and on through the Apple TV. The first version also did that when nothing was playing, so it now checks first.

Problems we found

Reading the soundbar's input format showed that a film with a Dolby Atmos soundtrack was arriving as plain 5.1, and Atmos music as stereo. The sound was being reduced somewhere between the Apple TV, an HDMI switch and the TV, and nothing on screen showed it.

Unpublished interfaces, two networks, and where most of the work went

Three of the connections, to the Apple TV, the router and the PlayStation, use interfaces their makers don't publish. A macOS update also stopped the app reaching the network until the permission was granted again. We keep a runbook of each fix.

The house has two separate networks, one from the broadband router and one from a second router. Devices move between them on their own, usually after a brief drop in Wi-Fi, and the two Sonos systems turned out to share one identity, so the app had to look for speakers on both networks and remember each one it had seen. Plugging the living-room equipment into the router by cable stops the moving.

Most of the work went into the behaviour of the devices and apps rather than the language model: which apps take links, how long the TV takes to wake over HDMI, and when "no audio" is normal.

Beyond the house: physical effects

Some of our work for immersive shows has used physical effects such as wind, haptics, heat and scent, timed to what's happening on screen or around an audience. There's no particular reason similar effects couldn't be tied into the same controls at home.

It could make for interesting experiments with films, games and other entertainment, with lighting, sound and physical effects responding together. That's something we'd like to explore rather than anything built into the house so far.

What's next

• An Apple TV app with the moods as tiles, and the moods on the Watch face.
• A mood that runs when a phone arrives home.
• More equipment: door sensors so "goodnight" can say whether the back door is shut, and buttons on the wall for a mood.
• Moving more requests to the on-device models, so fewer go to the cloud.

Bespoke versions

The same approach could suit a venue or an exhibition that needs a space to respond reliably to visitors, or a household that wants something built around its own equipment. If that's you, use our contact form and tell us a little about the space and what you'd want it to do.

Could we release the apps?

We're looking at making some of this available as a small set of iPhone and Apple Watch apps. The idea is to control Sonos speakers and Philips Hue lights directly over the home network, with Apple Music, without needing a Mac mini or a server run by us.

It's still an experiment and we haven't decided whether to release anything. Some of the features in our own home system, particularly Apple TV control and the router, rely on interfaces their makers don't publish and may not be suitable for a public App Store release. Streaming and music search would still need the internet.

If you'd find something like this useful, we'd like to hear which parts matter to you.

Email address

What equipment do you have? Tick any

Sonos speakers Philips Hue lights Apple Watch Apple TV Apple Music subscription Other

Which part interests you most?

Controlling lights and music together Quick controls from Apple Watch Getting someone's attention through speakers and lights Checking on the house Typed or spoken requests Something else

Anything else you'd like us to know? Optional I'd like Reflex Arc to email me if these apps become available. I understand there's no release date and I can ask to be removed at any time.

Register interest

We'll use your details only for updates about these apps. See our privacy policy for how we handle and remove your information.

Questions people ask

Can Sonos and Philips Hue be controlled from one app?

Yes, in a few ways. Sonos's own voice assistant can control Hue lights (in US English and French), Philips has announced Hue switches that control Sonos playback, and some independent apps cover both. We built our own so that moods can set the lights, music and TV together, from an iPhone, an Apple Watch or Siri.

Would Apple TV control be included in a public release?

Probably not at first. Controlling an Apple TV relies on interfaces Apple doesn't publish, which could cause problems in App Store review and later updates.

Why is my Sonos only getting stereo or 5.1 from my Apple TV?

Usually something in the HDMI chain is telling the Apple TV it can't take more: the TV, a switch or a receiver. With its Audio Format on Auto, the Apple TV sends the best it's told is supported. Try the Apple TV plugged straight into the TV's eARC-friendly input, check the TV's audio output settings, and see what the soundbar says it's receiving (the Sonos app shows it).

My Sonos has no sound from the TV but the picture is fine. What fixes it?

Re-select the TV input on the soundbar first. If that doesn't do it, switch the TV off and on again: it restarts the HDMI audio (eARC) link between the TV and the soundbar, which was what worked for us. Check the TV hasn't switched its sound back to its own speakers, too.

Can I control a TV or other equipment that isn't on the network?

Usually, if it has an infrared remote. A small Wi-Fi infrared transmitter placed where it can see the equipment learns the remote's buttons, and an app can then send them: power, inputs, volume and menus.

How does this compare with Home Assistant?

Home Assistant is excellent and connects to far more devices. This project is narrower: a few devices, moods across them, native Apple apps, and checking that each request actually did what was asked.

Apple TV, Apple Watch, Apple Music, Siri, Sonos, Philips Hue, Disney+ and BBC iPlayer are trademarks of their owners. We're not affiliated with any of them.