EDITH: Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze

A framework enabling robots to see what you see.

A person's egocentric view, eye gaze, and hand pose carry much of what a robot needs to interact and collaborate with them: what they are doing, what they are attending to, and what they want next.

EDITH is a framework that uses Project Aria Gen 2 glasses to stream these signals, along with the person's speech, into the robot's input in real time. It covers the hardware setup, data collection, and policy training and deployment.

In our paper, we designed a hierarchical policy that understands human intent from speech complemented by nonverbal signals (Section 4). There are many more use cases, and we hope this framework enables many more experiments.

git clone --recurse-submodules https://github.com/dgjun32/edith.git && cd edith
uv venv .venv --python 3.11
uv pip install --python .venv/bin/python -e '.[aria,realsense,policies,replay]'
source .venv/bin/activate

aria_gen2 auth pair                                                # pair the glasses once
cp configs/station.example.json configs/station.local.json         # set camera serials and joints
edith-eval --station configs/station.local.json --check --headless # device check
edith-dashboard --station configs/station.local.json               # live view

If you cloned without --recurse-submodules, run git submodule update --init. YAM control and GELLO teleop: README_LOCAL_YAM.md. Other robots: implement a RobotAdapter (example).

A typical robot environment observes only the robot's cameras and joints. Here the observation also includes the human: their real-time egocentric view, gaze, hands, and speech audio, synchronized with the robot's cameras and joint state.

from edith.config import load_station
from edith.envs.local_env import LocalSensorEnv
from edith.eval_utils import RobotEvaluationEnv
from edith.robot_utils.noop import ConfiguredNoOpRobot

config = load_station("configs/station.local.json")
robot = ConfiguredNoOpRobot(config)  # no motor connection
robot.start()
env = RobotEvaluationEnv(LocalSensorEnv(config), robot, robot.read(0))
try:
    obs = env.get_policy_observation()
finally:
    env.safe_end()

All streams share one clock and are aligned within 100 ms at 15 Hz.

edith-collect --station configs/station.local.json --duration 20 --headless \
  --output recordings/ep0/observations.h5
edith-replay recordings/ep0/observations.h5 --output-dir recordings/ep0/replay

Every observation stream is saved to HDF5; replay exports a video with audio. Example recording: examples/zero_action_observations.

Our paper uses this setup to make a robot follow a person's real-time nonverbal instructions: what they look at, point to, and say, as in "give me that".

Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze
Dongjun Lee*, Juheon Choi*, Dong Kyu Shin*, Sinjae Kang, Kimin Lee (KAIST, Config; *equal contribution)
Project page · arXiv

Coming soon: turning recorded episodes (Section 3) into training data.

Coming soon: training code and checkpoints for the EDITH policy. Until then, train a model elsewhere (e.g. π_0.5 with OpenPI), serve it at ws://127.0.0.1:8000, and evaluate it as in Section 5.

Every policy, from a single end-to-end model to a hierarchical planner + controller, implements one class: edith.policies.BasePolicy. edith-eval calls it with the Section 2 observation every step and sends the returned joint targets to the robot environment.

Start from the template, a directory with policy.json and policy.py:

cp -r edith/policies/template my_policy
edith-eval --station configs/station.local.json --policy-dir my_policy --headless

from edith.policies import BasePolicy

class Policy(BasePolicy):
    def act(self, obs):            # required: called every step
        return obs["robot_state"]  # one (D,) joint target, in robot_state order

    # optional: __init__(policy_dir, config) to load models,
    #           reset(obs) for the first observation, close() at the end

• act must return within one step (66 ms at 15 Hz). Run slow models such as planners and VLMs in a background thread and return the latest target; the runner keeps re-sending it to the robot in the meantime.
• Return one action per call. If your model predicts action chunks, buffer them inside the policy.
• obs also carries instruction and, when given, transcript (--transcript-file) and policy_event (--policy-events-file).

policy.json tells the runner which class to load and how to call it:

{"schema": "edith.policy.v1", "name": "my-policy", "entrypoint": "policy.py:Policy",
 "control_hz": 15, "action_space": "joint_position", "config": {}}

Bundled policies in edith/policies also run by name, e.g. --policy-dir policy_edith: the hierarchical example, where a Gemini planner queues keyframe-grounded subtasks for a low-level VLA, built on edith/policies/common.

Without --execute-actions the arms never move: sensors and the policy run, and actions are only checked (and recorded with --output). With it, the robot starts, and the first action must be within 0.05 of the current pose (--max-first-action-delta).

@article{lee2026hierarchicalpoliciesverbalegocentric,
      title={Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze},
      author={Dongjun Lee and Juheon Choi and Dong Kyu Shin and Sinjae Kang and Kimin Lee},
      year={2026},
      url={https://arxiv.org/abs/2606.10276},
}