Repository navigation
Add desktop control security policy and opt-in model #461
Copy link
Copy link
Open
Labels
agentcuaComputer Use AgentComputer Use Agentdomain:multimodalVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAenhancementNew feature or requestNew feature or requestp2low prioritylow prioritysecuritySecurity-sensitive changesSecurity-sensitive changestrack:consumer-appConsumer product track — mobile-first: voice + messaging + memory + skillsConsumer product track — mobile-first: voice + messaging + memory + skills
Milestone
Description
Summary
Desktop control is inherently more dangerous than browser use — it can interact with any application, see credentials on screen, and modify system state. This issue defines the security policy required before shipping CUA.
Risk Assessment
| Risk | Severity | Mitigation |
|---|---|---|
| Close critical applications | High | App allowlist, confirmation for destructive actions |
| Capture credentials on screen | High | Credential keyword detection, screenshot auto-redaction |
| Delete files via file manager | High | Block file manager interactions by default |
| Rapid uncontrolled clicking | Medium | Rate limiting (max 60 actions/min) |
| Control system settings | Medium | Block system preferences apps |
| Privacy — screenshots capture everything | Medium | No screenshot persistence by default |
Security Policy
@dataclass
class DesktopSecurityPolicy:
# Activation
enabled: bool = False # Must be explicitly enabled
require_env_var: bool = True # GAIA_DESKTOP_CONTROL_ENABLED=true
# Application control
allowed_apps: List[str] = None # None = all (except blocked)
blocked_apps: List[str] = [ # Never control these
"Terminal", "iTerm", "cmd", "powershell", # Shell access
"System Preferences", "Settings", # System config
"Keychain", "Credential Manager", # Credentials
]
# Input controls
mouse_enabled: bool = True
keyboard_enabled: bool = True
max_actions_per_minute: int = 60
blocked_key_combos: List[str] = ["ctrl+alt+del", "cmd+q"]
# Screenshot controls
persist_screenshots: bool = False # Don't save to disk by default
redact_credentials: bool = True # Blur password fields
# Safety
confirm_destructive: bool = True # Confirm before closing windows
max_session_duration_min: int = 30 # Auto-disable after 30 min
require_user_confirmation: bool = False # Optional: confirm each actionUser Experience
- User runs `gaia chat --enable-desktop-control`
- Warning banner displayed:
⚠️ Desktop control enabled. The agent can see and interact with your screen.
Only use this with trusted tasks. Press Ctrl+C at any time to stop. - Agent announces actions before performing them
- Session auto-expires after 30 minutes
Files
- `src/gaia/agents/chat/tools/desktop_tools.py` (security validation)
- `tests/unit/chat/test_desktop_security.py` (NEW)
- `docs/guides/desktop-control.mdx` (security section)
Acceptance Criteria
- Desktop control disabled by default — requires explicit opt-in
- Warning banner shown on activation
- Blocked apps cannot be controlled
- Rate limiting enforced
- Blocked key combinations rejected
- Auto-expiry after configured duration
- Unit tests for every security check
- Security policy documented
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
agentcuaComputer Use AgentComputer Use Agentdomain:multimodalVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAenhancementNew feature or requestNew feature or requestp2low prioritylow prioritysecuritySecurity-sensitive changesSecurity-sensitive changestrack:consumer-appConsumer product track — mobile-first: voice + messaging + memory + skillsConsumer product track — mobile-first: voice + messaging + memory + skills
Activity
feat(cpp): text extraction and chunking (#2822)