When an agent crosses a boundary
Goals drift. Oversight has to follow.
AI agents break goals into sub-actions. A goal set at one boundary can produce sub-actions at another, while the system's autonomy setting stays the same. Each step can look routine. In its consequences for a particular person, it is not.
A scheduling agent
Its goal is institutional: cut staffing costs while meeting demand. Along the way it could:
- Assign shifts (routine)
- Deny accommodation requests (normative)
- Cut hours for people nearing benefits eligibility (normative)
Routine scheduling can run on its own. The normative sub-actions go to a manager who decides.
62%
of recorded AI incidents involved boundary crossing, in a Bacon Center analysis of 1,491 incidents in the AI Incident Database.