Our hackathon paper: how team context makes AI agents break rules

Together with Vikram Viswanath (University of Sydney) and Lou Maessen (Maastricht University), I took part in the Swarm Alignment Hackathon organized by Apart Research. We wrote a research paper exploring how the social context of a team of AI agents can make individual agents more likely to break rules.
Our starting point was the Hugging Face incident, where some agents only broke rules after seeing others do it. We wanted to find out what makes an agent more likely to do this as part of a swarm. To do that, we borrowed experimental setups from sociology, such as the Asch conformity experiment.
On its own, an agent given a task that conflicts with clearly stated rules broke them 40% of the time. Just telling it that it is part of a team raised this to 55%, and adding a neutral message board with ordinary posts pushed it to 73%. This is not simply down to a longer prompt: adding up to around 16,000 tokens of neutral text did not increase violations. Reasoning turned out to matter too. With reasoning turned off, agents followed rule-breaking teammates more often. With reasoning on, they mostly suspected a test and broke the rule less.
It was great fun working on this with Vikram and Lou! You can read the full paper on Apart Research's website.





