From Tools to Teammates: (Dis)Trust in AI for Cybersecurity with Neele Roch

Human-Centered Security

When we collaborate with people, we build trust over time. In many ways, this relationship building is similar to how we work with tools that leverage AI.

As usable security and privacy researcher Neele Roch found, “on the one hand, when you ask the [security] experts directly, they are very rational and they explain that AI is a tool. AI is based on algorithms and it's mathematical. And while that is true, when you ask them about how they're building trust or how they're granting autonomy and how that changes over time, they have this really strong anthropomorphization of AI. They describe the trust building relationship as if it were, for example, a new employee.”

Neele is a doctoral student at the Professorship for Security, Privacy and Society at ETH Zurich. Neele (and co-authors Hannah Sievers, Lorin Schöni, and Verena Zimmermann) recently published a paper, “Navigating Autonomy: Unveiling Security Experts’ Perspective on Augmented Intelligence and Cybersecurity,” presented at the 2024 Symposium on Usable Privacy and Security.

In this episode, we talk to Neele about:

• How security experts’ risk–benefit assessments drive the level of AI autonomy they’re comfortable with.
• How experts initially view AI: the tension between AI-as-tool vs. AI-as-“teammate.”
• The importance of recalibrating trust after AI errors—and how good system design can help users recover from errors without losing their trust in it.
• Ensuring AI-driven cybersecurity tools provide just the right amount of transparency and control.
• Why enabling security practitioners to identify, correct, and learn from AI errors is critical for sustained engagement.

Roch, Neele, Hannah Sievers, Lorin Schöni, and Verena Zimmermann. "Navigating Autonomy: Unveiling Security Experts' Perspectives on Augmented Intelligence in Cybersecurity." In Twentieth Symposium on Usable Privacy and Security (SOUPS 2024), pp. 41-60. 2024.