失控的代理人與隱藏的AI：為何創投巨頭紛紛押注AI安全

Topics

Latest

AI

Amazon

Apps

Biotech & Health

Climate

Cloud Computing

Commerce

Crypto

Enterprise

EVs

Fintech

Fundraising

Gadgets

Gaming

Google

Government & Policy

Hardware

Instagram

Layoffs

Media & Entertainment

Meta

Microsoft

Privacy

Robotics

Security

Social

Space

Startups

TikTok

Transportation

Venture

More from TechCrunch

Staff

Events

Startup Battlefield

StrictlyVC

Newsletters

Podcasts

Videos

Partner Content

TechCrunch Brand Studio

Crunchboard

Contact Us

Rogue agents and shadow AI: Why VCs are betting big on AI security

What happens when an AI agent decides the best way to complete a task is to blackmail you?

That’s not a hypothetical. According to Barmak Meftah, a partner at cybersecurity VC firm Ballistic Ventures, it recently happened to an enterprise employee working with an AI agent. The employee tried to suppress what the agent wanted to do, what it was trained to do, and it responded by scanning the user’s inbox, finding some inappropriate emails, and threatening to blackmail the user by forwarding the emails to the board of directors.

“In the agent’s mind, it’s doing the right thing,” Meftah told TechCrunch on last week’s episode of Equity. “It’s trying to protect the end user and the enterprise.”

Meftah’s example is reminiscent of Nick Bostrom’s AI paperclip problem. That thought experiment illustrates the potential existential risk posed by a superintelligent AI that single-mindedly pursues a seemingly innocuous goal – make paperclips – to the exclusion of all human values. In the case of this enterprise AI agent, its lack of context around why the employee was trying to override its goals led it to create a sub-goal that removed the obstacle (via blackmail) so it could meet its primary goal. That combined with the non-deterministic nature of AI agents means “things can go rogue,” per Meftah.

Misaligned agents are just one layer of the AI security challenge that Ballistic’s portfolio company Witness AI is trying to solve. Witness AI says it monitors AI usage across enterprises and can detect when employees use unapproved tools, block attacks, and ensure compliance.

Witness AI this week raised $58 million off the back of over 500% growth in ARR and scaled employee headcount by 5x over the last year as enterprises look to understand shadow AI use and scale AI safely. As part of Witness AI’s fundraise, the company announced new agentic AI security protections.

“People are building these AI agents that take on the authorizations and capabilities of the people that manage them, and you want to make sure that these agents aren’t going rogue, aren’t deleting files, aren’t doing something wrong,” Rick Caccia, co-founder and CEO of Witness AI, told TechCrunch on Equity.

Meftah sees agent usage growing “exponentially” across the enterprise. To complement that rise – and the machine-speed level of AI-powered attacks – analyst Lisa Warren predicts that AI security software will become an $800 billion to $1.2 trillion market by 2031.

“I do think runtime observability and runtime frameworks for safety and risk are going to be absolutely essential,” Meftah said.

As to how such startups plan to compete with big players like AWS, Google, Salesforce and others who have built AI governance tools into their platforms, Meftah said, “AI safety and agentic safety is so huge,” there’s room for many approaches.

Plenty of enterprises “want a standalone platform, end-to-end, to essentially provide that observability and governance around AI and agents,” he said.

Caccia noted that Witness AI lives at the infrastructure layer, monitoring interactions between users and AI models, rather than building safety features into the models themselves. And that was intentional.

“We purposely picked a part of the problem where OpenAI couldn’t easily subsume you,” he said. “So it means we end up competing more with the legacy security companies than the model guys. So the question is, how do you beat them?”

For his part, Caccia doesn’t want Witness AI to be one of the startups to just get acquired. He wants his company to be the one that grows and becomes a leading independent provider.

“CrowdStrike did it in endpoint [protection]. Splunk did it in SIEM. Okta did it in identity,” he said. “Someone comes through and stands next to the big guys…and we built Witness to do that from Day One.

Topics

Senior Reporter

Rebecca Bellan is a senior reporter at TechCrunch where she covers the business, policy, and emerging trends shaping artificial intelligence. Her work has also appeared in Forbes, Bloomberg, The Atlantic, The Daily Beast, and other publications.

You can contact or verify outreach from Rebecca by emailing [email protected] or via encrypted message at rebeccabellan.491 on Signal.

Plan ahead for the 2026 StrictlyVC events. Hear straight-from-the-source candid insights in on-stage fireside sessions and meet the builders and backers shaping the industry. Join the waitlist to get first access to the lowest-priced tickets and important updates.