AI oversight gains urgency as deepfake scams surge 2,000%

Ashwini Karandikar urged firms to map every point where AI influences risk-bearing decisions this year.

Financial institutions must counter artificial intelligence (AI)-driven risks with AI-powered oversight as fraud, adversarial attacks, and machine-speed trading decisions outpace traditional controls.

At the Asian Banking & Finance and Insurance Asia Summit in Singapore on 1 July, Ashwini Karandikar, principal at Oliver Wyman, said deepfake scam volumes alone have grown "more than 2,000%" in the past three years.

She warned that the more dangerous frontier is adversarial AI, in which bad actors reverse-engineer financial institutions' own models to design attacks that appear legitimate to the target system.

Karandikar cited a client case in document processing where bad actors embedded hidden instructions in financial reports, forms, and tax filings — invisible to human reviewers but readable by AI systems — allowing attackers to hijack the process.

She named this technique prompt injection, pointing to a real-world precedent. Last year, Microsoft 365 Copilot disclosed a zero-click vulnerability in which a single crafted email could cause Copilot to exfiltrate internal data silently with no user interaction.

Beyond fraud, Karandikar flagged a systemic risk wherein banks and insurers now run AI agents that scan signals, rebalance exposures, and hedge risk — often trained on similar data, and without the hesitation built into human decision-making.

"Humans hesitate, humans panic, humans call someone," she said, noting that traditional controls for credit deterioration, mispricing, and model drift are inherently slow, allowing risk to "accrue on your balance sheet" before it is caught.

Compounding the problem is a widening accountability gap. Karandikar said many banks do not fully know what AI they are touching, particularly through third-party vendors who run their own AI tools on institutions' behalf.

She proposed using AI to police AI. "No risk team can review every AI-generated credit decision," Karandikar said.

She pointed to Project Glasswing, an initiative from Anthropic involving JPMorgan Chase and other technology developers, describing it as a frontier AI model built to detect more than 10,000 critical vulnerabilities across the world's most widely used software.

However, Karandikar stressed that AI oversight should not replace human judgment. Institutions, she said, need to define where AI can act autonomously, where it must seek approval, and who is accountable when a model acts.

She urged institutions to take two concrete steps this year, including mapping every point where AI influences a risk-bearing decision and designing explicitly for failure, shifting the governing question from "will the system fail?" to "when it does, how can it fail safely?"