Newer AI models missed more payment fraud in Coinbase’s benchmark

Coinbase reported Oct. 7 that more recent variations of 3 significant AI design households captured less deceitful payments and a smaller sized share of scams worth in a historic test of payment screening for its Onramp service, regardless of a the same choice policy. The findings challenge the presumption that updating a design enhances a current payment screener.

The business’s examination replayed 16,140 deals throughout 7,293 users, consisting of 813 validated deceptive deals. The associate covered 9 weeks before its threat representative presented, keeping all grew scams cases while tasting genuine traffic.

Each prospect examined current deal habits under repaired assistance and the very same policy for turning danger categories into choices. This separated the choice design’s habits within that setup, instead of comparing upgraded screening systems.

Arise from a repaired historic replay

Coinbase compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol). Every more recent variation had lower recall, a lower combined precision-and-recall rating called F1, and lower dollar-weighted recall. Remember procedures the share of scams cases a design captures; dollar-weighted recall procedures just how much of the overall scams worth it captures.

Sonnet’s recall fell 22.2 portion points and its dollar-weighted recall dropped 22.9 points. Opus’s recall decreased 0.8 points. Both more recent designs likewise had lower accuracy, implying a smaller sized share of deals they categorized as scams were in fact deceptive.

GPT revealed why one enhancing rating can be deceptive. Its accuracy increased 11.5 portion points, however recall fell 20.7 points and dollar-weighted recall fell 21.8 points. Its scams flags were more precise, while more scams cases and worth got away detection in the replay.

. The replay does not develop consumer losses from releasing those variations. Coinbase likewise stated it might determine the regressions without developing their cause.

Coinbase’s earlier online experiment compared including selective LLM evaluation with the existing designs and guidelines alone. That agent-enabled circulation tape-recorded 30% less deceitful deals and 22% less scams worth; it did not compare more recent design variations.

In their restrictionsthe SR-Fraud scientists state the exclusive dataset can not be launched, limiting independent duplication and generalization. Their associated payment-fraud research study Appeared Sept. 23 and was modified Sept. 30, before the October blog sites.

A different case for a custom-made design

In its Oct. 8 disclosureCoinbase reported that a post-trained Qwen3.5-9B design surpassed Opus 4.5 throughout 4 fraud-detection metrics. F1 enhanced 9.6 portion points and dollar-weighted recall increased 35.4 points. The business specialized it utilizing historic scams results and deterministic benefits stabilizing deceptive and genuine examples.

Individually, production measurements put average end-to-end LLM-request latency at 0.683 seconds versus 1.515 seconds for Opus 4.5, a 55% relative decrease. Faster reasoning and more powerful benchmark detection originated from various examinations.

For payment service providers, the upgrade concern is whether a prospect enhances scams protection under their real choice setup. Coinbase suggests screening that setup initially, then examining altered triggers or limits independently, with latency, dependability and expense together with detection quality.

Post context

Discussed in this post