Features

Drift detection

Automatic alerts when a model starts behaving differently from how it was behaving last quarter.

Models drift. The provider updates the underlying weights, the inputs change, the kinds of customers asking for advice shift, the prompt is tweaked. Drift detection is the capability that surfaces these changes early — before they show up as customer complaints.

How it works

For every (provider, version) in your model registry, Bedrock compares two trailing windows of completed review jobs:

  • Current window — the most recent 30 days
  • Baseline window — the 90 days immediately before that

It computes three rates in each window:

  • Rejection rate — fraction of jobs that ended in REJECTED
  • Modification rate — fraction that ended in APPROVED_WITH_MODIFICATIONS
  • Annotation frequency — average count of reviewer notes per job

It then takes the absolute delta (current minus baseline) for each metric. The deltas are graded by severity:

MetricWarning atAlert at
Rejection rate±0.05 (5pp)±0.10 (10pp)
Modification rate±0.05 (5pp)±0.10 (10pp)
Annotation frequency+1 per job+2 per job

Why these specific thresholds

The 5 / 10 percentage-point bands roughly correspond to one and two standard deviations on a typical UK adviser caseload of a few hundred jobs per quarter — the smallest move likely to be a real signal rather than statistical noise. The annotation thresholds are absolute counts because annotation behaviour is bounded and a single extra note per job represents a meaningful uptick in reviewer concern.

Querying the report

bash
curl https://api.bedrockcompliance.co.uk/v1/firm/me/models/drift \
  -H "X-Bedrock-Key: bk_live_..."

Returns a ModelDriftReport: the list of signals (sorted alert → warning → info), the thresholds in effect, and the window sizes used. See the schema in the Firm API reference.

Evidence produced

  • MODEL_DRIFT_DETECTED ledger event whenever a new alert fires
  • Time-series data via ModelTimelinePoint for charting
  • The full ModelDriftReport snapshot, addressable by date

Operational pattern

  1. Schedule a daily job to fetch the drift report.
  2. For any new alert-severity signal, open a Principal investigation.
  3. If the model has actually changed behaviour, freeze the version in the registry, switch advisers to the previous version, and run an impact assessment on the new one before reinstating it.

FCA mapping

  • PRIN 2A.5.1R — “A firm must monitor the outcomes that retail customers experience”
  • SYSC 8 — change management for outsourced operational functions

See also