Model Governance: Statistical Controls for Statistical Models

I’ve previously written about model retraining and model monitoring – two techniques that we use at RDC to manage the risk of our AI Review models producing a false-negative (that is, scoring a potential alert so that it is below a threshold value or more simply not alerting on someone who should be brought forward for review). This week, I’ll wrap up the model governance and model risk management discussion with the third technique that we use to lower this false-negative risk: input set anomaly detection. Whereas retraining uses analyst decisions from the recent past to keep models up to date, and monitoring focuses on predictions that are happening right now, anomaly detection focuses on what might happen in the future. Specifically this means what could happen if a model makes a decision based on new input data that is so different from anything the model has seen in the past that the prediction can’t be trusted. This is one of the most important distinctions between predictive models and human intelligence: predictive models can only reproduce past decision-making behavior; predictive models cannot think, use judgment or any other intellectually creative process that a person can. So, when a model is presented with something new – an anomaly, activity unlike anything the model has been trained on – the results are unpredictable and can’t be trusted.

One of the first, and most entertaining, examples of not handling new patterns well was created by researchers at MIT’s Lab 6 to fool Google’s InceptionV3 image recognition model. When Google acquired YouTube, they used the millions of images to train models to detect pictures that contained cats, then other animals, common household objects, etc. And the results were very, very impressive: InceptionV3 correctly identified cats and other things nearly all of the time, vastly outperforming any previous image recognition solution. However, the folks at Lab 6 maintained that there was so much potential variability in photographs that it would be easy to fool the Google model with images of things that any human could easily identify. This led to the now famous (in the AI community, anyway) image of a turtle that InceptionV3 identified with 90% confidence as a rifle; ditto a cat was identified as a bowl of guacamole. The important point is that if you cannot guarantee that your model won’t be asked to process input that is anomalous, you have to detect the anomalies an route them to people. That’s what RDC does with AI Review.

We use a simple technique called k-means clustering, which produces a small number of distinct sets of data points. Each set contains data points that are similar to one another. We perform k-means clustering on the data that each model was trained on, and then on an ongoing basis, on the data that is presented to the model. If an input data point ends up in one of the original clusters, the model’s prediction can be trusted; If, however, the data point ends up in a cluster of its own, it is anomalous. We route these potential alerts to human analysts, and ensure that they are included in future training sets.

Understanding how models work – and more significantly where they may not work – is important for the ongoing veracity of the results and independent maintenance of the models. AI currently expertly solves for expected activities, in turn saving time and money. That being said, identifying anomalies is critical – and human intelligence can help complete the analysis. RDC’s model governance, and our understanding of effective maintenance, is the backbone to valuable AI solutions.