Scrutable

In the Dutch news is an item about a new report released today that completely panned the use of a tool for estimating the risk of recidivism in people up for parole. The tool has been in use since 2018 and was based on the OxRec model, shown on this page, which was developed for use in Sweden.

The report was exceptionally damning. It found that in the implementation used in the Netherlands:

• Two formulas were swapped around. One formula applied to convicts, the other to suspects.
• An incorrect value was used for one coefficient, causing it to grossly understimate risks associated with drug use.
• The model was originally trained on data from Sweden from 20 years ago, and not retrained on data from the Dutch population, or even more recent data.
• The tool was structurally used in a situation where there is no information on serious mental health issues, which is one of the inputs needed in the model. They used the model as if everyone had perfect mental health, instead of recalibrating/retraining it for use without mental health data.
• Some of the inputs lead to discriminatory bias. In particular, postal code (a "neighbourhood score") and income were used, which tends to grossly discriminate against immigrants. The use of such input has in fact been ruled illegal without proper safeguards or mitigation, apart from extremely rare exceptions. There was no measurement, evaluation, let alone mitigation put in place here.
• The staff were told in their training for using the tool that their own judgement was unreliable. They had little choice but to rely on the software, which was presented as being fair and unprejudiced.
• Management failed to reflect on and evaluate the software and its use, so that its shortcomings went unnoticed for years.

Some of these criticisms had been raised from the start.

The use of this tool has been immediately halted.