“New Zealand Crunches Big Data to Prevent Child Abuse,” declared a Chronicle of Social Change headline on a 2015 story about The Chronicle’s favorite child welfare fad, predictive analytics, or as it should properly be called, data-nuking poor families.
The story quotes Paula Bennett, New Zealand’s former minister of
social development, declaring at a conference: “We now have a golden
opportunity in the social sector to use data analytics to transform the lives
of vulnerable children.”
If implemented, the story enthuses, it would be a “world first.”
All this
apparently was based on two studies that, it now turns out, used methodology so
flawed that it’s depressing to think that things ever got this far.
That’s revealed in a detailed analysis by Professor Emily Keddell of the University ofOtago.
The studies that supposedly proved the value of predictive
analytics attempted to predict which children would turn out to be the subjects
of “substantiated” reports of child maltreatment.
Among the children identified by the software as being at the
very highest risk, between 32 and 48 percent were, in fact, “substantiated”
victims of child abuse. But that means more than half to more than two-thirds
were false positives.
Think about that for a moment. A computer tells a caseworker
that he or she is about to investigate a case in which the children are at the
very highest level of risk. What caseworker is going to defy the computer
and leave these children in their homes, even though the computer is wrong more
than half the time?
But there’s an even bigger problem. Keddell concludes that
“child abuse” is so ill-defined and so subjective, and caseworker decisions are
so subject to bias, that “substantiation” is an unreliable measure of the
predictive power of an algorithm. She writes:
How
accurately the substantiation decision represents true incidence is … crucial
to the effectiveness of the model. If substantiation is not consistent, or does
not represent incidence, then identifying an algorithm to predict it will
produce a skewed vision …
Turns out, it is not consistent, it does not represent
incidence, and the vision is skewed. Keddell writes:
Substantiation
data as a reflection of incidence have long been criticized by researchers in
the child protection field … The primary problem is that many cases go
[unreported], while some populations are subject to hypersurveillance, so
that even minor incidents of abuse are identified and reported in some groups.
That problem may be compounded, Keddell says, by racial and
class bias, whether a poor neighborhood is surrounded by wealthier
neighborhoods (substantiation is more likely in such neighborhoods), and even
the culture in a given child protective services office.
Predictive Analytics Becomes Self-fulfilling Prophecy
Algorithms don’t counter these biases, they magnify it.
Having a previous report of maltreatment typically increases the
risk score. If it’s “substantiated,” the risk score is likely to be even
higher. So then, when another report comes in, the caseworker, not about to
overrule the computer, substantiates it again, making this family an even
higher “risk” the next time. At that point, it doesn’t take a computer to tell
you the children are almost certainly headed to foster care.
Writes Keddell:
“prior
substantiation may also make practitioners more risk averse, as it is likely to
heighten perceptions of future risk to the child, as well as of the
practitioner’s own liability, and lead to a substantiation decision being
made.”
So the predictive analytics become a self-fulfilling prophecy.
Keddell also highlights the problems when even accurate data are
misused by fallible human beings:
Several researchers note the tendency for individualised risk scores to be utilised in negative ways in practice, where actuarial approaches are prioritized over professional judgement. While statistical modellers may understand the tentative nature of statistical prediction or correlation … practitioners tend to treat statistical data, especially when stripped of its explanatory variables, as solid knowledge, which can function as a received truth.
But it turns out there may be one area where predictive analytics
can be helpful. Keddell cites two studies in which variations on analytics were
used to detect caseworker bias. In one, the researchers could predict which
workers were more likely to recommend removing children based on questionnaires
assessing the caseworkers’ personal values.
In
another, the decisions could be predicted by which income level was described
in hypothetical scenarios. A study using similar methodology uncoveredracial bias.
So how about channeling all that energy now going into new ways
to data-nuke the poor into something much more useful: algorithms to detect the
racial and class biases among child welfare staff? Then we can protect children
from biased decisions by “high risk” caseworkers.
