A story
on NPR Friday about the crashes of two Boeing 737 Max 8 planes begins this
way:
The investigation … points, for the moment, at software. There’s evidence that a program called MCAS pointed the planes’ noses down repeatedly without pilots even knowing, ultimately leading to the deaths of hundreds of people.
The story then discusses our increasing reliance and,
perhaps, overreliance on software without even knowing how the software works.
The story concludes with a warning from Prof. Eben Moglen of
Columbia University School of Law, described by NPR as an advocate for
“software transparency.” Says Prof.
Moglen:
Even if you didn’t think this had anything to do with politics, it is still true that systems that don’t explain themselves to the human beings who interact with them are dangerous.
Child welfare, of course, has everything to do with
politics. And the latest fad in child
welfare is “predictive analytics,” in which child welfare agencies use software
that’s supposedly able to predict who is likely to abuse a child. The many problems with this are summarized here. But one of those many problems is the fact
that the software is – secret.
In most cases, the algorithms are created by for-profit companies;
so, obviously, they don’t want us to know what goes into them. But even the wildly
overhyped model in Pittsburgh, where they’re constantly bragging about
transparency, isn’t nearly transparent enough.
In Pittsburgh, they’ll gladly tell you the list of items the
algorithm considers. That means at least
we can see how often they are actually measuring nothing more than the
likelihood of being poor – what Prof. Virginia Eubanks, author of Automating
Inequality, calls “poverty profiling.” But they won’t say how the
various items are weighted.
This would be bit like a hypothetical executive for a
hypothetical aircraft manufacturer telling a pilot: “So, Captain, at certain
times the software on this new plane will affect the position of the nose – but
we won’t tell you when it kicks in, and we won’t tell you if it pushes the nose
down or pulls it up.”
Whatever one may think of what Boeing did or failed to do,
no one is suggesting they did anything like that.
It would be preposterous.
So why isn’t it considered preposterous in child welfare?
It should be. Because systems that don’t explain themselves
to the human beings who interact with them are dangerous.
