Normally, it would be just the kind of story I'd love – the kind I'd rush to send to the NCCPR Child Welfare News Exchange, my e-mail list of more than 300 reporters around the country who cover child welfare: An editorial singing the praises of Alabama's reformed child welfare system for doing an outstanding job of keeping children safe.
After all, any visitor to www.nccpr.org knows that this organization repeatedly has cited Alabama as, relatively speaking, a national model. (A member of NCCPR's Board of Directors was co-counsel for plaintiffs in the landmark class-action lawsuit that helped make it a model). Indeed, Alabama's success appears to be a thorn in the side to the trade association for child welfare agencies, the Child Welfare League of America – if a recent snide crack from a top CWLA official using misinformation to try to smear the reforms is any indication.
But I didn't send the story, and I won't. That's because, while there are plenty of good data to support the contention that Alabama does a very good job of keeping children safe, including independent monitoring reports and overall data on reabuse, the editorial wasn't based on those data.
Instead, the editorial was based on Alabama's performance on its second Child and Family Services Review. And CFSRs, as these federal reviews of child welfare systems are known, are C-R-A-P.
The methodology is so absurd that a Ouija board or a coin toss would be as likely to produce an accurate picture of a state's child welfare system.
And that makes them dangerous. It's easy for good systems to fail and for bad systems to get an undeserved seal of approval. For example, during the first round of CFSRs, the Washington D.C. system was one of the few rated as doing a good job of keeping children safe. This was at a time when the system was so dreadful it had become the only system in the nation placed in receivership and run directly by a federal court. Arizona also got a passing grade for safety on its first CFSR. There, again, I could have used this to my advantage when, in 2003, the Governor started a foster-care panic that still hasn't ended. But children in Arizona were not safe then. They are even less safe now, thanks to Arizona's perennial panic, and most recently, thanks to some particularly shoddy journalism from a newspaper in Tucson – but that's a story for another day.
The major reason CFSRs are worthless – or worse - is that the CFSR ratings are based largely on an examination of only 65 cases from three counties in each entire state. The margin of error in a sample that small is so huge that it is impossible to draw meaningful conclusions. For example, based on this tiny sample, the Alabama CFSR concluded that the state prevents children from being reabused 100 percent of the time. They're good in Alabama – but they're not that good.
Defenders of this inane process say that the CFSR reviewers also look at statewide statistics. But when the state stats and the tiny sample produce different conclusions, the sample takes precedence. And the state data have flaws of their own. For starters there's the fact that one state, Kansas, has found a way to cheat – exploiting a loophole to keep thousands of foster care placements out of the statistics. (See NCCPR's report on Kansas child welfare and these previous posts: Turning foster children into unpersons , What Dorothy learned and Kansas forgot, It's not Oz, it's Kansas)
Another problem is the way statewide performance is measured. For example, one measure is average length of stay. That actually gives an advantage to lousy systems that take children at the drop of a hat and then "throw them back" in a month or two. States that do a good job at avoiding taking children in the first place are penalized, because their average length of stay is almost certain to be higher. And, in fact, Alabama lost out on this measure.
This is only the start of what's wrong with the process. Other problems include the ability of an administration to politicize the process – and there is evidence the Bush Administration did just that in one state. There is much more in NCCPR's full analysis, The Trouble With CFSRs which I've recently updated.
None of this should be taken as criticism of the newspaper that wrote the editorial. The paper is in fact one of the smartest in the nation when it comes to child welfare – more than a decade ago, they may well have saved the Alabama reforms when a lousy governor was trying to sabotage them. Rather, this is an appeal to all reporters to temper their understandable love of a scorecard with some checking into how the score was calculated.
