Conduit for Commerce recently released its 2017 legislative rankings for Arkansas state legislators. Accountability is a wonderful thing, and we’re in favor of measures that promote it. Unfortunately, there are pervasive flaws in the way Conduit measures voting behavior, and these flaws seriously impair the ability of the public to glean useful information from Conduit’s rankings.
We have some experience at the Advance Arkansas Institute in assigning scores to legislative votes. We release a Freedom Scorecard after every biennial legislative session that considers legislators’ votes from a free-market, individual freedom, good-government perspective; we plan to release our newest set of ratings later this year. It is undeniable that undertaking a project of this sort involves a certain bias: some votes are selected, some votes are left out. As Conduit for Commerce notes, projects of this sort are not an exact science. But while bias is inherent in any legislative scoring effort, there are plenty of ways that unnecessary bias can be reduced. As long as the scoring body is transparent with its criteria, discloses its perspective, and is consistent in applying its criteria, then these scorecards can provide valuable information.
Unfortunately, in these areas, Conduit for Commerce falls short.
Perhaps the most fatal flaw of their rankings is that Conduit for Commerce does not disclose its methodology. For a scorecard to be useful, readers must know why a legislator received a certain ranking. Why are some votes better than others? Do some votes have more weight than others when a score is calculated? Suppose a legislator is absent and doesn’t show up for work — is that counted as a “no” vote? If nobody knows how an organization selected votes and determined how these votes are judged, it is impossible to use them to make an informed judgment.
The question of determining ranking is of special importance to the Conduit for Commerce numbers, because Conduit provides positive and negative scores for legislators. In some ranking systems, legislators receive a positive score for a “good” vote (from the scorer’s perspective), but do not receive a negative score for a “bad” vote. The worst score those who are judged can achieve is a zero. This has the benefit of highlighting when legislators take positive actions on issues of importance to the scorer. One might think that a zero score would be the same as somebody who never showed up to vote at all. But that apparently isn’t how the Conduit scores work — 20 out of 35 state Senators have negative scores. What could this mean? Well, to repeat an earlier point: nobody has any idea what it means, really, because Conduit’s methodology is completely invisible. One especially notable consequence of Conduit’s system is (as former state representative Nate Bell noted on Facebook) that the highest-scoring state senator in Conduit’s rankings missed half the votes that Conduit supposedly measured. Should you really get credit if you just don’t show up for work? Or maybe the minority of votes that the top-scoring senator actually cast were just (in Conduit’s judgment) much, much more important and had much greater weight. To repeat: nobody knows, because of Conduit’s hidden methodology.
Furthermore, the fact of negative scores underscores another issue with this ranking system. It looks as if Conduit for Commerce weighs one vote differently from another (although, to beat a dead horse, Conduit’s secrecy means nobody can be sure). We think some ‘yes’ votes are scored as positive, but other ‘yes’ votes are scored as negative. When the judge decides against judging each vote as equally important — or decides to dock points for a particularly “bad” vote — then that is one more dimension of raw, unnecessary subjectivity. Perhaps a bill given a heavier weighting in the scoring does reduce government spending (one of Conduit for Commerce’s criteria) more than a bill given a lighter weighting, or maybe that bill just happens to be on an issue that the judge feels strongly about. Who knows?
For legislative scorecards, it is impossible to have an objective weighting system that can differentiate the more important bills from the less important bills. It involves making one more biased judgment in a field that is already subjective. Unfortunately, this particular choice makes it easy to include biases in the system that create a preferred result in favor of one group of legislators or against others. Given the lack of transparency about scoring criteria, Conduit’s opting for opaqueness makes it even more pointless to rely on the scores in this ranking system.
A final problem is the scoring of committee votes. While these can be informative in looking at the actions of an individual legislator, using these committee votes is problematic when providing overall scores for legislators. Different legislators serve on different committees. They do not all take the same votes. Suppose Senator Jane Smith takes an important vote on a bill in committee that is scored, but her colleague Sam Jones — who shares the same electoral office and the same values — does not share the same committee assignment. In that case, Jones may receive a lower score through no fault of his own, even if the two senators would vote identically on all bills. There are indeed important votes in committee, but using those on legislative scorecards introduces an inconsistency that does not allow users to make a fair comparison between legislators. A system of legislative rankings is not materially different from having everyone run a hundred-yard dash at a track meet — and if you give a few people a ten-yard head start, you can expect this to affect everyone’s relative scores.
(I asked Brenda Taylor of Conduit for Commerce to explain some of these issues over the phone; she’s listed as the contact person for Conduit’s rankings. She was apparently too busy to speak to me, but she sent me a few texts. When I texted her “It looks like if you’re a legislator and you sit on certain committees, you have a different potential score than if you sit on other committees, right?”, she ultimately responded with a text that read “Committee votes were considered and they did not distort the result.” That statement could be a perfectly good answer to many questions, but it failed to answer the question I had actually asked. Her response was directly on point, however, when compared to a number of non-response responses I received from her over the span of 24 hours when I texted her with other questions about Conduit’s rankings, such as telling me (for example) that I should have listened to her interview on talk radio that occurred yesterday for her answer, that I should listen to her interview on talk radio that will occur tomorrow for her answer, that I should use the text that accompanied Conduit’s announcements for her answer, that I should text her tomorrow while she was on talk radio to receive her answer, that she couldn’t answer any questions because she had a client at the office, that she couldn’t answer any questions because she had a carpenter at her house, that she couldn’t answer any questions because someone was knocking at her door, that she was “tied up presently,” that I needed to send all questions by email and not by text, that I should call Joe Maynard and ask him my questions, and that she didn’t want to explain how Conduit’s ranking worked because she didn’t think I could understand her answers. I kind of wish she had provided that final response first; it would have saved us both a lot of time. I don’t want to jump to conclusions, but occasionally I got the idea that she just did not want to answer questions about Conduit’s rankings. — DG)
Other commentators have made the point that Conduit’s substantive judgments just seem wrong. It’s worth noting, for instance, that Conduit (which bills itself as an advocate for small business) continues to downgrade legislators who support tort reform — a judgment that many people who run small businesses would find eyebrow-raising. (Relatedly: as Rep. Charlie Collins noted on Facebook, Conduit gave him a worse score than ultraliberal legislative activists like Rep. Warwick Sabin and Rep. Vivian Flowers; apparently, Conduit thinks that he has lots to learn from them.) But Conduit is allowed to support or oppose any measure it wants: what is more relevant in our opinion is less the arbitrariness of their political judgments than the arbitrariness of their method of scoring votes.
To sum up, Conduit for Commerce’s legislative rankings have radical flaws that impair their usefulness. As it stands now, we don’t see much difference between Conduit’s rankings and the random results that you’d get if you called each legislator’s name and then picked a number out of a bingo basket. This is not to say that the organization’s goals are misguided. Judging legislators based on whether their votes limit government or reduce spending is always a good thing. But how you judge is also important. Fixing some of the flaws in Conduit’s ranking system — and making its workings transparent — would be far more effective in shining light on legislators’ actions. We hope to do that job right when we release our scorecard later this year.