Chron reader rep, pollster defend DeLay poll
The Houston Chronicle and its pollsters continue to stand by a poll and reporting that have come under strong criticism from the local blogosphere. This weekend, Chronicle reader representative James Campbell penned a column in defense of the poll and the reporting, and Lone Star times has posted an email from Professor Murray defending the poll.
Before addressing those responses, I'm going to focus on the criticism of the Republican primary results that has been raised thus far (mostly at DeLay vs. World and at Lone Star Times):
1) The GOP primary results are unreliable because of flaws in the survey methodology
As Evan has pointed out (here and here), the screen of likely Republican primary voters is very weak. Murray/Stein asked the entire sample (N=560) if they were "likely" or "certain" to vote in the Republican primary, not whether they had voted in previous primaries. 38% (N=213) of registered voters said they plan to vote in the Republican primary. As Evan notes, in 2004, 0.65% of registered voters voted in the GOP primary. Simply put, the Murray/Stein subset can't be said to be reliably representative of Republican primary voters in the district.
2) The small sample size introduces more unreliability.
Because the sample of self-identified "certain" Republican primary voters (N=149) and self-identified "certain" or "likely" Republican voters (N=213) is small, the primary results are rendered even more questionable. Dr. David Hill contends that the (N=149) sample produces an error of +/- 9%.
3) Steve Stockman is identified as a "former Republican Congressman," which likely overstates his support.
These three factors combined render the poll's GOP primary results unreliable. That unreliability is not reflected in the Chronicle's original reporting, which stated:
Only about half of likely GOP primary voters now rate DeLay favorably, and only 39 percent are committed to voting for him in March.
This reporting is inaccurate, because those numbers simply aren't reliable. There's no getting around that at this point.
Keep that in mind as we turn to the Chronicle reader representative, who wrote a Sunday column in defense of the Chronicle's polling that neglected to address most substantive criticism of the poll, did not include any critical comments from other pollsters (say, local pollster David Hill), but did give Professor Murray a chance to explain why his poll is not flawed (Murray did actually concede the second point above).
Here is Campbell's definitive statement:
We invite readers to review the poll online. The dispassionate will agree that no matter how the numbers are analyzed, the emerging picture will show eroded support for DeLay in the district.
I conceded the eroded support in my headline to our first post on this topic: Recent news reflected by new Chron poll on DeLay.
It's not a surprise to those of us who analyze polls that a poll rushed to the field after a run of bad news for a candidate is going to reflect that bad news. However, the Chronicle's reporting went beyond that, as the excerpt above demonstrates. Because the poll's Republican primary numbers are unreliable, the Chronicle's reporting on them simply goes beyond what a "dispassionate" analysis of the poll will support.
Campbell continues:
Moreover, the charge that the poll's methodology was flawed is inaccurate. Neither the Chronicle nor the pollsters deliberately distorted poll results. The poll did not use the entire sample, including Democrats, to measure DeLay's support in the GOP primary.
The Chronicle and Murray/Stein are to be commended for posting additional information to the web in order to clear up the question of how DeLay's support in the GOP primary was measured, since the Chronicle's original postings did give the wrong impression (full crosstabs would have been even nicer, and would still be appreciated -- and this misunderstanding was not caused by critics who simply interpreted the data that the Chronicle first made available). However, that additional information revealed the three flaws noted above in the measure of GOP primary support, one of which Murray concedes is a problem:
"The only data reported in the paper was from people in Distict 22 who said they were certain and likely to vote in the Republican primary," Murray said. "If you have a sample with 560 people total, that's usually when you report that the margin of error is plus or minus 4 percent. But if you look at a subgroup, like certain or likely Republican primary voters, that's just 213 people. Well, the margin of error is larger, probably about 7 percent at that level. Sometimes people (pollsters) aren't as consistent in pointing out that for subgroups, the error margins are larger."
Murray conceded that critics may have a valid point that the poll should have noted when its margin of error was greater for subgroups.
Not "may." We do have a valid point. And the unreliability is even greater because of the fact that the screen for primary voters is poor, and Steve Stockman's support is almost certainly overstated by the way he is identified.
The poll's Republican primary results are simply not reliable, and the Chronicle reported them as if they were, inaccurately. That's not to say the Chronicle or Murray/Stein deliberately distorted the results. I haven't said that, and neither has Evan. Indeed, I'm more inclined to think this poll was rushed to the field quickly, with flaws, and that reporters and editors who don't fully understand survey research simply reported on basic numbers that are flawed. Mistakes do happen.
Indeed, that interpretation of the matter is supported somewhat by an email Professor Murray sent to KSEV-700 personality Edd Hendee. I'm going to quote liberally from the email to illustrate the point (emphasis is supplied by me):
This survey was not "my survey." The Houston Chronicle decided they wanted to do the survey late in the week of January 2nd after Congressman Delay announced he would not seek to regain the Majority Leader position. I was out of town at the Southern Political Science Association meeting in Atlanta from Wednesday thru Sunday of that week and thus was not contacted by the newspaper. Rather, they talked with Bob Stein, the Dean of Social Sciences at Rice, and he agree that he and I would work with the paper on this project, as we had done on numerous surveys in the past. He assumed I would have no objection but did not talk with me about the survey until I got back to town Sunday afternoon.
By that time, Bob had worked up a draft of the survey in consultation with the Houston Chronicle reporters and had talked to our director of the polling center at UH, Chris Mainka, to make sure we could work this unexpected poll into our survey schedule as we were beginning work on a local school survey. Mr. Mainka said we could do this on short notice and ordered the sample from a commercial vendor. I reviewed the draft and suggested a number of additional questions and some changes in question wording. The suggestions I made were not accepted because Bob and the newspaper wanted to get the survey done quickly and did not want to add questions such as the ones I put forward.
[snip]
Critics have also said the Chronicle should have polled a different population – say just Republican primary voters. As the sponsor of the poll, the newspaper makes the call as to whom they want to interview. In this case, it was the registered voters in the Congressman’s district – all registered voters – and that is their call as a news-gathering organization. They made this clear in their stories and that is their only obligation in this regard. If another news organization or entity wants to poll people who’ve voted in recent Republican primaries, fine, but that is not what the Houston Chronicle wanted to do and they paid for the poll.
The entire email is here.
That's all well and good, but it totally glosses over the substantive criticism raised by Evan. Because the screen used by the Murray/Stein poll was so weak, the sample size was so small, and Steve Stockman's identification almost certainly overstated his support, the poll's GOP primary results are not reliable, and the Chronicle's reporting of them is therefore misleading.
Otherwise, Murray confirms what we suspected -- the Chronicle and its reporters (more likely editors, but Murray said reporters) drove the poll questions and pushed a flawed poll into the field. If we take Professor Murray at his word, it was mainly Professor Stein who helped assemble the rushed, flawed poll for clients at the Chronicle.
BLOGVERSATION: DeLay vs. World, Houtopia, Isolated Desolation, Off the Kuff, About: Chron, Slampo's Place.
Posted by Kevin Whited @ 01/23/06 11:59 PM | Print |
Previous Entry | Home | Next Entry
ADVERTISING
DISCLAIMER
All content © 2004-09, blogHOUSTON and the respective authors.
blogHOUSTON.net is powered by Nucleus.
Site design and Nucleus customization are by Kevin Whited.