I don't have the patience or energy to tear into this in depth, but today's Right Wing Histrionic Talking Point of the Day is less amusing and more infuriating than usual. Don't you know, the AP poll that gave Obama a 60% approval rating is a result of "poll cooking" by the AP. According to the polling experts at Newsbusters. And comment sections on your local newspaper website.
Apparently the fact that the sample in the AP poll (n = 1001) did not contain the exact same number of Republicans and Democrats – that's what polls are supposed to have, after all! – is de facto evidence that AP is rigging the numbers. The poll in question had 46% Democratic identifiers and 29% Republicans. The chicanery is obvious. Pant-shitting rage ensues.
The response is unsurprising, since that is exactly how someone who knows dick about polling would interpret this.
The 46% is just about what we would expect for Democrats + Leaners. The 29% for Republicans + Leaners is lower than I would expect. In a different AP poll conducted this spring, the split was 43% D to 40% R. That's the kind of variance one gets when taking a random sample of 1000 people in a country with 200,000,000 adults. Random sampling is absolutely fundamental to polling, and the word "random" means that sometimes we will get results that are not precisely what one would expect.
Let's say there are 100,000 poker chips in a bag – some unknown mix of red and black. We mix them up so they are completely random. We don't want to count all of them so we select 500 at random to estimate the red-black mix. We find 300 black and 200 red. Therefore we conclude that the bag contains 60% black and 40% red, plus or minus a margin of error of about 5% (i.e. we are 95% certain that black chips are between 55% and 65%, but the best guess is around 60%). That is how a random sample works.
Now let's say that you call 1000 people using a random phone dialer, 600 of whom are "red" and 400 of whom are "black". But you're convinced that the bag actually contains more black chips, so you pull out a few hundred more chips, throw out all the new red ones, and add more black chips to the original sample. That is how a manipulated, biased, and egregiously fucked-with sample works.
If 35% of the population is Republican and 40% are Democrats, we would EXPECT a random sample to have 35% R / 40% D. Right? Right. And if I flipped a quarter 100 times we would EXPECT to get 50% heads, right? Right. But will I get exactly 50 heads every time I flip a quarter 100 times? Nope. I might get 51. Or 47. If I do it enough times I might even get a wacky result like 37 heads. It's random. Raaaaaaaaaaandom. That means each outcome is totally independent of the one prior. A probability is merely an expectation. A random sample of 1000 people in an electorate with 200,000,000 voting-age adults is going to vary from our expectations, sometimes considerably.
That's why polls have…margins of error. The AP poll in question has a MoE of +/-4.4%. The MoE creates a 95% confidence interval, so the data actually tell us that we are 95% certain that Obama's approval is between 56% and 64%. There is only a 5% chance that his rating is higher or lower than that. Now, if reality is on the low end of that interval, would 56% really be so surprising? He has been hovering at 50% for a while, occasionally popping up to 51-52 or dropping to 45-48. A little bin Laden bump to 55-58 wouldn't be so shocking, would it?
It would be if you're Newsbusters and, among other things, you don't understand a random sample or margins of error.
Idiots. The lot of them.
eau says:
That's why we love you ed – for your forgiving nature and effortless diplomacy. Well, that and phrases like "pant-shitting rage".
Middle Seaman says:
Thanks for 1st class is Statistics for Dummies.
They are not idiots; they are evil.
wetcasements says:
Math. Logic. Reality-based evidence and conclusions.
These are anathema to the GOP.
RandyH says:
I love the name of your website – "Gin and Tacos." Tonight I am drinking Gin (and Tonic) and I am eating really awesome Tacos. They are great companions. Thank You for the constant suggestions to do such magical food/drink combos.
1st timer says:
I would wager that if 1,000 random adults were asked, they would want to hear more about pant-shitting. I know I do. I would think that "both sides do it" gets as old to me as everybody else sooner or later. Also.
Xynzee says:
@wetcasements: is that meant to be factual statement?
cromartie says:
Jesus, can't they just post something from their in house pollster, Rasmussen, and get on with it?
Alex says:
Seems to me that significantly fewer people always identify as Republican, especially if you're not restricting your sample to registered or likely voters. Here's a chart from 2009 (just what showed up in the top of a google search) showing similar ratios averaged over a lot of polls: http://www.huffingtonpost.com/2009/04/30/party-id_n_725948.html
Not a surprise. Not out of line with other polling.
FMguru says:
I thought pollsters often reweighted things when their sample population was out of wack with the (presumed) distribution of the whole population. If you're polling a district with a 70/30 split between whites and blacks, and your sample comes up 50% African American (who give the President an overall 80% approval rating) and 50% white (who give him a 40% rating), then the raw result is 60%. But if you re-weight it so that the overall approval rating is proportionate to the population, then the 80% approval given by blacks makes us 30% of the final rating and the 40% approval of white makes up 70% of the final rating, and you get 52% approval rating. Isn't this why good pollsters show their cross-tabs and sampling details? So you can balance them against populations to counteract sampling anomalies?
Furthermore, party identitication isn't a fixed rock-solid trait for everyone, and there are a couple of percentage points worth of voters who aren't particularly tied to one party or another, but will say they are depending on their mood and current events. Obama whacks Osama means that more people are willing to call themselves Democrats this week (there's a natural bandwagon effect where people like to identify with winners), so the results may actually be an accurate reflection of the state of the population.
Weakly aligned Democrats have found a reason (this week) to call themselves Democrats, instead of saying they're "independents", We just might be looking at a "coming home" effect, which is likely to be transitory. They'll wander back into independent territory as soon as the news cycle changes and some other shiny object catches their eye and they fall back out of love with the D's. Their defining trait is fickleness, which is why they flicker between independent and Democratic. A mirror image is probably going on over the R side – weakly committed Republicans are more likely to call themselves independent this week, because Republican isn't looking like such a hot thing to be at this moment. The numbers will eventually shift back to their natural status, but for now, this poll is quite likely legit.
ladiesbane says:
The timing of the poll, the phrasing of the questions, the emotional triggers / logic progression of previous questions, the region or source or carny blow-off that tips respondents into the poll…these are some of the things I want to know whenever I hear poll results. There is a lot I don't know about polls, and it's fascinating. But nosing around the Newsbusters site left me queasy. Next time, I'll skip the link and take your word for it.
Rick Massimo says:
Remember; they're only idiots when they want to be.
When the (black) President turns over his birth certificate, they know all about Photoshop layers and the sequential numbering of birth certificates issued in Hawaii in 1961.
When one of our sainted troops in Iraq said that some of our other sainted troops in Iraq were committing atrocities, they knew all the properties of various bullet casings.
When a 12-year-old dares say that public health insurance helped his family, they knew all about real estate values, countertop materials and school tuitions.
But when it's bad news for a Republican, or good news for a Democrat, they're all "Fucking magnets, how do they work?"
Tim H. says:
Doesn't have to be a statistical thing at all, party affiliation is not so set in stone for a lot of people, and with Osama bagged, BHO and Democrats in general are looking better.
anotherbozo says:
This may only support the poll in question, but I myself never talk to pollsters. A response which likely is mirrored by people who
are moderately intelligent
are too busy to hang on the phone with strangers
are mistrustful of callers who ask to take "a few minutes of your time"
think Americans are polled to death as it is
have nuanced opinions that don't match with the neat categories presented
How many of us hangups would skew the result? Polls count only those willing to give their opinions over the phone and discriminate against the thoughtful and otherwise engaged. Which means they skew toward the right, right?
Kulkuri says:
@anotherbozo, you're on the right track and I would add polls are done with people with landlines, which can skew the result, usually to the right.
60% is a high number, but aWol's numbers were even higher after 9/11.
John says:
Notice they never have anything to say about party identification numbers when Rasmussen comes out with a poll.
Just sayin'.
Hobbes says:
People don't understand statistics. They don't. You can put explanations like this in front of otherwise intelligent people's faces and they'll still tell you you're wrong, because what you're showing them doesn't confirm their bias and therefore "make sense". And statistics should make sense, they reason, so if yours don't, you must be doing it wrong.
I got in a fight around Super Bowl time about whether it was traditional or not for a quarterback to be given the MVP award if he a) didn't blow and b) had two or more high-performing team mates. I argued that there was and ended up getting so frustrated I did eight pages worth of statistical analysis confirming that I was right. My opponent maintains I was doing it wrong, but has yet to get back to me with his corrections…
Bill says:
For someone who doesn't have the patience or energy to tear into this in depth, you sure spend a lot of words on the topic. However, given the undoubted skewed nature of the sample, the Obama fan club must be really worried that their icon couldn't even outbump Bush.
johnsmith1882 says:
@Rick Massimo
Bingo. They are schooled in the art of doublethink. They can convince themselves "To know and not to know, to be conscious of complete truthfulness while telling carefully constructed lies, to hold simultaneously two opinions which cancelled out, knowing them to be contradictory and believing in both of them."
@Bill
Totally, dude. When Obama thumps Newt Gingrich 56% to 43% next November, I'll be really worried about that bump in W Bush's approval ratings from 8 years ago.
Jason Fossella says:
Statistics are hard, because the nuances (95% confidence, margins of error) contradict a lot of natural cognitive biases. I never had a statistics course until sophomore year, and only because I was an engineer- liberal arts majors just did algebra and moved on. So most people don't understand these things.
That said, pretty sure the people at Newsbusters understand both statistics and that most people don't understand statistics.
JohnR says:
It doesn't matter any more. There is nothing, literally nothing, that will be accepted as either correct or wrong if it conflicts with the ideological requirements of the present GOP. Explanations, evidence, facts; these can be invented out of anything you want – fantasies, conspiracy theories, Tom and Jerry cartoons, bad dreams, it's all the same. The VSP will muse thoughtfully about it and it will become reality ('structural unemployment', anyone?). A handful of bloggers and a couple of token columnists will go all Mugato ("I feel like I'm taking crazy pills!") but they will be disdainfully sneered at by everyone who counts.
Southern Beale says:
Speaking of polls (sorta) Fred Thompson has hopped on board the National Popular Vote campaign which would award the presidency to the winner of the popular vote.
WhiskeyTango says:
Sometimes statistics don't even matter.
I'm 65, and I have a land-line. That may be why I seem to be getting a lot of pollster calls lately.
Last week I got one where the pollster said something like "Most people think that murdering babies is wrong. Do you think murdering babies is wrong?" I knew who I was talking to, so I said "No. I'm OK with murdering babies, sometimes."
End of interview. "Thank-you-for-your-time-goodbye."
Pretty sure they published the results of that poll as "98% Oppose Abortion"
WhiskeyTango says:
On another topic: I also got one of those calls where they ask if you're happy with the way the country is going.
What exactly the Hell do they hope to learn from that question?
I'm happy that Obama is President, and not Sarah Palin.
I'm infuriated at our unemployment rate, and with the Republicans for doing everything they can to keep it high.
Happy that we got a health care bill, sad that it doesn't do what it needed to do.
Terrified that the know-nothings in the House don't seem to know what not raising the debt limit will do.
Amazed and disappointed that so many people have been snookered into voting against their own interests.
And so on. I've started to ignore the nuances when asked that question, and answer "Yes", because when those polls are publicized, I think they are taken to mean "Do you like the President?"
leon says:
I'm with Ladiesbane on this one – I'll let Ed filter out the stuff from Newsbusters that's worth examining. Like I let Jon Stewart watch Fox News for me.
Desargues says:
…Idiots. The lot of them…
Actually, that depends. Let's see how probable that is. Since we're talking about 'News'max, the partition class should be
(1) they're lazy;
(2) stupid;
(3) argue in bad faith;
(4) the conjunction of (1) and (2), since they're not contradictory;
(5) a combination of (1) and (3);
(6) possibly (2) and (3);
(7) causes (1), (2), and (3) taken together;
(8) honestly mistaken.
So, without further evidence to decide, they're at most 12.5% likely to have made an excusable mistake with this.
The verdict I lean toward is JohnR's.
Fifth Dentist says:
Here's the thing:
Afte 9-11 Dubya had 90-plus percent approval, meaning that even Democrats almost universally supported him.
That is, the president on whose watch the worst terrorist attack ever happened in America had the support of the American people.
Not to blame Bush for the attack, but can you imagine if this weekend some swarthy people shot up a mall and killed three people? Would Republicans support the president? (If you answered "yes" to this question, please remove yourself from the gene pool immediately, preferably by mutilating your genitals with a rusty pair of pliers, and refer to the wingnut reaction to the Muslim doctor's rampage at Fort Hood).
A few days after the bin Laden killing I was talking with a Republican coworker. Obama deserved no credit, he said, for this. A few minutes later I worked into the conversation something about what if it had gone wrong. Without hesitation he said it would have been Obama's fault.
And there you have it: If by some chance the economy was now churning out 300,000 jobs a month, it would be due to Bush's tax-cuttin' and have nothing to do with Obama. That didn't happen, so it's all Obama's fault.
JazzBumpa says:
The thing that troubles me about polling is that people are not like black and red chips in a bag. There are all sorts of differences – regional, racial, age-based, and within even a single person: time of day, hunger, and degree of sobriety. Therefore, what assurance is there that a RAAANDOM sample is in any way representative of the population as a whole? I know margin of error is supposed to get at that, but implicit in margin of error is that the sample represents the population. But why do we assume that it does? A sample of 1000 from a population of 200,000,000 might be fine for a carload of dried beans. But not for a population that is heterogeneous. I've never seen anyone take on this issue.
The other thing that bothers me is that the structure of the poll and questions in it can encourage desired answers. It's a treasure trove for the dishonest.
Sadly,
JzB
Mark says:
"We find 300 black and 200 red. Therefore we conclude that the bag contains 60% black and 40% red, plus or minus a margin of error of about 5% (i.e. we are 95% certain that black chips are between 55% and 65%, but the best guess is around 60%). That is how a random sample works."
Actually, it isn't. In the scenario you outline, choosing 300 black and 200 red is extremely extremely unlikely. Somewhere in the neighborhood of 1 in a million.
And if you did draw 300 black and 200 red, the 95% confidence would be much tighter than 55%-60%. You'd have to draw a lot less chips to get such a wide interval. If 55% of the chips were black, you'd only draw 300 or more black chips somewhere between 0.5% and 1% of the time, which is to say, it is an unlikely enough coincidence that it does not fall within the 95% confidence interval. I don't have the time to do the calculations exactly, but based on some hand-wavy calculations, your 95% CI would be more like 58%-62%.
Your basic argument is correct. I don't post this just to be outrageously pedantic, just to show that you can mislead yourself about how likely various sampling errors are if you don't either have enough experience to know when something sounds wrong, or actually do the math.
Further reading:
http://en.wikipedia.org/wiki/Binomial_distribution
http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval
-Mark