September 23, 2004

MARGIN OF ERROR MARGINALISED

Terry McCrann identifies a major flaw in opinion poll reporting:

The sheer, breathtaking and fundamental incompetence of the Canberra Press Gallery - and in particular its supposed 'leading lights' - is once again on public display.

Indeed, that incompetence is splashed almost proudly but almost certainly ignorantly across the front pages of their very own newspapers and on the network nightly news.

An equal share of the blame has to be taken by their editors and the media organisations they work for. And the opinion pollsters.

The generous interpretation is that, in the words of a former preacher: "They know not what they do." I suggest, stunningly, most of them really don't.

But that merges with a more basic reality: they don't want to let the facts get in the way of a good story. Even if it's just to footnote it.

There is almost no margin of error in this criticism. Because that's exactly the point -- you are almost never told about the margin of error in opinion polls.

McCrann is right. Unlike in the US, where margin of error is a feature of poll reporting, in Australia this is rarely mentioned. Read the whole thing. (And scope out all of McCrann's Herald Sun copy, now available online.)

Posted by Tim Blair at September 23, 2004 02:37 AM
Comments

You hit a pet peeve of mine.

No poll knows its margin of error. The MOE given on polls assumes proper statistical design, and polls suffer from all kinds of validity problems: improper randomization (cell phones and screening answering machines), mortality (people refusing to answer), and treatment interaction (asking a series of questions tends to make your sample homogeneous). And I'm not even taking into account potentially leading questions. With no controls on any of this, there is no way to calculate an MOE.

In defense of the polls, some of these are inherent and others are extrememly costly to avoid; so from a realistic standpoint, they're probably doing the best they can.

Posted by: Ron at September 23, 2004 at 03:52 AM

...in the words of a former preacher: "They know not what they do."

Is this just an odd, ironic aside, or does McCrann not know the origin of this quote?

Posted by: Angie Schultz at September 23, 2004 at 04:48 AM

I'm a reporter. Been one for 35 years. I went to J-school waaaaay back when the driving ethic in reporting was accuracy. You got it right or got lost.

That was before my generation took hold of the education system and politicized the living hell out of it. This is why news consumers rarely see serious, ongoing coverage of stories that fall outside the politically correct mold. Facts be damned, indeed.

McCrann's right. Reporters today really don't know what they do. They don't realize they've been politicized since kindergarten and they don't realize that in ways subtle and sledgehammer, they practice gotcha journalism against all they view as not one of them. They need to spin as much as they need to eat.

Only problem is, they (or most of the ones I know, anyway) have no clue they're doing it. They rationalize, justify, obfuscate or just sneer when their story angles are questioned. They went to school, and you're just a reader. They were there and you're just living vicariously through them.

And don't get me started about polls ...

Posted by: Gary at September 23, 2004 at 05:19 AM

Nice to see you having a good old bash at the Murdoch media and their useless poll reporting.

The MSM media barons of Australia like Murdoch and Packer have just gotta stop being so lazy in the way they let their local media outlets twist the poll results with no qualification of margin of error.

Damn it, you'd almost think that they were biased at the Tele and Oz with the way they keep playing down Labor's growing support. But hey, it looks like the penny dropped yesterday and the Great Man has sent word to the troops to start softening up for a switch to Latham.

In the global media world of Murdoch it's always important to back the local winner - no matter their political stripes.

Posted by: Media Joe at September 23, 2004 at 09:40 AM

I don't think he was making a partisan comment, Joe.

He was lamenting the treatment given to opinion polls by all facets of the Australian media - Muroch, Packer, Channel 7, ABC, Fairfax; you name it, they're all guilty of treating polls as being torn straight from the pages of the Holy Bible.

Here's a hint: in a two-party system, polls with a 3% MoE can get the margin wrong by up to 6 points. And that's on a good day. When things are close, a poll's margin of error means your guy is either ahead or behind, take your pick. When things aren't close, you don't need a poll to tell you there's trouble brewing.

Essentially, they're infotainment - Morgan, Galaxy, even Newspoll and AC Nielsen. Infotainment that's treated as Gospel by the news media, with no qualification and no acknowledgement that they might be wrong.

That's Timbo's point. How you managed to interpret it as some sort of partisan assault and respond in similar spirit is well and truly beyond me.

Posted by: Grand Old Elephant at September 23, 2004 at 10:39 AM

Timbo,
Leftwing ratbag Chris Sheil has had an idiot rabbitting on about this since its inception.


Better late than never.

By the why Brad De Long hasa good liitle wrap of this re-US elections today

Posted by: Homer Paxton at September 23, 2004 at 12:53 PM

Sure McCrann made some interesting points about polls but then turned his article into a petty Fairfax-bashing exercise. If he wants to retain any credibility he would highlight this pathetic analysis in his own paper.
Analysing poll trends is less problematic, but declaring a winner based on a crappy road trip is everything McCrann despises but just can't highlight.

Posted by: jockman at September 23, 2004 at 01:55 PM

Agreed. The road trip is 100% laughable as a forecasting tool. It made a fairly interesting read in that it's a little different to most other election coverage, but positing it as a crystal ball that tells us who's going to win the election is an insult to the paper's readership.

Posted by: Grand Old Elephant at September 23, 2004 at 02:10 PM

You guys have been sucked into treating the media seriously. Tim's got an excuse - he's one of them - but for the rest of Australia, the media and their reporting has a credibility rating lower than used car dealers. So what McCann says is taken for granted because most people don't believe most of what the media reports anyway. You only have to check the declining sales of newspapers and the low ratings of news reports to realise this essential fact of contemporary society: the public do not believe what they read/see/hear in the media.

Posted by: Freddyboy at September 23, 2004 at 02:15 PM

Good to see you waking up over here tim. Now have a cup of coffee and a slow look around. There's an election on.

Posted by: cs at September 23, 2004 at 02:43 PM

I was about to post a comment ....this being one of my pet hates.
Freddyboy beat me to it.
Nothing left to say on that point.
Lots of dross in blogdom, but lots af good stuff as well and it is sounding a deathknell for TV and print journalists who cling to the hope of being on the next Woodward and Bernstein gravy-train. Those two have a lot to answer for as they started the rot.
Not to question the right or otherwise of their position but the Barnum and Bailey promotion of themselves and media generally as judge, jury and executioner.
Often referred to as the "Fourth Estate" which presupposes an influence that can "govern" governments they are in effect more like "Fourth Rate" and that it being charitable.
People still read newspapers and watch curent affairs on TV....myself included....but like most people I am eager for news and hold the vain hope that the next issue or the next program will hold something worthwhile.
Nah!
Had its day.
Good riddance!

Posted by: JD at September 23, 2004 at 05:35 PM

"playing down Latham's support". Now that's laughable, he doesn't have any that's why he sidled up with Commie Bob the gum nut.

JWH is still the most popular choice and no MOE will alter that figure.

Im hoping Labor have the media giving them another eight point lead on election day just to see Latham whack on another sook fest when he gets hosed.

As for the media 'playing down" anything Latham does, man are you actually plugging your tele in when you sit in front of it.

I can't remember at any point in time where an opposition was given such a sweetheart run by the media. The man could rape mother theresa and they'd cover it up..

Posted by: scott at September 24, 2004 at 12:38 AM

Scott, the media in Australia is owned by Murdoch and Packer, with a few other wannabes. No private MSM outlets support Labor at the present time.

Moreover, the non-ABC media is staffed largely by right leaning hacks with a few token lefty types kept around to give the appearance of broad opinion. How you can conclude that MSM media like the Telegraph, Sun Herald, Channel 10, Channel 7, Channel 9, The Australian, The Fin Review are backing Labor is ridiculous. They are clearly favouring Howard. Only the ABC, SMH and The Age provide any sort of indepth balanced news reporting.

MSM is owned by big business and big business backs the conservative parties.

The same is largely true in America, but most right wing bloggers have convinced themselves that the MSM is somehow left wing. Get real. It's pro business and who ever is winning gets their support.

What's now happening in Australia is that Murdoch's little spies are getting hard evidence that Labor might just win this yet, and Murdoch will begin to back the new winner as that evidence solidifies in the days ahead.

Murdoch cannot afford to make enemies of Labor and he will do what he always does best and that is to back a winner.

Posted by: Media Joe at September 24, 2004 at 08:41 AM

Ever heard of Fairfax, talk radio and the ABC Joe?

Labor is on the nose in NSW mate and that's definite. We know what a pratt this Latham guy is. We know that he will kill of enterprise with his uselessly prescriptive nanny state tax and spend policies.

And if he can't win here he's not going to put his party in power.

Posted by: Toryhere at September 24, 2004 at 08:51 AM

Labor is on the nose in NSW mate and that's definite.

Didn't you see yesterday's Newspoll Tory? Had Labor ahead in NSW 52/48! No doubt Murdoch bias?

Posted by: cs at September 24, 2004 at 12:16 PM

Ron, you're totally right. But of course reporting even the theoretical sampling error would catalyze the public's learning process a bit. When you can get folks to realize that you get larger MoE every time you subset within a sample (e.g. women, 18-25, etc.), watch how fast their whole view of polls changes...

Posted by: DrSteve at September 24, 2004 at 01:29 PM

Only the ABC, SMH and The Age provide any sort of indepth balanced news reporting.

As soon as I can stop laughing, I will attempt to post an insightful response to this.

Posted by: Johnny Wishbone at September 24, 2004 at 03:22 PM

Hey Toryhere. I guess you're saying that Howard doesn't actually actually do any taxing and spending of his own.

Give it a break Toryhere, Howard is by far the worse tax and spend PM we have every had. Show me the evidence in the national treasury accounts that proves otherwise.

No Toryhere, it's your mate Howard who is taxing Australians more than any other government in Australian history. And when he's in a bind he chucks buckets and buckets of money at any sectional interest group that he needs votes from.

I don't understand why you aren't demanding that the surplus be paid back to us as tax cuts. We are talking about 25 billion in forward estimates over the next 5 years. But for some reaons it's okay with you for Howard to then spend that money on whatever pork he needs to bribe the likes of you with.

Posted by: Media Joe at September 24, 2004 at 06:39 PM

bugger the polls, what are the bookies saying ?
I'm at work, so this stupid SMARTFILTER won't let me look at http://centrebet.com.au

Posted by: martin at September 24, 2004 at 08:26 PM

Media Joe
Treasury figures show that Governement spending under Howard has decreased as an amount of GDP.

CS
Look at today's Herald poll. Oh yes, and look at centerbet. They've got Ken Ticehurst as a red hot favourite in Dobell, a seat he won by the narrowest of margins last time. If the ALP aren't expected to win back Dobell, they are hardly likely to win too many seats in NSW. In any case, the really interesting factor is that the ALP's primary vote has collapsed. The fact is that the polls are notoriously bad at getting the two party preferred vote right. And the COalition primary vote is much higher than the ALP's. Hence, why the Coalition has a healthy margin in the betting on Centerbet.

Posted by: Toryhere at September 26, 2004 at 12:25 AM

I'm a journo too and completely agree with everything Gary had to say. Polls are bollocks and I haven't seen one reported in accordance with the AJA code of ethics in years. Now, I only left J-school in 1998, and they were still drilling the ethics side of reporting into us. I think we must have devoted a month or two to reporting of polls and surveys alone - printing the exact wording of the survey question, the number of people surveyed, who commissioned the survey (not just who conducted it), etc - then I got out of J-school and find there doesn't seem to be an editor in the country who gives a stuff about the code of ethics we'd studied so hard and learnt to respect in the name of fair reporting. The media culture has gotten out of hand and I have grave fears there's no way back.

Posted by: Karl at September 27, 2004 at 06:00 PM