Monday, May 30, 2016

Good survey questions versus bad survey questions

Continuing on yesterday's theme...

You are going to see a lot of polls.  Most are either worthless or only valuable in one way.  Clinton versus Trump vote intention polls are valuable, but ignore any one poll.  Look at the polling averages.  I just go with RealClearPolitics.  There are fancier versions, but you don't need them.  Everyone reading this knows who Nate Silver is.  Nate Silver has the best "guitar face" of any statistician on the planet.

What's "guitar face?"  Have you ever seen a guitarist contort his or her face while playing something?  It is usually something simple, like a string bend, but the contorted face is intended to convey the impression that the guitarist is doing something difficult.  On the other hand, go watch a youtube video of Django Reinhardt.  Django Reinhardt had two fingers on his left hand mostly paralyzed, so he started playing lead lines using two fingers.  With two fingers, he could do things that I will never be able to do with four.  And he did it with a stone-faced expression, usually with a cigarette dangling calmly from his mouth.  Go watch, listen, and learn.

Nate Silver is a poll aggregator.  He takes an arithmetic mean, adds a bunch of not-very-helpful bells and whistles, and ultimately does what anyone capable of computing an average can do.  But, the bells and whistles convince people who haven't studied statistics that he is accomplishing some grand feat.  If you want to know where the campaign stands, look at the simple polling average.  RealClearPolitics is fine.  Nate Silver is just contorting his face.

So, pay attention to vote intention polls, but only insofar as they affect the polling averages.

Now, the bad survey questions.

1)  Hypothetical questions.  There is no point in asking people what they would do if...  They don't know!  This is particularly prominent now in the hypothetical matchup polls.  You will see, for example, Berniacs claiming that Sanders is more electable because he does better in the hypothetical match-up polls against Trump.  The problem with that question is that nobody but the Democrats are paying attention to Sanders.  If Sanders were the nominee, Republicans would be attacking him rather than Clinton, and those numbers would change.  The guy calls himself a "socialist."  We don't elect those.  Not nationwide, anyway, and the Vermonters are just too stoned off their gourds read about the history of central planning.  But I'm just being unmutual again.

Basic point:  people can't tell you what they would do if...

2)  Why.  Ignore questions that ask people why they believe what they believe, or why they do what they do.  Two problems:  people don't know why, and if they did, they wouldn't tell you.  The most important political fact about a person is that person's party identification.  That tells you more about their political beliefs and voting patterns than any one trait.  And it isn't close.  So, what makes someone a Democrat or a Republican?  We don't really know.  According to a growing body of research started by John Alford, Carolyn Funk & John Hibbing, there's a significant genetic component!  Yes, really!  Will people tell you that?  Of course not.  People don't know why they do what they do, or think what they think.  When they are asked, they will rationalize.

And even if they know, they won't necessarily tell you.  People's motives aren't always pristine, but they don't want to admit it...

So, consider the following question: are you motivated by racism?  Of course not!  Those who know they are racist won't admit it, and many might not even know it!  Consider the phrase, "I'm not racist, but..."  You know what comes next:  something really, really racist.  But, if people grew up watching cross-burnings and lynchings, then even if they just have some negative stereotypes, they will compare themselves to the KKK and think of themselves as "not racist."

So, don't ask people about their motives.

3)  Candidate characteristics.  You can't ask people about the traits they associate with various candidates.  A rational person would assess candidates' characteristics and form an overall evaluation of the candidate on that basis.  But, most people are both stupid and irrational.  (I'm being unmutual again).  They do things ass-backwards.  They start with overall evaluations based on things like party, and then rationalize those evaluations by ascribing positive characteristics to the candidates they like, and negative characteristics to the candidates they don't like.  So, if I don't like a candidate, I am inclined to view that candidate negatively.  Whether I describe that candidate as stupid or dishonest is irrelevant because the specific negative attribute I ascribe is just a rationalization anyway.

Guidelines:  simple, direct, non-hypothetical questions.  Anything more complicated is likely to be misleading.

By the way, this post was prompted by a comment in another thread.  If there is something you want addressed, let me know.  And readership appears to be going up, so keep spreading the word.  The more readers there are, the more willing I am to write.

3 comments:

  1. Here is a reason I question the argument that Bernie would drop under fire.

    His livability is much higher than HRC or Trump.

    Even Republicans view him as likable (Except ardent Trump supporters), and he has been under fire for the past month, and yet his likability remains high.

    ReplyDelete
  2. Here is a reason I question the argument that Bernie would drop under fire.

    His livability is much higher than HRC or Trump.

    Even Republicans view him as likable (Except ardent Trump supporters), and he has been under fire for the past month, and yet his likability remains high.

    ReplyDelete
    Replies
    1. Republicans are only more likely to say they like him because they don't know anything about him. Once Republican leaders start bashing him as a guy trying to destroy America by turning it into the Soviet Union, that supposed advantage disappears.

      Republicans turned a bland, milquetoast, Constitutional Law professor with a kumbaya attitude into a secret Muslim-Atheist, Kenyan illegal immigrant, communist trying to institute death panels. And he didn't call himself a socialist.

      So, what really happens when the Republicans set their sights on Bernie? He crumbles to dust.

      Delete