Friday, November 4, 2016

Why the race is close to tied, and why Nate Silver's method couldn't see that coming

As I see the race right now, we have a big mystery.  James Comey's decision to announce... something... created so much movement that it isn't clear how much we should weigh the polls before the Comey announcement.  What happens on Tuesday?  I don't know.

A couple of weeks ago, though, I was bashing Nate Silver for giving Trump too much of a chance.  So, what's the deal?  The deal is that I am entirely consistent, and the answer comes from Part V in the "Nate Silver is full of shit" series.

Back in that post, Clinton was sitting on a 5.5 point lead nationwide, and I chastised Silver for the pointlessness of combinatorical games with the electoral college when the national polls were that far apart.  In Part V specifically, though, I said that what needed to happen for Clinton to lose would be for either the polls to be systematically wrong in a way that Silver's methods had a hard time capturing, or for there to be an intervening event, which Silver's method couldn't predict.

That's kind of what happened, isn't it?  I even gave you a book recommendation in that post.  So, let's talk more about that book.  Tetlock's book, Expert Political Judgment, poses an uncomfortable challenge to those of us in predictive endeavors.  Perhaps history hinges so often on "black swan" type events that there is no point in even trying to make predictions.  A black swan event is a rare and unforeseeable event that changes everything.  Comey's announcement certainly looks to me like a black swan event. It was Anthony Weiner's computer!  Of course, many people are actually pretty good at predictions, and even if nobody could have foreseen the Comey announcement, Tetlock's analysis is all about the differences between those who do better than others.  One of my observations in Part V of the series was that Tetlock points out the weakness of reductive approaches to prediction.

Nevertheless, we can't forget about those black swans.  That was my point in Part V of the "Nate Silver is full of shit" series.  Silver's math wasn't about forecasting black swans.  Honk, honk, my flying friends.

6 comments:

  1. Black swans are reflected in polling errors. They were part of his math to the extent that they existed in previous elections.

    If 6 previous elections are your data, and polling done 2 weeks out has been off by 5 or more one time out of those 6, then that should be reflected in your model. As you get closer to the election, the number of polling swings that big likely drops....but not necessarily.

    Wleizen and (Stimson?) look at polling data and it actually gets LESS predictive over the course of the winter/spring, before really improving in accuracy in the summer (once we know the nominees). In their case, the lack of certainty over nominees is clearly messing with the stats, but it doesn't change what the stats are.

    Silver's math is partially about black swans. His model was seeing all those undecideds and saying that they might get "Comeyed" and swing one way. The undecideds often do go one way or another by 60/40 or even 70/30 ratios. But, there usually aren't a lot of them this late. This year, there were. A late swing could hurt HRC 2016 more than BHO 2012 because of that.

    ReplyDelete
    Replies
    1. No, polling errors are built on the assumption of random sampling error, and cannot factor in the possibility of a black swan. Wrong math. Factoring in that would have required a duration model with a parametric estimate of the probability of a black swan, declining over time, and his model didn't have that. No, his model absolutely isn't and never could be about black swans. Sampling error is completely different.

      Delete
    2. It's not just sampling error, though.
      It also factors in errors in predictions. So, if polls in previous elections have been off by 5 points, that's included in there. (At least, the way Silver talks about his model, he implies that) I don't think he uses sampling errors alone; I think he also has an imputed confidence interval using revealed polling errors in the past.

      Delete
    3. That still doesn't give you a probability of a black swan. That requires a separate hazard model, which absolutely isn't in his model, and he has no clue how to build one in. Frankly, the only way to do it is what you hate: prediction markets! Or, at least the kinds of group assessments that Tetlock puts together. Silver, though, has no clue how to do it and doesn't even try.

      Delete
  2. Now, if Silver's data lacks any black swans, it will have certainty where there might not be any.
    But, that's a data problem, not a model problem.

    (And I have problems with his model, but mostly because of how heavily he builds in "past-is-prologue")

    ReplyDelete
    Replies
    1. My point is that we don't know if the data, pre-black swan, mean anything at this point.

      Delete