Last year I made a number of predictions for 2016 to see how well calibrated I am. Here is the results:
| Prediction | Correct? |
| No nuclear war: 99% | 1 |
| No terrorist attack in the USA will kill > 100 people: 95% | 1 (Orlando: 50) |
| I will be involved in at least one published/accepted-to-publish research paper by the end of 2015: 95% | 1 |
| Vesuvius will not have a major eruption: 95% | 1 |
| I will remain at my same job through the end of 2015: 90% | 1 |
| MAX IV in Lund delivers X-rays: 90% | 1 |
| Andart II will remain active: 90% | 1 |
| Israel will not get in a large-scale war (ie >100 Israeli deaths) with any Arab state: 90% | 1 |
| US will not get involved in any new major war with death toll of > 100 US soldiers: 90% | 1 |
| New Zeeland has not decided to change current flag at end of year: 85% | 1 |
| No multi-country Ebola outbreak: 80% | 1 |
| Assad will remain President of Syria: 80% | 1 |
| ISIS will control less territory than it does right now: 80% | 1 |
| North Korea’s government will survive the year without large civil war/revolt: 80% | 1 |
| The US NSABB will allow gain of function funding: 80% | 1 [Their report suggests review before funding, currently it is up to the White House to respond. ]
|
| US presidential election: democratic win: 75% | 0 |
| A general election will be held in Spain: 75% | 1 |
| Syria’s civil war will not end this year: 75% | 1 |
| There will be no NEO with Torino Scale >0 on 31 Dec 2016: 75% | 0 (2016 XP23 showed up on the scale according to JPL, but NEODyS Risk List gives it a zero.) |
| The Atlantic basin ACE will be below 96.2: 70% | 0 (ACE estimate on Jan 1 is 132) |
| Sweden does not get a seat on the UN Security Council: 70% | 0 |
| Bitcoin will end the year higher than $200: 70% | 1 |
| Another major eurozone crisis: 70% | 0 |
| Brent crude oil will end the year lower than $60 a barrel: 70% | 1 |
| I will actually apply for a UK citizenship: 65% | 0 |
| UK referendum votes to stay in EU: 65% | 0 |
| China will have a GDP growth above 5%: 65% | 1 |
| Evidence for supersymmetry: 60% | 0 |
| UK larger GDP than France: 60% | 1 (although it is a close call; estimates put France at 2421.68 and UK at 2848.76 – quite possibly this might change) |
| France GDP growth rate less than 2%: 60% | 1 |
| I will have made significant progress (4+ chapters) on my book: 55% | 0 |
| Iran nuclear deal holding: 50% | 1 |
| Apple buys Tesla: 50% | 0 |
| The Nikkei index ends up above 20,000: 50% | 0 (nearly; the Dec 20 max was 19,494) |
Overall, my Brier score is 0.1521. Which doesn’t feel too bad.
Plotting the results (where I bin together things in [0.5,0.55], [0.5,0.65], [0.7 0.75], [0.8,0.85], [0.9,0.99] bins) give this calibration plot:
Overall, I did great on my “sure bets” and fairly weakly on my less certain bets. I did not have enough questions to make this very statistically solid (coming up with good prediction questions is hard!), but the overall shape suggests that I am a bit overconfident, which is not surprising.
Time to come up with good 2017 prediction questions.
Nice post, Anders. A quibble: if you’re going to be mathematical about it (and why not), shouldn’t you weight the predictions according to importance? Shouldn’t Trump and Brexit have more impact on the calibration than, say NZ changing its flag?
Your reasons for mistaken forecasts will not be the same for every case.
Some will just be bad luck – Nobody expects the Spanish Inquisition! 🙂
And some will be much more significant than others.
Being wrong on Brexit and Trump is an important indicator that most of the media and politicians have lost contact with about half of society.
Evidence for supersymmetry: 60%
Really? What were you thinking? How many times have people said: this time we’re going to get evidence for supersymmetry? 5 times? Shouldn’t this have caused a crisis of model-checking and caused you to reevaluate the relevant energy scale? Is there any reason to think that it is accessible? Other than wishful thinking?
(I am not talking about the question of whether supersymmetry is true! I am assuming that you are 100% committed to supersymmetry.)
…
This reminds me of the history of heliocentric models.
Aristarchus (c250BC) measured the size of the Sun. He found that it was 10x the diameter of the Earth. This convinced him that the Earth revolved around the Sun. He bit the bullet of parallax and concluded that the fixed stars were really far away.
What if someone, maybe Copernicus, had redone the measurement and found that actually the Sun was 10x bigger than that? But also that the necessary parallax was 10x bigger? Is this evidence for or against heliocentrism? I think that discovering that 10x bigger distances are necessary should increase the scale of plausible distances by more than 10x.
(The first person to question the ancient measurements was Kepler, but at that point he and Galileo had other evidence for heliocentrism and so this didn’t have much affect on the debate.)