January 12, 2010
The Coalition of the End-of-the-World Unwilling
[0912.5480] The Black Hole Case: The Injunction Against the End of the World by Eric E. Johnson is a paper on the legal problems of handling existential risk and radical uncertainty. I like it because it cites our paper - Johnson has even read and understood it! Less can be said about the bloggers who misunderstood his paper and think he is suggesting the US should bomb Switzerland...
The real chain of confusion seems to be Johnson's paper -> Business Week -> democrat bloggers. The typical result of not really reading the sources but assuming one knows exactly what the previous source is talking about (and that they also have read the sources).
Actually, taking military action against somebody doing potentially world-endangering activities doesn't seem that wrong. We have just war theory, we accept intervening for humanitarian reasons (hard to get more humanitarian than trying to save all humans) and there have been wars and interventions to prevent development of merely GCR-level technology (Israel vs. Syrian and Iraqui reactors).
But handling radical uncertainty may be problematic legally and politically, as demonstrated by the recrimations over the Iraq WMDs. If Elbonia is trying to develop a Heim-theory reactor that would be a terrible threat only if a fringe physics theory was true, should the international community intervene? What if the sultan of Foobaria was seriously trying to summon Cthulhu?
It seems that sensibly allowing that our theories could be wrong puts a bit of probability mass on extreme and odd possibilities, and multiplied by the value of human extinction (which is at least as bad as 7 billion individual deaths, possibly much worse) this could become rather hefty. You might estimate that the chance that Heim theory is right may be one in 10,000 and the Elbonian project will have a 1/1000 chance of blowing up in this case. That makes the total badness at least 700 dead people. But these people are statistical people rather than real people, and the whole analysis hinges on very suspect priors. So we should expect both a lot of biases to come into play (new, unusual risks, low level of controllability, low probabilities, fictional example bias, etc) and people to diverge on their estimates - if they even take the issue seriously, given the anti-silliness bias and attention conservation we all have. We might agree that Elbonia looks more dangerous than Foobaria because most of us think elder gods are less likely than bad physics experiments, but we shouldn't trust that judgement much.
These biases are a valid reason to be sceptical of military interventions against low-probability existential threats. The question is whether the world community would be willing to intervene against a high-probability threat? At some point the probability would be high enough that some nation would unilaterally regard the threat as big enough to intervene despite the diplomatic costs. But below that there would be a region where the community would have reason to act as a group. Could we set up functioning legal norms for this?