(Originally a twitter thread)
When @fermatslibrary brought up this 1940 paper about why we have nothing to worry about from nuclear chain reactions, I first checked that it was real and not a modern forgery. Because it seems almost too good to be true in the light of current AI safety talk.
Yes, the paper was real: Harrington, J. (1940). Don’t Worry—it Can’t Happen. Scientific American, 162(5), 268-268.
It gives a summary of a recent fission experiment that demonstrate a chain reaction where neutrons released from a split atom induces other atoms to split. The article claimed this caused widespread unease:
“Wasn’t there a dangerous possibility that the uranium would at last become explosive ? That the samples being bombarded in the laboratories at Columbia University, for example, might blow up the whole of New York City ? To make matters more ominous, news of fission research from Germany, plentiful in the early part of 1 939, mysteriously and abruptly stopped for some months. Had government censorship been placed on what might be a secret of military importance ?
The press and populace, getting wind of these possibly lethal goings-on, raised a hue and cry.”
However, physicists were unafraid to of being blown up (and blowing up the rest of the world).
“Nothing daunted, however, the physicists worked on to find out whether or not they would be blown up, and the rest of us along with them.”
Then comes a good description of the recent French experiment in making a self-sustaining chain reaction. The resulting neutrons are too fast to interact much with other atoms, making the net number dwindle despite a few initial induced fission. And since it runs out, there is no risk.
There are some caveats, but don’t worry, scientific consensus seems to be firmly on the safety side!
“With typical French – and scientific – caution, they added that this was per haps true only for the particular conditions of their own experiment, which was carried out on a large mass of uranium under water. But most scientists agreed that it was very likely true in general.”
This article was 2 years before the Manhattan Project started, so it is unlikely to have been due to deliberate disinformation: it is an honest take on the state of knowledge at the time. Except of course that there was actually a fair bit to worry about soon…
(Foreword from apps.dtic.mil/sti/tr/pdf/ADA , commenting from the coldest part of the Cold War some decades later.)
Note that the concern in the article was merely self-sustaining fission chain reactions, not the atmospheric ignition by fusion discussed later in the Manhattan project and dealt with in the famous (in existential risk circles) report E. J. Konopinski, C. Marvin, and E. Teller, “Ignition of the Atmosphere with Nuclear Bombs,” Los Alamos National Laboratory, LA-602, April 1946. The idea that nuclear chain reactions could blow up the world was actually decades old by this time, a trope or motif that had emerged in the earliest years of the 20th century. Given that the energy content in atomic nuclei was known to be vast, that fissile isotopes occur throughout the Earth, and the possibility of a chain reaction at least conceivable after Leo Szilard’s insight in 1933, this was not entirely taken out of thin air.
Human engineering can change conditions *deliberately* to slow down neutrons with a moderator (making a reactor) or use an isotope where hot neutrons cause fission (the atomic bomb). The natural state is not a reliable indicator of the technical state.
It cannot have escaped the contemporary reader that this is very similar to many claims AI will remain safe. It is not reliable enough to self-improve or perform nefarious tasks well, so the chain reaction runs down. Surely nobody can make an AI moderator or find AI plutonium!
More generally, this seems a common argument failure mode: solid empirical evidence against something within known conditions cannot just be extrapolated reliably outside the conditions. What is needed is for the argument to work is (1) the conditions cannot be changed, (2) the result can be smoothly extrapolated, or (3) the impossibility needs to be relevant to the risk.
For nuclear chain reactions both (1) and (2) were wrong (moderators and plutonium). Arguments that AI will always hallucinate may be true, but that does not mean safety follows, since hallucinating humans (the results apply equally to us) are clearly potentially risky.
I think this is a relative to Arthur C. Clarke’s “failure of nerve” (not following extrapolation implications, often leading to overconfident impossibility claims) and “failure of imagination” (not looking outside the known domain or acknowledging there could be anything out there) he discusses in (1982). Profiles of the Future: An Inquiry to the Limits of the Possible.
Also, when reading the article I thought about my discussions with Tom Moynihan about how many tropes are earlier than the discoveries or events enabling them to become real – in the Scientific American article we already have the planet-destroying explosion and scientists “going dark” for military secrecy.
The funny thing is that this allows enlightened writers to poke fun at those naive people who merely believe in tropes, rather than the real science. The problem is that sometimes we make tropes true.
possible scenarios, one of which (
) will come about. They have probability 
. We allocate a unit budget of effort to the scenarios: 
. For the scenario that comes about, we get utility 
(diminishing returns).
. 
corresponds to even allocation, 1 proportional to the likelihood, >1 to favoring the most likely scenarios. In the following I will run Monte Carlo simulations where the probabilities are randomly generated each instantiation. The outer bluish envelope represents the 95% of the outcomes, the inner ranges from the lower to the upper quartile of the utility gained, and the red line is the expected utility.
case: we have two possible scenarios with probability 
and 
(where 
utility on average, but if we put in more effort on the more likely case we will get up to 0.8 utility. As we focus more and more on the likely case there is a corresponding increase in variance, since we may guess wrong and lose out. But 75% of the time we will do better than if we just allocated evenly. Still, allocating nearly everything to the most likely case means that one does lose out on a bit of hedging, so the expected utility declines slowly for large 
.
case (where the probabilities are allocated based on a flat 
), but we are not able to allocate perfectly. A more plausible model would give us probability estimates instead of the actual probabilities.
). The larger N is, the less likely it is that we focus on the right scenario since we know nothing. The rationality of ignoring irrelevant information is pretty obvious.
. In fact, we can mix in just a bit (
) of the true probability and get a fairly good guess where to allocate effort (i.e. we allocate effort as 
where 
is uncorrelated noise probabilities). The optimal alpha grows roughly linearly with 
in this case.
to the different scenarios we get better information about the probabilities, and can now reallocate. A simple model may be that the standard deviation of noise behaves as 
where 
is the effort placed in exploring the probability of scenario 
. So if we begin by allocating uniformly we will have noise at reallocation of the order of 
. We can set 
, where 
is some constant denoting how tough it is to get information. Putting this together with the above result we get 
. After this exploration, now we use the remaining 
effort to work on the actual scenarios.
and the gain is just the utility difference between the uniform case 
, which we know is pretty small. If C is small (i.e. a small amount of effort is enough to figure out the scenario probabilities) there is an optimal nonzero 
