Hidden variable theories were proposed to remove the randomness from quantum mechanics.
Einstein had toyed with the idea of hidden variables himself. He tried to modify quantum mechanics by synthesizing the classical wave and particle descriptions and by allowing the wave function take the role of a “guiding field”—guiding the physically real point-particles. However, he quickly lost interest, having convinced himself that the guiding field was still capable of exerting “spooky” non-local influences.
The great mathematician John von Neumann had done an early analysis of the hidden variable issue. He claimed to prove that hidden variables are necessarily incompatible with quantum mechanics. But this was a subject to certain general conditions that he supposed could be placed on hidden variable theories. These conditions seemed reasonable enough at first; but they became suspect over time.
In 1952 David Bohm developed a model (Bohm unknowingly extended the idea that Louis de Broglie had proposed in 1927 and subsequently abandoned) that showed the possibility of producing an internally consistent interpretation of the quantum mechanical formalism by introducing an additional ontological element in the description of the state of an individual system. In Bohmian mechanics a system of particles is described in part by its wave function. The evolution of the wave function is governed by the Schrödinger’s equation, but this only provides a partial description of the system. This description is completed by the specification of the actual positions of the particles (every particle has an actual, definite position, even when it’s not being observed). Changes in the positions of the particles are given by another equation, known as the “pilot wave” equation (or “guiding equation”) which expresses the velocities of the particles in terms of the wave function. In effect the configuration of a system of particles evolves via a deterministic motion choreographed by the wave function.
To understand the implications of Bohm’s model, let us revisit the double slit experiment. When a particle is sent into a two-slit apparatus its motion is controlled by a hidden guiding pilot wave that influences its location. The pilot wave passes through both slits simultaneously while each particle travels through one slit or the other. Interference in the pilot wave leads to the observed interference pattern. A measurement at the slits will collapse the pilot wave and reveal where the electron was all along.
The hidden pilot waves lead to a fully deterministic theory with the guiding field capable of exerting “spooky” non-local influences that characterizes a quantum system. One can calculate the location of each particle using the wave function provided the initial state of a system is known.
However, the Bohmian interpretation of quantum mechanics failed to gain traction among leading physicists including Einstein. The main criticisms had to do with the contrived nature of Bohm’s theory, as if it was deliberately designed to furnish predictions identical to conventional quantum mechanics. Critics argued that hidden variables are not necessary because of three reasons. First, the mathematical formalism of quantum mechanics is simpler without hidden parameters. Second, the simple formalism is fully capable of predicting results that are confirmed by experiments. Third, introduction of hidden parameters does not produce any verifiable supplementary predictions, so the question as to whether hidden parameters exist, fall in the domain of metaphysics rather than physics.
Suffering from a lack of experimental validation, it remained on the back burners of theoretical physics until John Bell’s inequalities (see page 14) provided a mechanism to empirically test the hidden variable hypothesis against the standard interpretation of quantum mechanics.