Capturing light in motion: New frontiers open for visualizing the world

Freezing high-speed processes in motion has been a recurring theme in photography since Eadweard Muybridge stunned the world in 1878. His The Horse in Motion series of photographs exposed that, in full flight, all four of a horse’s hooves are simultaneously off the ground, shattering the illusion of the graceful gallop. The public’s imagination was captured again when Harold ‘Doc’ Edgerton—using a stroboscopic flash that could fire a burst of light lasting microseconds—froze a speeding bullet in motion in 1964’s Bullet through Apple photograph. Since then, technological developments have seen real-time image-capture at several million frames per second expose many of nature’s fastest processes.

But capturing light in motion is a whole different ballgame, requiring a camera that can record scenes on the order of a trillion frames per second. So, when Ramesh Raskar from the Massachusetts Institute of Technology (MIT) Media Lab presented a 20-second video clip of a light pulse travelling in slow motion through a Coke bottle, bouncing off the cap, and reflecting back toward the bottle’s bottom during a 2012 TED Talk, it was a sensation. No one had recorded light in flight before. No one had been able to visualize the world at light speed.

For Daniele Faccio, it was also a lightbulb moment. “I was just blown away,” he recalls. “I didn’t think that was possible.” Faccio was at Heriot-Watt University in Edinburgh, UK, at the time, and had for several years been working on the idea of using light propagating through optical fibers to mimic black hole processes. These lab-grown black hole analogs can act as a testing ground for black hole theories, without the inconvenience of having to travel to an actual black hole. In 2010, he and his team had managed to create a laboratory proxy for Hawking radiation, a phenomenon by which a black hole should give off a stream of particles from its outer boundary. But whenever he presented the work, the one question Faccio was always asked was: “Can we see it?”

Inspired by Raskar, Faccio was determined to see his Hawking radiation analog moving along a fiber. The technique Raskar used—streak camera imaging, which measures the power of optical pulses over time—was “horrendously expensive,” so Faccio looked for alternatives. He found one down the road at the University of Edinburgh. “A group led by Robert Henderson was developing single-photon avalanche diodes (SPADs) to do fluorescence lifetime imaging,” explains Faccio. “They had the same kind of temporal resolution that Raskar was using with his streak camera, which made us think that maybe we could use SPADs for freezing light in motion. That’s when everything started to fall into place.”

SPAD cameras are a type of image sensor in which each pixel possesses an electronic element. Unlike CMOS sensors, where each pixel measures the amount of light that reaches it within a given time frame, in SPAD sensors each photon received by the pixel is converted into an electric charge, with the resulting electrons multiplying to form a single large electric pulse. This enables rapid and clear image capture free from signal noise, and it opens the door to high-precision distance measurements and photon time stamping.

With a prototype SPAD camera in hand, the Heriot-Watt team decided to test out the device, creating the first video of a laser pulse in motion bouncing off mirrors in free space. Seeing that the technique worked, they then wanted to find out what else it could do. Again, Faccio turned to Raskar for inspiration.

Raskar and his colleagues had already used streak cameras to reconstruct images of 3D objects hidden behind walls. They did so by using a technique known as non-line-of-sight (NLOS) imaging that analyzes light scattered multiple times in a surrounding environment. But generating the final images took several hours of acquisition time and an equal amount for data retrieval, making it impractical for real-world applications in which you need to find a hidden (often moving) object or person, like search-and-rescue or combat.

In 2015, Faccio proved SPAD cameras could perform NLOS imaging in real time, albeit without reconstructing the detailed 3D shape of the object. “We were the first to show two things: that SPAD cameras can be used for NLOS imaging, and that they can detect the position of a hidden moving target in real time,” he explains. “We showed that this technology could go from a very nice and amazing conceptual idea to something that could be done cheaply and at close-to video frame rates.”

He placed objects behind a wall and then set up a 32 x 32 pixel SPAD camera and laser pointing at a space on the floor to the side and beyond the wall. The imaging system recorded the time it took for laser light to scatter off the floor, reflect off the hidden object, and return to the floor, where it was picked up by the SPAD camera. Temporal and spatial information recorded by the SPAD camera then fed an algorithm to retrieve the target’s position and track its speed and position in real time when in motion.

Since then, others have developed the NLOS technique even further. For example, Gordon Wetzstein at Stanford University and colleagues built highly efficient algorithms that process raster-scanned single-photon data on a wall to reconstruct a 3D scene hidden around a corner. More recently, his team pushed SPAD data capture to its limits, in one project reconstructing objects hidden behind a thick slab of polyurethane foam, and in another imaging moving 3D objects through the keyhole of a closed door.

Using a wave migration algorithm, hidden objects are reconstructed by measuring echoes of light with a SPAD camera. Photo credit: Gordon Wetzstein

With these and many more experiments proving the ability of SPADs, by the time Faccio had got round to actually recording his Hawking radiation analog moving along a fiber in 2016—the original reason for using SPADs—he realized that the technique was useful for much more than just capturing light in flight. “We had moved on a bit,” he recalls. “Our attitude was: We’ve taken a video of our black holes moving down a fiber—amazing. Now what can we do?”

Both Wetzstein and Faccio see a host of applications for SPADs. “Think about a self-driving car that can see around corners and make safer decisions that way,” suggests Wetzstein. “Or the recent earthquake in Turkey and Syria: how helpful would it have been to have had imaging systems that can see inside the structures of collapsed buildings?” Adds Faccio: “It can also be used for bioimaging microscopy or for super-resolution imaging, looking through fog.”

Companies such as Hamamatsu, Sony, Canon, and STMicroelectronics have realized SPADs’ potential, too, bringing significant improvements in SPAD capabilities. “The big companies have reached a very high level of miniaturization and high performance,” explains SPAD engineering expert Federica Villa of the Polytechnic University of Milan, Italy. “They have their in-house technologies that allow them to achieve a very dense array of SPAD sensors, very many pixels, and also a very small form factor.”

These attributes are opening new commercial applications to explore. For example, Hamamatsu and Sony have developed SPAD depth sensors aimed at improving driver-assistance technology and self-driving vehicles. The sensors are intended for automotive lidar systems because they can determine, with high precision, the distance to an object by detecting the time of flight of a signal emitted from a light source until it returns to the image sensor, after being reflected by an object.

The main advantage of SPAD-based lidar is that it can detect single photons, which is the secret to high-precision depth measurement even from a long distance, allowing more accurate determination of the positions and shapes of vehicles and pedestrians.

SPADs could also soon be improving smartphone and security low-light camera performance, driving better medical and scientific imaging instruments, and enhancing augmented-reality experiences. “Hamamatsu is actively developing SPAD arrays with improved characteristics like high photon detection efficiency, lower dark count rate, and low crosstalk,” says Hamamatsu Applications Engineer Klea Dhimitri. “SPAD arrays are enabling a wide range of applications, from Raman spectroscopy to flow cytometry to lidar and quantum technologies.”

But these applications merely touch the surface of what SPADs could mean to science and society in the future. To fully explore the gamut of possibilities SPADs and other light-based technologies open, Faccio moved to the University of Glasgow in 2017 where his Extreme Light Group—now consisting of around 25 researchers—has since worked on a host of experiments harnessing light to answer fundamental physics questions and develop devices that benefit society.

Topics range from highly practical to extremely speculative, and not all involving SPADs. For example, some members of the team are developing a touch-free diagnostic and monitoring device for cardiovascular disease. The device captures and interrogates images from a high-frame rate CMOS camera of the back-reflected speckle pattern induced by a weak laser beam shone at a patient’s neck.

Another project aims to construct a quantum microscope. Working in collaboration with international partners, including Villa who built 24 ´ 24 pixel SPAD arrays for the project, Faccio and colleagues harnessed a quantum phenomenon called (HOM) interference to create microscopic images of surfaces. The researchers hope to refine their design to build a practical quantum microscope capable of resolving extremely small features like cell membranes or even strands of DNA.

One of the Extreme Light Group’s quantum microscopes. Photo credit: Kevin Mitchell

A more speculative research direction builds on this project. Faccio intends to use the HOM imaging setup to see if acceleration can generate entanglement. “Last year, we demonstrated that theoretically the curved space that you get from acceleration can actually generate entanglement,” explains Faccio. “When we finally do the experiment, it’s going to be a very important result because there’s an indirect link to gravity.”

All the fundamental forces of the universe are known to follow the laws of quantum mechanics, except one. Gravity is the odd one out, refusing to bend to quantum descriptions and instead demanding its own theory: general relativity. But general relativity and quantum mechanics don’t mix when subjected to the most extreme conditions the universe can produce. Therefore, the argument goes, there must be a truer description of gravity—quantum gravity—that we have just not yet found.

Faccio is not claiming positive results from his experiment would provide long-sought evidence for quantum gravity, but it could pave the way for a deeper understanding of gravity’s nature. “Could it demonstrate quantum gravity, strictly speaking? No,” he says. “Is it relevant? I think absolutely.”

On first inspection, there appears to be no clear bigger picture or unifying aim to the work in the Extreme Light Group. “We’re doing stuff from quantum in curved spacetime, quantum sensing, computational imaging, looking behind corners, looking down microscopes, cancer research,” says Faccio. In fact, he had to split the group into three labs—the Computational Imaging Lab, Quantum Physics Lab, and Human Augmentation Lab—to provide team members structure when they “were getting a bit lost” in the radical cross-disciplinarity and varied research themes they were pursuing.

But zooming out, a picture emerges of a subtle influence pervading all of the Extreme Light Group’s work for the past decade: a single TED Talk inspiring a researcher to capture and image light.

Benjamin Skuse is a science and technology writer with a passion for physics and mathematics whose work has appeared in major popular science outlets.