New pattern squeezes more performance from microgrid polarization cameras
The camera in your cell phone employs a pattern of color filters, essentially trading spatial resolution for sensitivity to color. Though less familiar, the same idea applies to microgrid cameras that trade spatial resolution for sensitivity to polarization.
Polarization cameras have broad application in areas like remote sensing, medicine, and manufacturing. This is the story of conceiving, patenting, and commercializing a new microgrid pattern that makes the best trade between spatial resolution and polarization sensitivity, along with some other improvements along the way.
Since the 1990s, microgrid cameras have been built around a few variations of a 2 × 2 patterned array of polarizers. The advantage of a microgrid camera is the ability to capture all polarimetric and spatial information in a single snapshot. It also produces false polarization artifacts, especially near edges. The tradeoff is often worthwhile to prevent blurring due to camera or object motion and to keep the overall volume of the camera small compared to other techniques. In 2009, J. Scott Tyo, Bradley Ratliff, and Charles Lacasse at the University of Arizona provided a theoretical explanation for these artifacts as a form of interference between the polarized and unpolarized parts of the image.
Their paper was on my mind in 2013 as I listened to Keigo Hirakawa from the University of Dayton talk about color filter array design strategies for improving camera resolution through minimizing the risk of interference, or aliasing. I realized that Hirakawa’s strategies could be translated from color to polarimetric cameras by building on the theoretical foundation provided by Tyo, Ratliff, and Lacasse. I was a research engineer at the US Air Force Research Laboratory (AFRL) and Hirakawa was working with my colleague, Ken Barnard, as part of a summer faculty program. I was able to tell Hirakawa about my idea the next day. Within hours, he had completed the calculation that gave us the 2 × 4 microgrid array that we published as a journal article in 2014 and patented in 2018.
The 2 × 4 microgrid improves upon the 2 × 2 pattern by better separating the polarized and unpolarized parts of the image in a way that is analogous to separating two interfering radio stations so that both can be heard more clearly. Our idea prompted research teams around the world to publish their own variations on the concept, but all other proposed patterns involve tradeoffs. None of these proposed patterns, including ours, were turned into a prototype. Prototyping has been slow because the idea is nonintuitive: How can a pattern spread over a larger area lead to better resolution of smaller image features?
In 2021, David Chenault and his team at Polaris Sensor Technologies started working on a new generation of their 2 × 2 microgrid Pyxis camera. The original Pyxis was a 2 × 2 microgrid matched with a 640 × 512 microbolometer array and was the first commercially successful infrared polarization microgrid camera. Chenault presented us with the opportunity to add a 2 × 4 prototype into their much larger (1280 × 1024 pixel) next generation Pyxis program. Thanks to funding from Sean Mahoney, then with the AFRL Small Business office, we were able to make this happen.
The advantage of the 2 x 4 microgrid is evident in aliasing artifacts that manifest as falsely polarized, zipper-like edges around the two vehicles. Also, the lower overall contrast in the 2 x 2 image is the result of both optical crosstalk and higher noise. Photo credit: Daniel LeMaster.
The new, larger format Pyxis hardware development effort at Polaris was an ambitious attempt to push the state-of-the-art on several fronts. Smaller pixels mean that the microgrid array needs to be even closer to the microbolometers to minimize optical crosstalk. This is because optical crosstalk between detectors reduces sensitivity to weak polarization signals. Larry Pezzaniti, chief technology officer at Polaris, found that the 2 × 4 microgrid also minimizes the detrimental effects of the remaining optical crosstalk. Another win for the 2 × 4 microgrid!
In December 2023, Hirakawa, Joseph Raffoul, and I traveled to Polaris for a side-by-side comparison of the new 2 × 2 and 2 × 4 microgrid cameras. Raffoul, Hirakawa’s former student and now postdoc, had recently defended his PhD dissertation, which included a log-based de-mosaicking algorithm for microgrids (both 2 × 2 and 2 × 4). This algorithm squeezes out every bit of image quality from any microgrid in fewer processing steps than the conventional approach. Working alongside Chenault, Pezzaniti, and David Crandall at Polaris, we showed that the 2 × 4 microgrid did indeed increase effective resolution over an otherwise identical 2 × 2 camera—both in the lab and in real-world imagery.
We also found something unexpected. The underlying uncooled infrared detector array is subject to line noise. This random effect (not to be confused with fixed-pattern noise) is always present but can largely be ignored in regular microbolometer imagery because the signal strength of the image is much stronger than the line noise. In the microgrid imaging case, we are separating out a much smaller polarimetric signal that isn’t strong compared to the line noise. We found that the line noise signal interferes with the 2 × 2 polarimetric signal, while the 2 × 4 polarimetric signal is shifted away from the line noise and no interference occurs. Altogether, then, the 2 × 4 alleviates the detrimental effects of optical crosstalk, improves resolution, and moves the polarimetric signal away from harm by line noise. Three big wins, all coming from just rearranging the filter pattern on the detector array.
The net result is demonstrated in degree-of-linear-polarization (referred to as DOLP) images. DOLP is the ratio of the polarized to unpolarized components of the image and is a key discrimination feature in polarization imaging. The advantage of the 2 × 4 is evident in aliasing artifacts that manifest as falsely polarized, zipper-like edges in 2 × 2 images. The lower overall contrast in the 2 × 2 image is a combination of optical crosstalk, making the objects appear to be less polarized and therefore closer to the level of the noise, and in the additional horizontal line noise that cannot be separated from the true DOLP signal. These improvements are essentially free, the only difference in how these two images were collected and processed is in the physical patterning of the microgrid array, everything else is the same.
Polaris has completed a prototyping run of the new 2 × 4 camera and is accepting orders. The story of how this invention went from a flash of inspiration to a commercial product is incomplete without acknowledging the role of SPIE meetings in our success. Would I have appreciated the connection between Hirakawa’s work in color and Tyo’s work in polarization if I hadn’t heard him and his collaborators discussing the idea in presentations, over coffee breaks, and at lunch? Would Chenault, Pezzaniti, and I have confidence in each other if we hadn’t spent years building our relationships and sharing our work at SPIE meetings? Not likely on either count. SPIE meetings are about building relationships just as much as building your portfolio and strengthening your technical skills. Be sure to make the most of them.
Daniel A. LeMaster is an SPIE Fellow and senior scientist for sensors and perception at the US Department of Transportation.
