By Wayne Gillam / UW ECE News

In a first-of-its-kind achievement, researchers at UW ECE and Princeton University have shown that a camera containing a large aperture, ultra-flat optic can record high-quality color images and video comparable to what can be captured with a conventional camera lens. Shown above: an illustrative comparison of a standard refractive lens next to an ultra-flat optic developed by the research team. The ultra-flat optic is hundreds of times smaller and thinner. When this metalens replaces a conventional camera lens or stack of lenses, the savings in volume, weight, and device battery life can be substantial. Illustration by Liz Sabol / Princeton University

Cameras are everywhere. For over two centuries, these devices have grown increasingly popular and proven to be so useful, they have become an indispensable part of modern life. Today, they are included in a vast range of applications — everything from smartphones and laptops to security and surveillance systems to cars, aircraft, and satellites imaging the earth from high above. And as an overarching trend toward miniaturizing mechanical, optical, and electronic products continues, scientists and engineers are looking for ways to create smaller, lighter, and more energy-efficient cameras for these technologies.

Ultra-flat optics have been proposed as a solution for this engineering challenge, as they are an alternative to the relatively bulky lenses found in cameras today. Instead of using a curved lens made out of glass or plastic, many ultra-flat optics, such as metalenses, use a thin, flat plane of microscopic nanostructures to manipulate light, which makes them hundreds or even thousands of times smaller and lighter than conventional camera lenses. But there is one, big problem. A type of optical distortion known as “chromatic aberration” limits the ability of ultra-flat optics to produce high-quality color images when the optic has a large aperture — the opening in the lens that allows light into the camera. A large aperture increases light throughput to create images similar to what most cameras can produce today. For years, this fundamental limitation of ultra-flat optics has been seen by many as an impassable barrier.

That is, until now.

This research advance is an outgrowth of a collaboration between UW ECE and Physics Professor Arka Majumdar (left) and Felix Heide (center), an assistant professor of computer science at Princeton University. UW ECE Professor Karl Böhringer (right) also contributed to the effort. Photos by Ryan Hoover / UW ECE, and Steven Schultz / Princeton University

In a first-of-its-kind achievement described in a Nature Communications paper, researchers at UW ECE and the computer science department at Princeton University have shown that a camera containing a large aperture, ultra-flat optic can record high-quality color images and video comparable to what can be captured with a conventional camera lens. This notable accomplishment challenges a commonly held belief that sharp, full-color imaging would be impossible using a single, large-aperture metalens.

The ultra-flat optic the research team developed is a metalens that is only one micron thick. When affixed to its supporting substrate, it is still only 300 microns thick — about the width of four human hairs laid side by side. Altogether, it is hundreds of times smaller and thinner than a standard refractive lens. So, when this metalens replaces a conventional camera lens or stack of lenses, the savings in volume, weight, and device battery life can be substantial.

This ultra-flat optic could be applied to almost any camera, and it would be especially useful for any imaging system constrained by size or weight. Smartphone and laptop cameras are among the first applications for this metalens that come to mind, but it could be applied to a vast range of other technologies, such as cars, drones, or satellites that need lightweight imaging systems. Even medical instruments, such as endoscopes and angioscopes, could benefit from the smaller systems that this ultra-flat optic could enable, allowing physicians to see deeper inside the body to diagnose and treat diseases.

“Our work shows capability, what can be done with ultra-flat optics. I think our research pushes the field forward, and there will be a lot more of this type of work in the future.” — UW ECE Research Assistant Professor Johannes Fröch

This achievement is an outgrowth of a longstanding collaboration between the senior authors of the paper, Arka Majumdar, a UW ECE professor who holds a joint appointment in physics, and Felix Heide, an assistant professor of computer science at Princeton University. Majumdar, Heide, and their research teams have produced some dramatic advances in optics over the last few years, such as shrinking a camera down to the size of a grain of salt while still capturing crisp, clear images, and engineering a camera that can identify images at the speed of light. Majumdar’s lab team also has a strong track record of reimagining optics for smartphone cameras and other devices. This latest research advance emerged from a previous joint effort overseen by Majumdar and Heide, which was led by Ethan Tseng, a doctoral student in Heide’s lab who was also a co-author of this paper.

Lead authors UW ECE Research Assistant Professor Johannes Fröch (left) and Praneeth Chakravarthula, an assistant professor of computer science at the University of North Carolina at Chapel Hill (right)

Lead authors of the paper were UW ECE Research Assistant Professor Johannes Fröch and Praneeth Chakravarthula, an assistant professor of computer science at the University of North Carolina at Chapel Hill. Chakravarthula was a postdoctoral scholar in Heide’s lab at Princeton University when this research took place.

“Previously, it was assumed that the larger the metalens is, the fewer the colors there are that can be focused,” Fröch said. “But we went beyond that and beat the limit.”

“We treated this as a holistic system,” Chakravarthula added. “That allowed us to leverage the complementary strengths of optics and computation, where we didn’t design these different parts of the imaging system sequentially, but instead, we jointly optimized them to maximize performance.”

Other co-authors of the paper included UW ECE alums Shane Colburn, Alan Zhan, Forrest Miller, Anna Wirth-Singh, and Zheyi Han as well as former UW ECE postdoctoral researcher Quentin Tanguy and Jipeng Sun, a doctoral student in Heide’s lab. UW ECE Professor Karl Böhringer also co-authored the paper and contributed to the effort, supervising the students who fabricated the ultra-flat optic in the Washington Nanofabrication Facility. Böhringer is the director of the Institute of Nano-Engineered Systems, of which Majumdar is a faculty member.

AI-powered computation enables high-resolution images

In most imaging systems, multiple refractive lenses are used because a single lens cannot bring all colors into focus. This issue, specifying “chromaticity,” becomes exacerbated in ultra-flat optics. Many scientists and engineers even consider metalenses to be hyperchromatic because all the light cannot be brought into focus at a given point. This limits the ability of ultra-flat optics to have broader apertures and still be able to do well in terms of imaging visible light.

“We treated this as a holistic system. That allowed us to leverage the complementary strengths of optics and computation, where we didn’t design these different parts of the imaging system sequentially, but instead, we jointly optimized them to maximize performance.” — Praneeth Chakravarthula, assistant professor of computer science at the University of North Carolina at Chapel Hill

Previous to this research advance, it was not thought possible to build metalenses with large apertures that can produce a high-quality image. Most earlier efforts with metalenses were working with camera apertures that were less than one millimeter in size. In comparison, the aperture in the camera the research team engineered is one centimeter in size, significantly larger. The team demonstrated that with a strong computational backend co-designed with the optical hardware, even larger apertures are possible.

“People have tried purely physics-based or heuristic, handcrafted optical designs to address this issue, but in our work, we treat it as a computational problem,” Chakravarthula said. “We used AI tools to figure out what should be the shape of these lens structures and what should be the corresponding computation.”

The computational backend of the team’s optical system incorporated AI — a probabilistic diffusion-based neural network. This AI-powered backend takes in the data received from the ultra-flat optic and outputs images with lower haze, better color accuracy, more vivid hues, and better noise reduction. This all results in high-quality color images that are almost indistinguishable from what can be captured with a conventional camera.

“Previously, I was always considering problems from the optical side of the system,” Fröch said. “But this project really showed me that if you consider the whole system and then try to leverage the strength of each part — the optics and the computational backend — they can work synergistically to produce this really good image quality that we’ve shown here.”

Working toward even sharper images, new modalities

The camera prototype designed by the research team. This engineering prototype contains the ultra-flat optic with a one-centimeter lens aperture. Photo provided by Johannes Fröch

Next steps for the research team include further refining and improving image quality produced by their ultra-flat optic. They are also planning to explore different modalities for the optical system they developed that could be useful for augmenting human vision. These modalities involve capturing and working with information from light that is beyond what is visible to the human eye.

To illustrate, many animals, such as butterflies, can see far beyond the spectrum of light visible to humans and gain useful information from different characteristics of light, such as its polarization — the orientation of light waves as they travel through space. Animals use this information to find food, evade predators, and attract mates. In a similar fashion, humans can use light beyond what people are able to see to enable multimodal sensing for polarization or spectral information. An example of this is light detection and ranging, or LiDAR, which is currently being used in autonomous vehicles and in smartphones to assist with augmented reality, virtual reality, and depth-perception applications. The research team anticipates that their ultra-flat optic could be applicable to these sorts of technologies.

Commercialization of this ultra-flat optic is also a distinct possibility in the near future. Metalenses are suitable for mass manufacturing in foundries using nanoprint lithography, which makes the optics affordable and scalable. The team is currently talking with a UW professor in the ophthalmology department, who is interested in creating small, lightweight, hand-held devices that would be easier to use for eye inspections. Fröch also said there are startups that might be interested in commercializing this technology. He noted as well that the team’s research could open new avenues for others in the field of optics to explore.

“I think the overall takeaway here is that even when there are perceived limitations to solving a certain problem, it doesn’t mean it’s not possible to solve it,” Fröch said. “Our work shows capability, what can be done with ultra-flat optics. I think our research pushes the field forward, and there will be a lot more of this type of work in the future.”

The article, “Beating spectral bandwidths for large aperture broadband nano-optics,” was published March 28 in Nature Communications. The work was supported by the National Science Foundation, the Defense Advanced Research Projects Agency, a Packard Foundation Fellowship, a Sloan Research Fellowship, a Sony Young Faculty Award, an Amazon Science Research Award, and a Google Ph.D. Fellowship.