Nano-camera lens reveals the hidden mirror world of ...

If you want to learn more, please visit our website Nanocamera.

Come to the dark side, we have right-handed light

Reza Khorasaninejad

A new kind of camera lens is illuminating a hidden mirror world by revealing the “handedness” of light in the pictures it takes. The lens could one day be used to sort helpful drugs from their potentially dangerous mirror versions.

Many molecules come in two different types, a left-handed and a right-handed version. Although both types contain the same atoms, they are arranged as mirror images of each other and can have different chemical properties. Biological molecules like amino acids seem to favour a certain handedness, though we don’t know why.

Sometimes this mirror world can cause problems. For instance, the drug thalidomide was prescribed in the 1950s and 60s as a cure for morning sickness, but withdrawn after it was realised that the right-handed version led to birth defects. That makes it important to be able to tell twin molecules apart.

Advertisement

On the one hand…

One way to identify molecules with mirror versions – a property known as chirality – is to look at how they scatter light waves. The handedness is imprinted on the direction the waves vibrate, or their polarisation. But current techniques for measuring polarisation involve using multiple lenses and other optical elements like beam-splitters and filters, which can degrade the image quality.

Now Reza Khorasaninejad of Harvard University and his colleagues have come up with a single nanotechnology lens that can do the same job. The lens is made from a layer of titanium dioxide that has been etched by a beam of electrons into rows of pillars just 600 nanometres high, sitting on top of an ordinary sheet of glass.

In a row, each rectangular pillar is at an angle to the one before it, so that the orientation of the pillars along the line seems to rotate clockwise or anticlockwise. Alternating rows twist in opposite directions, creating two side-by-side images without the need for bulky optics. “We have huge control over the light shaping,” says Khorasaninejad. “The weight, size and compactness of the structure is very small.”

To test out the lens, the team took a picture of a Chrysina gloriosa, a beetle whose shell is known to reflect left-handed light (above). In the future, Khorasaninejad says they hope to improve the resolution of the lens to let them pick out left- from right-handed molecules, making it useful for developing safe drugs.

Journal reference: Nano Letters, DOI: 10.1021/acs.nanolett.6b01897

Differentiable metasurface proxy model

The proposed differentiable metasurface image formation model (Fig. 1e) consists of three sequential stages that utilize differentiable tensor operations: metasurface phase determination, PSF simulation and convolution, and sensor noise. In our model, polynomial coefficients that determine the metasurface phase are optimizable variables, whereas experimentally calibrated parameters characterizing the sensor readout and the sensor-metasurface distance are fixed.

Our learned, ultrathin meta-optic as shown in (a) is 500 μm in thickness and diameter, allowing for the design of a miniature camera. The manufactured optic is shown in (b). A zoom-in is shown in (c) and nanopost dimensions are shown in (d). Our end-to-end imaging pipeline shown in e is composed of the proposed efficient metasurface image formation model and the feature-based deconvolution algorithm. From the optimizable phase profile, our differentiable model produces spatially varying PSFs, which are then patch-wise convolved with the input image to form the sensor measurement. The sensor reading is then deconvolved using our algorithm to produce the final image. The illustrations above “Meta-Optic” and “Sensor” in (e) were created by the authors using Adobe Illustrator.

Full size image

The optimizable metasurface phase function ϕ as a function of distance r from the optical axis is given by

$$\phi (r)=\mathop{\sum }\limits_{i=0}^{n}{a}_{i}{\left(\frac{r}{R}\right)}^{2i},$$

(1)

where {a0, …an} are optimizable coefficients, R is the phase mask radius, and n is the number of polynomial terms. We optimize the metasurface in this phase function basis as opposed to in a pixel-by-pixel manner to avoid local minima. The number of terms n is user-defined and can be increased to allow for finer control of the phase profile, in the experiments we used n = 8. We used even powers in the polynomial to impart a spatially symmetric PSF in order to reduce the computational burden, as this allows us to simulate the full FOV by only simulating along one axis. This phase, however, is only defined for a single, nominal design wavelength, which is a fixed hyperparameter set to 452 nm in our optimization. While this mask alone is sufficient for modeling monochromatic light propagation, we require the phase at all target wavelengths to design for a broadband imaging scenario.

To this end, at each scatterer position in our metasurface, we apply two operations in sequence. The first operation is an inverse, phase-to-structure mapping that computes the scatterer geometry given the desired phase at the nominal design wavelength. With the scatterer geometry determined, we can then apply a forward, structure-to-phase mapping to calculate the phase at the remaining target wavelengths. Leveraging an effective index approximation that ensures a unique geometry for each phase shift in the 0 to 2π range, we ensure differentiability, and can directly optimize the phase coefficients by adjusting the scatterer dimensions and computing the response at different target wavelengths. See Supplementary Note 4 for details.

These phase distributions differentiably determined from the nano-scatterers allow us to then calculate the PSF as a function of wavelength and field angle to efficiently model full-color image formation over the whole FOV, see Supplementary Fig. 3. Finally, we simulate sensing and readout with experimentally calibrated Gaussian and Poisson noise by using the reparameterization and score-gradient techniques to enable backpropagation, see Supplementary Note 4 for a code example.

While researchers have designed metasurfaces by treating them as phase masks5,35, the key difference between our approach and previous ones is that we formulate a proxy function that mimics the phase response of a scatterer under the local phase approximation, enabling us to use automatic differentiation for inverse design.

When compared directly against alternative computational forward simulation methods, such as finite-difference time-domain (FDTD) simulation33, our technique is approximate but is more than three orders of magnitudes faster and more memory-efficient. For the same aperture as our design, FDTD simulation would require the order of 30 terabytes for accurate meshing alone. Our technique instead only scales quadratically with length. This enables our entire end-to-end pipeline to achieve a memory reduction of over 3000×, with metasurface simulation and image reconstruction both fitting within a few gigabytes of GPU RAM.

If you are looking for more details, kindly visit Camera Module Manufacturers.

The simulated and experimental phase profiles are shown in Figs. 1b and 3. Note that the phase changes rapidly enough to induce aliasing effects in the phase function; however, since the full profile is directly modeled in our framework these effects are all incorporated into the simulation of the structure itself and are accounted for during optimization.

Neural feature propagation and learned nano-optics design

We propose a neural deconvolution method that incorporates learned priors while generalizing to unseen test data. Specifically, we design a neural network architecture that performs deconvolution on a learned feature space instead of on raw image intensity. This technique combines both the generalization of model-based deconvolution and the effective feature learning of neural networks, allowing us to tackle image deconvolution for meta-optics with severe aberrations and PSFs with a large spatial extent. This approach generalizes well to experimental captures even when trained only in simulation.

The proposed reconstruction network architecture comprises three stages: a multi-scale feature extractor fFE, a propagation stage fZ→W that deconvolves these features (i.e., propagates features Z to their deconvolved spatial positions W), and a decoder stage fDE that combines the propagated features into a final image. Formally, our feature propagation network performs the following operations:

$$\begin{array}{l} \hskip-15pt{{{{{\mbox{Feature}}}}\; {{{\mbox{Propagation}}}}}\atop {\downarrow}}\\ \;\;\,{{{\rm{O}}}}= {f}_{{{{\rm{DE}}}}}\big(\; f_{{{{\mathrm{Z}}}}\rightarrow {{{\mathrm{W}}}}} \big(\;{f}_{{{{\rm{FE}}}}}\, ({{{\rm{I}}}}),\,{{{\rm{PSF}}}}\big)\big),\\ {{\uparrow}\atop{{{\mbox{Decoder}}}}}\quad {{\uparrow}\atop{{{\mbox{Feature}}}}\; {{{\mbox{Extraction}}}}}\end{array}$$

(2)

where I is the raw sensor measurement and O is the output image.

Both the feature extractor and decoder are constructed as fully convolutional neural networks. The feature extractor identifies features at both the native resolution and multiple scales to facilitate learning low-level and high-level features, allowing us to encode and propagate higher-level information beyond raw intensity. The subsequent feature propagation stage fZ→W is a deconvolution method that propagates the features Z to their inverse-filtered positions W via a differentiable mapping such that W is differentiable with respect to Z. Finally, the decoder stage then converts the propagated features back into image space, see Supplementary Note 5 for architecture details. When compared against existing state-of-the-art deconvolution approaches we achieve over 4 dB Peak signal-to-noise ratio (PSNR) improvement (more than 2.5× reduction in mean squared error) for deconvolving challenging metasurface incurred aberrations, see Supplementary Table 11.

Both our metasurface image formation model and our deconvolution algorithm are incorporated into a fully differentiable, end-to-end imaging chain. Our metasurface imaging pipeline allows us to apply first-order stochastic optimization methods to learn metasurface phase parameters \({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}}\) and parameters \({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{DECONV}}}}}}}}}\) for our deconvolution network fDECONV that will minimize our endpoint loss function \({{{{{{{\mathcal{L}}}}}}}}\), which in our case is a perceptual quality metric. Our image formation model is thus defined as

$${{{{{{{\bf{O}}}}}}}}={f}_{{{{{{{{\rm{DECONV}}}}}}}}}\left({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{DECONV}}}}}}}}},{f}_{{{{{{{{\rm{SENSOR}}}}}}}}}\left({{{{{{{\bf{I}}}}}}}}* {f}_{{{{{{{{\rm{META}}}}}}}}}\left({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}}\right)\right),{f}_{{{{{{{{\rm{META}}}}}}}}}\left({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}}\right)\right)$$

(3)

where I is an RGB training image, fMETA generates the metasurface PSF from \({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}}\), * is convolution, and fSENSOR models the sensing process including sensor noise. Since our deconvolution method is non-blind, fDECONV takes in \({f}_{{{{{{{{\rm{META}}}}}}}}}({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}})\). We then solve the following optimization problem

$$\{{{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}}^{* },{{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{DECONV}}}}}}}}}^{* }\}= \mathop{{{{{{\rm{arg}}}}}}\, {{{{{\rm{min}}}}}} }\limits_{{{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}},{{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{DECONV}}}}}}}}}}\mathop{\sum }\limits_{i=1}^{M}{{{{{{{\mathcal{L}}}}}}}}({{{{{{{{\bf{O}}}}}}}}}^{(i)},{{{{{{{{\bf{I}}}}}}}}}^{(i)}).$$

(4)

The final learned parameters \({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{META}}}}}}}}}^{* }\) are used to manufacture the meta-optic and \({{{{{{{{\mathcal{P}}}}}}}}}_{{{{{{{{\rm{DECONV}}}}}}}}}^{* }\) determines the deconvolution algorithm, see Supplementary Note 4 for further details.

Imaging demonstration

High-quality, full-color image reconstructions using our neural nano-optic are shown in Fig. 2 and in Supplementary Figs. 19, 20, 21, 22, 23. We perform comparisons against a traditional hyperbolic meta-optic designed for 511 nm and the state-of-the-art cubic meta-optic from Colburn et al.10. Additional experimental comparisons against alternative single-optic and meta-optic designs are shown in Supplementary Note 11. Ground truth images are acquired using a six-element compound optic that is 550,000× larger in volume than the meta-optics. Our full computational reconstruction pipeline runs at real-time rates and requires only 58 ms to process a 720 px × 720 px RGB capture.

Compared to existing state-of-the-art designs, the proposed neural nano-optic produces high-quality wide FOV reconstructions corrected for aberrations. Example reconstructions are shown for a still life with fruits in (a), a green lizard in (b), and a blue flower in (c). Insets are shown below each row. We compare our reconstructions to ground truth acquisitions using a high-quality, six-element compound refractive optic, and we demonstrate accurate reconstructions even though the volume of our meta-optic is 550,000× lower than that of the compound optic.

Full size image

The traditional hyperbolic meta-optic experiences severe chromatic aberrations at larger and shorter wavelengths. This is observed in the heavy red blurring in Fig. 2a and the washed-out blue color in Fig. 2c. The cubic meta-optic maintains better consistency across color channels but suffers from artifacts owing to its large, asymmetric PSF. In contrast, we demonstrate high-quality images without these aberrations, which are observable in the fine details in the fruits in Fig. 2a, the patterns on the lizard in Fig. 2b, and the flower petals in Fig. 2c. We quantitatively validate the proposed neural nano-optic by measuring reconstruction error on an unseen test set of natural images, on which we obtain 10× lower mean squared error than existing approaches, see Supplementary Table 12. In addition to natural image reconstruction, we also measured the spatial resolution using standard test charts, see Supplementary Note 10. Our nano-optic imager achieves a spatial resolution of 214 lp/mm across all color channels at 120 mm object distance. We improve spatial resolution by an order of magnitude over the previous state-of-the-art by Colburn et al.10 which achieved 30 lp/mm.

Characterizing nano-optics performance

Through our optimization process, our meta-optic learns to produce compact PSFs that minimize chromatic aberrations across the entire FOV and across all color channels. Unlike designs that exhibit a sharp focus for a single wavelength but significant aberrations at other wavelengths, our optimized design strikes a balance across wavelengths to facilitate full-color imaging. Furthermore, the learned meta-optic avoids the large PSFs used previously by Colburn et al.10 for computational imaging.

After optimization, we fabricated our neural nano-optics (Fig. 3), as well as several heuristic designs for a comparison. Note that commercial large-scale production of meta-optics can be performed using high-throughput processes based on DUV lithography which is standard for mature industries such as semiconductor integrated circuits, see Supplementary Note 3 for details. The simulated and experimental PSFs are shown in Fig. 3 and are in strong agreement, validating the physical accuracy of the proxy metasurface model. To account for manufacturing imperfections, we perform a PSF calibration step where we capture the PSFs using the fabricated meta-optics. We then finetune our deconvolution network by replacing the proxy-based metasurface simulator with the captured PSFs. The finetuned network is deployed on experimental captures using the setup shown in Supplementary Fig. 7. This finetuning calibration step does not train on experimental captures, we only require the measured PSFs. Thus, we do not require the experimental collection of a vast image dataset.

The proposed learned meta-optic is fabricated using electron-beam lithography and dry etching, and the corresponding measured PSFs, simulated PSFs, and simulated MTFs are shown. Before capturing images, we first measure the PSFs of the fabricated meta-optics to account for deviations from the simulation. Nevertheless, the match between the simulated PSFs and the measured PSFs validates the accuracy of our metasurface proxy model. The proposed learned design maintains consistent PSF shape across the visible spectrum and for all field angles across the FOV, facilitating downstream image reconstruction. In contrast, the PSFs of the traditional meta-optic and the cubic design proposed by Colburn et al.10 both exhibit severe chromatic aberrations. The red (606 nm) and blue (462 nm) PSFs of the traditional meta-optic are defocused and change significantly across the FOV. The PSFs for the cubic design exhibit long tails that leave post-deconvolution artifacts.

Full size image

We observe that the PSF for our optimized meta-optic exhibits a combination of the compact shape and minimal variance across field angles, as expected for our design. PSFs for a traditional hyperbolic meta-optic (511 nm) instead have significant spatial variation across field angles and severe chromatic aberrations that cannot be compensated through deconvolution. While the cubic design from Colburn et al.10 does exhibit spatial invariance, its asymmetry and large spatial extent introduce severe artifacts that reduce image quality. See Fig. 3 and Supplementary Note 8 for comparisons of the traditional meta-optic and Colburn et al.10 against ours. We also show corresponding modulation transfer functions (MTFs) for our design in Fig. 3. The MTF does not change appreciably with incidence angle and also preserves a broad range of spatial frequencies across the visible spectrum.

For more Noir Pi Camerainformation, please contact us. We will provide professional answers.