The cameras on the Pixel smartphones are the best on any smartphones since their introduction. Most companies utilize dual camera setup to capture portrait pictures. Whereas, Google Pixel smartphones do that better than those phones with a single camera itself. And for this year’s Pixel 3, Google leaned heavily on machine learning than ever before to improve depth estimation.
Previously, portrait pictures on the Pixel smartphones were the result of using ‘Stereo Depth’ with the help of PDAF – Phase Detection Autofocus pixels. It worked on the principle of obtaining pictures from different angles and estimating the depth of foreground and background objects. The depth difference between foreground and background is calculated with the help of the ‘Parallax‘ effect.
But this method could not deliver perfect portrait shots in a difficult situation as the calculation of depth would mostly have errors due to the limited viewpoints. Thus, to reduce those errors on the Pixel 3, Google trains a ‘convolutional neural network‘, written in TensorFlow. This will take the PDAF pixels as input and will predict the depth by machine learning.
To train this neural network, Google built their own custom rig known as ‘Frankenphone‘ which can carry five Pixel 3 smartphones. All these phones would capture pictures simultaneously within a tolerance of ~2 milliseconds. Thus, with the help of this rig, Google calculated high-quality depth by using ‘structure from motion‘ and ‘multi-view stereo‘.
To obtain a good portrait shot, the above-mentioned machine learning based depth estimation should run faster so that user don’t have to wait. This is taken care of by TensorFlow Lite which is a cross-platform machine learning model for phones and embedded devices. This model with a help of the GPU present in the Pixel 3 can process a full resolution, multi-megapixel PDAF image faster than before.
As for the end users, all they have to do is tap on the shutter icon and the magic happens in a short amount of time. Machine learning is what makes the Pixel cameras – ‘the best in the smartphone market‘.
