Time of Flight System Design

Introduction

Many machine vision applications now require high-resolution 3D depth images to exchange or augment standard 2D imaging. These solutions rely on the 3D camera to supply reliable depth information to ensure safety, particularly when machines are operating close to humans. The cameras should also provide reliable depth information while operating in challenging environments, for example in large spaces with highly reflective surfaces as well as in the existence of other moving objects. Many products to date used low-resolution range-finder type solutions to provide depth information to augment 2D imaging. However, this approach has numerous limitations. For applications that take advantage of higher resolution 3D depth information, CW CMOS ToF cameras provide the highest performance solutions on the market. Some of the system’s features enabled by high-resolution CW ToF sensor technology are described in greater detail in Table 1. These system features also mean consumer use cases for example video bokeh, facial authentication, and measurement applications, as well as automotive use cases such as driver alertness monitoring and automated in-cabin configuration.

Table 1. Continuous Wave Duration of Flight System Features

System Feature	Enablers
Depth precision and accuracy	o Modulation frequency o Modulation schemes and depth processing
Dynamic range	o Readout noise o Raw frame rate
Ease of use	o Calibration procedure o Temperature compensation o Eye-safety monitoring
Outdoor operation	o Sensitivity at 940 nm o Illumination power and efficiency
2D/3D fusion	o Pixel size o Depth and 2D IR images
Multisystem operation	o In-pixel cancellation of interfering light o Camera synchronization

Continuous Wave CMOS Time of Flight Camera Overview

A depth camera is really a camera where each pixel outputs the length between your camera and the scene. One method to measure depth would be to calculate time it takes for that light to visit from a light source around the camera to some reflective surface and to your camera. This travel time is often referred to as the time of flight (ToF).

A ToF camera is comprised of several elements (see Figure 1) including:

a light source-such like a vertical-cavity surface-emitting laser (VCSEL) or edge-emitting laser diode-that emits light within the near-infrared domain. Probably the most commonly used wavelengths are 850 nm and 940 nm. The sunshine source is usually a diffuse source (flood illumination) that emits a beam of light having a certain divergence (aka, field of illumination or FOI) to light up the scene in front of the camera.

• a laser driver that modulates the intensity of the sunshine emitted by the source of light.
• a sensor having a pixel array that collects the returning light from the scene and outputs values for each pixel.
• a lens that focuses the returning light around the sensor array.
• a band-pass filter co-located using the lens that filters out light beyond a narrow bandwidth around the light source wavelength.
• is really a processing algorithm that converts output raw frames in the sensor into depth images or point clouds.

One can use multiple approaches to modulate the sunshine inside a ToF camera. An easy approach is to use a continuous wave modulation-for example, a square wave modulation with 50% duty cycle. In practice, the laser waveform isn't a perfect square wave and may look closer to a sine wave. A square laser waveform yields better signal-to-noise ratio for any given optical power, but also introduces depth nonlinearity errors due to the existence of high-frequency harmonics.

A CW ToF camera measures the time difference td between the emitted signal and the return signal by estimating the phase offset φ = 2πftd between the fundamentals of these two signals. The depth can be estimated from the phase offset (φ) and speed of light (c) using:

where fmod may be the modulation frequency.

A clock generation circuit within the sensor controls the complementary pixel clocks that respectively control the buildup of photo-charges within the two charge storage elements (Tap A and Tap B), as well as the laser modulation signal to the

laser driver. The phase from the returning modulated light could be measured relative to the phase from the pixel clocks (see right side of Figure 1). The differential the returning modulated light and to the phase of the returning modulated light in accordance with the pixel clock.

Using principles of homodyne detection, a measurement is made with multiple relative phases between pixel clock and laser modulation signal. These measurements are combined to look for the phase of the fundamental within the returning modulated light signal. Knowing this phase allows calculation of times it takes the light to visit from the source of light to the object being observed and to the sensor pixel.

Advantages of High Modulation Frequencies

In practice, there are nonidealities for example photon shot noise, readout circuit noise, and multipath interference that can cause errors in the phase measurement. Using a high modulation frequency reduces the impact of these errors around the depth estimation.

This is easy to understand if you take a simple example where there is a phase error εφ-that is, the phase measured through the sensor is φ = φ + εφ. The depth error will be:

Therefore, the depth error is inversely proportional to the modulation frequency, fmod. This really is illustrated graphically in Figure 2.

This simple formula explains in large part why ToF cameras rich in modulation frequency have lower depth noise and smaller depth errors than ToF cameras with lower modulation frequency.

One drawback of utilizing a high modulation frequency would be that the phase wraps around faster, meaning the number that may be unambiguously measured is shorter. The most popular method of getting around this limitation is to use multiple modulation frequencies that wrap around at different rates. The cheapest modulation frequency provides a large range with no ambiguity but larger depth errors (noise, multipath interference, etc.), while higher modulation frequencies are utilized together to reduce depth errors. A good example of this scheme with three different modulation frequencies is shown in Figure 3. The final depth estimate is calculated by weighting the unwrapped phase estimates for the different modulation frequencies, with higher weights being assigned to the larger modulation frequencies.

If the weights for each frequency are chosen optimally, the depth noise is inversely proportional to the root mean square (rms) from the modulation frequencies chosen within the system. For a constant depth noise budget, increasing the modulation frequencies enables reducing the integration time or the illumination power.

Other System Aspects Critical to Performance

There are plenty of system features to consider when creating a high perfor- mance ToF camera, most of which are covered briefly here.

Image Sensor

The image sensor is a key component in a ToF camera. The effects of most depth estimation nonidealities (for instance, bias, depth noise, and multipath artifacts) are reduced once the average modulation frequency of the system increases. So get a telephone the sensor has a high demodulation contrast (capability to separate photoelectrons between Tap A and Tap B) at high

modulation frequency (countless MHz). The sensor must also have a high quantum efficiency (QE) in the near-infrared wavelengths (for instance, 850 nm and 940 nm), so that less optical power is needed to generate photoelectrons within the pixel. Finally, a minimal readout noise aids in the dynamic range of the camera by allowing to detection low return signals (far or low reflectivity objects).

Illumination

The laser driver modulates the sunshine source (for example, VCSEL) at high modulation frequency. In order to maximize the amount of useful signal in the pixel for any given optical power, the optical waveform will need fast rise and fall times with clean edges. The combination of laser, laser driver, and PCB layout in the illumination subsystem are all critical to achieve this. Addititionally there is some characterization necessary to find the optimal optical power and duty cycle settings to maximize the amplitude of the fundamental in the Fourier transform from the modulation waveform. Finally, the optical power also needs to be delivered inside a safe manner with some safety mechanisms built-in at the laser driver and system level to ensure Class 1 eye safety limits are respected all the time.

Optics

Optics plays a key role in ToF cameras. ToF cameras have certain distinct characteristics that drive special optics requirements. Firstly, the field of illumination of the source of light should match the concept of view of the lens for optimum efficiency. It is also essential that the lens itself should have high aperture (low f/#) for better light collection efficiency. Large apertures can lead to other trade-offs around vignetting, shallow depth of field, and lens design complexity. A minimal chief ray angle lens design will also help lessen the band-pass filter bandwidth, which improves ambient light rejection and therefore improves outdoor performance. The optical subsystem should also be optimized for that desired wavelength of operation (for example, anti-reflective coatings, band-pass filter design, lens design) to maximise throughput efficiency and reduce stray light. There's also many mechanical requirements to make sure optical alignment is inside the desired tolerances for the end application.

Power Management

Power management is also critically important inside a high-performance 3D ToF camera module design. The laser modulation and pixel modulation generate short bursts of high peak currents, which places some constraints on the power management solution. There are some features in the sensor integrated circuit (IC) level that can help lessen the peak power use of the imager. There are also power management techniques that may be applied in the system level to help ease the requirements on the power source (for instance, battery or USB). The primary analog supplies for any ToF imager typically require a regulator with good transient response and low noise.

Depth Processing Algorithm

Finally, another large number from the system-level design may be the depth processing algorithm. The ToF image sensor outputs raw pixel data that the phase information needs to be extracted. This operation requires different steps which include noise filtering and phase unwrapping. The creation of the phase unwrapping block is a measurement of the distance travelled through the light in the laser, to the scene, and to the pixel, known as range or radial distance.

The radial distance is usually converted into point cloud information, addressing the information for a particular pixel by its real-world coordinates (X,Y,Z). Often, end applications only use the Z image map (depth map) instead of the full point cloud. Converting radial distance into point cloud requires understanding the lens intrinsics and distortion parameters. Those parameters are estimated during geometric calibration of the camera module. The depth processing algorithm may also output other information such as active brightness images (amplitude of the return laser signal), passive 2D IR images, and confidence levels, which can be used in end applications. The depth processing can be done around the camera module itself or perhaps in a host processor somewhere else in the system.

An summary of the various system-level components covered in this article is shown in Table 2. These topics is going to be covered in more detail in future articles.

Table 2. System-Level Components of 3D Duration of Flight Cameras

System-Level Component	Key Features
ToF Imager	Resolution, high demodulation contrast, high quantum efficiency, high modulation frequency, low readout noise
Illumination Source	High optical power, high modulation frequency, eye-safety features
Optics	High light collection efficiency, minimal stray light, narrow bandwidth
Power Management	Low noise, good transient response, high efficiency, delivers high peak power
Depth Processing	Low power, supports various kinds of output depth information

Conclusion

Continuous-wave time of flight cameras is really a powerful solution offering high depth precision for applications requiring high-quality 3D information. There are many things to consider to ensure that the very best degree of performance is achieved. Factors such as modulation frequency, demodulation contrast, quantum efficiency, and readout noise dictate performance in the image sensor level. Additional factors are system-level considerations, which include the illumination subsystem, optical design, power management, and depth processing algorithms. All of these system-level components are critical to achieve the highest precision 3D ToF camera system. These system-level topics is going to be covered in more detail in subsequent articles.