SUPPLEMENTARY INFORMATION

Transcription

1 DOI: 1.138/NPHOTON.234 Detection and tracking of moving objects hidden from view Genevieve Gariepy, 1 Francesco Tonolini, 1 Robert Henderson, 2 Jonathan Leach 1 and Daniele Faccio 1 1 Institute of Photonics and Quantum Sciences, Heriot-Watt University, David Brewster Building, Edinburgh, EH14 4AS, UK 2 Institute for Micro and Nano Systems, University of Edinburgh Alexander Crum Brown Road, Edinburgh EH9 3FF, UK To whom correspondence should be addressed; gg132@hw.ac.uk; d.faccio@hw.ac.uk. S1 Background subtraction and data processing As explained in the text, in a case where an object-free background is impractical or impossible to acquire, the median of histograms recorded at different times can be used to estimate the background. We show in figure S1 an example of a background calculated from the median of eight recorded positions. Figure S1a shows the histograms of a given pixel for the eight positions of the target mentioned in the main text. We can see that the peaks coming from the target are shifting in time when the target changes position, between 2.5 ns and 4.5 ns. The rest of the signal is coming from fixed objects: the walls and the ceiling. At each point in time, we record the median value of the 8 positions. Figure S1b shows the result of the median-calculated background (object-present background) compared to the object-free background. It gives a fairly good approximation of the background, especially for the peaks coming from a fixed object. In a real-life implementation, the object-present background can be calculated with the first few acquisitions. If there is no object moving, but a target is present, this algorithm will fail to detect the target; but as soon as the target starts moving, it will take around 15 seconds ( 5 acquisitions) to record a background and begin to accurately locate the target. The best approximation of a background will come from a target that is moving in both the x and y directions. However, we underline that during this initialisation period of 15 seconds, the camera is acquiring data which does provide information regarding the movement of the target: this data is simply less accurate with respect to the data acquired at later times, but nonetheless indicates target movement. To retrieve the position of the hidden target, we also need to know the position r i of pixel i on the floor. The camera is looking down at the floor (it is located at a 46 cm height above the floor and is imaging a region, on the floor, located 67 cm ahead), but still records a 32x32 square image. This image actually corresponds to a field of view that is trapezoidal and stretched both spatially and temporally, with respect to a squared field of view perpendicular to the line of sight of the camera. To correctly NATURE PHOTONICS 1

2 DOI: 1.138/NPHOTON.234 Signal from target a Signal from environment b Figure S1: Background subtraction. To avoid acquiring a background in controlled conditions, where we know the target is not present, we can rely on a few acquisitions where the target is at different positions. a) In the eight histograms shown here, the target was positioned at different places leading to a variation in its detected signal. b) The background calculated from the median of the histograms in a (object-present) is very close to the background acquired without the target present (object-free). retrieve the positions of a hidden target, we need to first determine the actual position ( r i = x i,y i ) of each pixel of the camera. To do so, we measure the dimensions of the field of view and its distance to the camera to reconstruct the actual shape of the field of view on the floor. In a real-life application, knowing the height and angle of the camera with respect to the floor will be sufficient to determine the geometry of its field of view and therefore know the positions (x i,y i ) of each pixel in the field of view. We also correct for temporal distortion of the recorded data: because the field of view is not perpendicular to the camera s line of sight, photons recorded at the top of the field of view take longer to reach the camera than the ones coming from the bottom. Again, knowing the geometry of our imaging system, we correct the measured t i accordingly. We then use the values of r i, t i and σ ti to retrieve the position of a hidden target, as explained in the next section. S2 Position retrieval The mathematical model we develop for locating objects around corners relies on the time histograms recorded by each pixel of the SPAD camera. These histograms give a probability distribution of arrival times for photons scattered by a hidden target. This time distribution can be mapped into a probability 2 NATURE PHOTONICS

3 DOI: 1.138/NPHOTON.234 SUPPLEMENTARY INFORMATION density in space for the target position. A spatial probability density is calculated for each pixel, and the product of all probabilities constitutes the joint probability density of finding the target at a specific position in space. We first treat each pixel independently. The scattered light from an object at position r o is detected at pixel i (located at position r i ) at time t i. The arrival time t i corresponds to the time the light takes to propagate (at speed c) from the laser position on the floor ( r l ) to the object and then from the object to the pixel: c t i = r o r l + r i r o (S1) Figure S2 illustrates the variables and relevant distances used in the calculation. Solving equation S1 for the object position r o gives infinitely many solutions lying on the surface of an ellipsoid defined by foci r l and r i : the possible positions in space that can generate a signal at pixel i at time t i lie on an ellipsoidal surface with evenly distributed probability. In the absence of any uncertainties in the measured signals, the resulting probability density of the object s location i ( r o ) calculated from the data collected by pixel i is therefore given by i ( r o ) { 1 if r o r l + r i r o = c t i otherwise. (S2) i (r o ) Laser spot Point i r o r l r i r o Figure S2: Retrieving the position of a hidden target For each pixel i, a signal is detected at time t i. t i is the time it takes for a pulse to propagate from the laser spot to the object ( r o r l ) and from the object to the pixel ( r i r o ). We determine a probability density in space i ( r o ) of where the object can be located to send a signal at pixel i at time t i. We write here with a proportional sign and will treat the normalisation issue ( S i ( r o )d r o = 1) at the end of the development. The function above can be represented in a simpler form by using ellipsoidal coordinates with foci r l and r i i ( r o ) δ(ε ct i ), where ε = r o r l + r i r o. (S3) NATURE PHOTONICS 3

4 DOI: 1.138/NPHOTON.234 In any real implementation, the histogram h i (t) recorded by any given pixel i will contain an uncertainty on the arrival time of the signal. As a result, the probability density i ( r o ) will no longer be a uniform ellipsoid, and the uncertainty in the time histogram can be mapped onto the spatial probability density as: ( ε ) i ( r o ) δ(ε ct)h i (t)dt = h i. (S4) c In our experiments, the signal recorded in h i (t) has a Gaussian form with a standard of σ ti. The uncertainty is originating from different sources, for example the jitter on the system and the finite size of the target. As a result of the Gaussian form of the recorded signal, the general expression of the probability density i ( r o ) expressed in S4 then becomes i ( r o ) exp [ (ε/c t i) 2 ]. (S5) 2σt 2 i Here t i is the mean arrival time registered by the pixel i and σ ti is its standard deviation. Figure S2 shows an example of the spatial distribution of i ( r o ). In our setup we aim to locate the object in x and y, on the plane of the floor, making the assumption that such an object is not moving considerably in the vertical direction. We therefore set a two dimensional search space at a given height close to ground level and r o takes the form r o =(x o,y o ). We take the height of search to be the height of the chest of our foam mannequin z o = 17 cm, as this is the region of highest reflectivity. In a real-life implementation, the height can be appropriately estimated based on the type of target we wish to track or locate. An error z in estimating this height will result in the worst case in an error r o of the same order in the determination of the target s r o coordinates, although it will be typically much smaller and decreases rapidly for objects that are further away: r o (z / r ) z. As mentioned above, the calculated probability densities i of each pixel are multiplied to obtain the joint probability density P ( r o ). However, there is a risk that a given pixel i will lead to an unsignificant fit of the signal and that the values of t i and σ ti will be unreliable. To avoid that these unreliable i ( r o ) affect the joint probability density by multiplying relevant densities by zero, we take the probability P i ( r o ) associated with the pixel i to be a linear combination of the probability density i ( r o ) and a uniform probability density P uniform that will prevent any point in space to be multiplied by zero, P i ( r o )=α i i ( r o )+(1 α i )P uniform. (S6) Here, α i is a coefficient between and 1, related to the reliability of the probability density P i ( r o ). Our choice of α i for each pixel depends on how well the probability density P i ( r o ) overlaps with the space in which we are searching the object. More precisely we set α i = A i /A where A i is the area contained in the search space where the probability density P i ( r o ) is over a certain threshold (half its maximum) and A is the area of the search space. Given a number of pixels n, each with associated probability P i ( r o ), the joint probability density P ( r o ) is defined as P ( r o )=N where N is a normalisation constant. n P i ( r o ). i=1 (S7) 4 NATURE PHOTONICS

5 DOI: 1.138/NPHOTON.234 SUPPLEMENTARY INFORMATION S3 Signal-to-noise ratio We measured the signal-to-noise (SNR) ratio for all eight positions of the target shown in figure 3 of the main text. We find that, for a fixed distance between the camera and the field of view, the SNR is proportional to 1/r 4 =1/( r o r l r o r i ) 2, as expected by the attenuation of the spherical wave propagating from the laser spot to the object, then from the object to the field of view of the camera. This dependance is clearly seen from the data shown in figure S SNR 1 5 Experimental data linear fit /r 4 (m -4 ) Figure S3: Signal-to-noise ratio of experimental data. The signal-to-noise ratio decreases with the attenuation of the signal in 1/r 4. In the experiments reported here, we obtain a SNR of 2 for a target at a distance r o r l r o r i = 6 cm. A few improvements, both in the setup and in the detection hardware, could contribute to improve this distance up to possibly 1 meters: 1. The SNR is proportional to the area of the target. For a human, this implies an increase in signal by a factor 3 compared to the target used in our experiments. The improvement for cars is potentially even higher. 2. Some characteristics of the SPAD array are likely to improve in the future (and are indeed already under development). For example, the fill factor of the current detector is only 2%. With arrays where the electronics are vertically integrated, or using microlens arrays, it will be possible to increase the fill factor to 8%, allowing a further factor 4x improvement. 3. We are currently integrating signal over only 1 frames, as our read-out speed is limited by the USB 2. connection of the SPAD array. Work is currently being done to incorporate USB 3. technology to the camera, so that the read-out will be greatly accelerated and a higher number of integration frames can be used. Using the increased data rates of USB 3., we can expect the SNR to increase by a factor We could readily gain a factor 5x by optimising the near-infrared sensitivity of the SPAD detectors. Indeed, this has been recently demonstrated with sensitivities at our operating wavelength of the order of 25-3%, to be compared with the 5% sensitivity in our current system [1, 2]. We also note that many of these numbers (8% fill factor with microlens arrays, 25% photon efficiency) are already available in commercial SPAD arrays operating at 15 nm (InGaAS technology). NATURE PHOTONICS 5

6 DOI: 1.138/NPHOTON.234 Taking into account all of these contributions, it is likely that we can improve the signal by a factor 6, which means we could track human-size targets at around 1 meters. Some factors, in turn, could contribute to decrease the signal-to-noise ratio. Mainly, we use a white surface and target in our experiments: a change in the material could potentially decrease the SNR by a factor 1. This would translate into a detection range of around 6 m rather than 1 m. S4 Uncertainty of the position retrieval We measure the uncertainty or precision in x and y of our technique by calculating the mean square weighted deviation, namely σ 2 x, σ 2 y, of the retrieved probability densities P ( r o ). As can be seen in figure 3 of the main text, the precision varies according to the position of the hidden target. Figure S4 shows how σ x and σ y vary as a function of the target s distance, more precisely as r o r i, i.e. the distance between the object and the field of view. Uncertainty (< x, < y ) (cm) 8 < x 6 < y r o -r i (m) Figure S4: Uncertainty of experimental data. Uncertainty of retrieved position for the eight positions of the target shown in figure 3 of the main text. target position y (cm) σx (cm) target position y (cm) σy (cm) a target position x (cm) b target position x (cm) Figure S5: Simulated uncertainty of the position retrieval algorithm. The uncertainties σ x and σ y of our retrieval algorithm depend on the target s position. a) The precision in x is maximal on the line crossing the two foci of the probability density ellipses we use in the retrieval algorithm (y = 83) and increases as we go away from this line in the y direction. For a given y position, we also observe the uncertainty σ x to decrease with increasing distance. b) The uncertainty σ y has a different behaviour, and increases with distance, as the ellipses i get larger. 6 NATURE PHOTONICS

7 DOI: 1.138/NPHOTON.234 SUPPLEMENTARY INFORMATION We also observe that the precision of the method depends on the specific position in x and y of the target. To study this dependance, we perform simulations of the signal emitted by a single-point target scanning a range of positions in x and y. The signal emitted by this target is taken to have a temporal deviation of σ t = 218 ps, which is the average deviation that we find from our experimental data. From the simulated signal, we apply the same position retrieval algorithm to find a probability density P ( r o ) and we calculate its weighted standard deviations σ x and σ y. In figure S5, we show the obtained uncertainties as a function of the target s position (x, y). The uncertainty is encoded in color in these two graphs. We observe in figure S5 a complex dependance of the precision on the target s position, which can be explained with geometrical arguments by considering the probability density ellipses i used in our retrieval. The behaviour of the uncertainty is explained schematically in figure S6: as can be seen, the ellipses resulting from a target at a large distance are both large and similar to each other (as they converge towards the limiting mathematical function which is a circle). This leads to a large overlap and hence a large uncertainty in the y direction but reduced uncertainty in the x direction. We also study the effect of another important parameter in the uncertainty σ x and σ y of our retrieval: the number of pixels i from the field of view used in the retrieval algorithm. Although 124 pixels (32x32) Figure S6: Geometry of the position retrieval algorithm and its expected precision. The dependance of σ x and σ y can be understood from the geometry of our retrieval process. We rely on ellipses to describe the probability density of an object s position corresponding to our measurements. If the object is close (green), the ellipses are smaller and have a more pronounced curve where they overlap. When the object is farther (red), the ellipses become flatter at the region of overlap, leading to higher uncertainty in y, but lower uncertainty in x. A similar argument can be made to understand the variation of σ x as a function of the y position of the object, where the uncertainty in x varies according to the curvature of the ellipses at the position of overlap. NATURE PHOTONICS 7

8 DOI: 1.138/NPHOTON.234 a b Uncertainty (cm) Uncertainty (cm) 1 5 Experimental Data # pixels used Simulations #pixels used Figure S7: Effect of the number of pixels on the precision of the retrieval algorithm. Using a lower number of pixels than the 124 of the camera for the retrieval algorithm leads to a lower precision. This correlation is observed in the experimental data a shown in figure 3 of the main text, and is confirmed by simulations b. This dependancy can explain one effect of a lower signal-to-noise ratio on the precision of the data: if the signal cannot be detected on all pixels of the field of view, a lower number of pixels will be used for the position retrieval and the uncertainty will be higher. σ x σ y σ x σ y are technically available for the retrieval, the decreasing signal-to-noise ratio leads to a loss of signal on some of these pixels. In figure S7a, we show the precision obtained on the eight positions of figure 3 of the main text as a function of the number of pixels that were used in the retrieval algorithm for these positions. We observe that the uncertainty does increase quite significantly when the number of pixels is decreased. To confirm this effect, we perform simulations where only a portion of the 124 pixels, chosen in a random fashion, are used for the retrieval algorithms. The results of the simulation are shown in S7b and clearly show a dependance between the number of pixels and the obtained precision. This dependancy shows how a loss of signal of a portion of our pixels will affect the precision of our method, and is also relevant to consider in the design of similar systems that would perform tracking of hidden objects with a number of single-pixel detectors rather than with a SPAD array. In such a case, the relative position and distribution of pixels will also play an important role in the expected precision of the system. S5 Tracking moving objects We show in the main text the results for an experiment where the target is moving at 2.8 cm/s on a track positioned at x = 4 cm. Two other sets of data were acquired, for tracks positioned at x = 2 cm and x = 3 cm and the object is moving along the track at 3.9 cm/s and 5.3 cm/s, respectively. We show in Figure S8 consecutive positions acquired for the target moving along these two other tracks. These results are similar to those shown in Figure 4 of the main text, although they are more precise as the target is closer to the wall. We can also see that the difference in distance covered by the target moving at different speed. The Supplementary Video provides all retrieved positions for all tracks, where we clearly see a difference in speed as the target changes track. 8 NATURE PHOTONICS

9 DOI: 1.138/NPHOTON.234 SUPPLEMENTARY INFORMATION S6 Tracking multiple targets Our method can be extended to tracking multiple hidden objects, provided that the signal originating from the objects do not significantly overlap. As a proof of principle, we recorded the signal from two targets separated by 45 cm. Once the background is retrieved from the recorded signal, we can distinguish the two signals coming from the two distinct targets, as they produce backscattered spherical waves that can be distinguished both in time and in space, as illustrated in Fig. S9. For this proof-of-principle experiment, we first found the multiple peaks in each histogram and applied our retrieval algorithm individually to each of the two separate signals. In the inset of figure S9, we can see that the two signals are spatially separated in each frame. The retrieved probability densities are in good agreement with the positions retrieved from single-target measurements. References [1] Webster, E.A.G., Grant, L.A. & Henderson, R.K. A high-performance single-photon avalanche diode in 13-nm CMOS imaging technology. Electron Device Letters, IEEE 33, (212). [2] Mandai, S., Fishbrun, M.W., Maruyama, Y. & Charbon, E. A wide spectral range single-photon avalanche diode fabricated in an advanced 18 nm CMOS technology. Opt. Express 2, (212). 12 time (s) time (s) Field of view 1 9 Field of view y (cm) 6 9 y (cm) x (cm) x (cm) a b Figure S8: Tracking moving object. The target is moving in a straight line along the y-direction, from bottom to top, at a speed of 3.9 cm/s in a and 5.3 cm/s in b. The retrieved probability density of the target s location are represented in color, where the color indicates the time at which the position is recorded. The acquisition was done in real-time as the object was moving along the tracks. NATURE PHOTONICS 9

10 DOI: 1.138/NPHOTON y (cm) x (cm) Figure S9: Detecting multiple targets. When multiple targets are present, they produce different signals that can be distinguished both in time (distinct peaks in the histograms) and in space. The inset shows one frame of the recorded movie, where we clearly see two backscattered spherical waves passing through the field of view, as indicated by the blue (target 1) and purple (target 2) lines. The retrieved positions are in good agreement with the positions found by measuring the signal measured from one single target at a time. The blue and purple dots represent the point of maximum retrieved probability in the case of a single-target measurement for target 1 and 2, respectively. 1 NATURE PHOTONICS