As someone with first-hand experience in implementing pass-through AR, I found the review well-made (e.g., they measured photon-to-photon latency) and extremely interesting. Here's a few observations I made:
1. 30 Hz frame rate and 50ms photon-to-photon latency sound so bad that I'm surprised they launched this at all. I would be very interested in trying this out to see how (un)comfortable the experience is. What I would have expected is pass-through feed rate synchronized to display frame rate (i.e., 90 Hz) and photon-to-photon latency no more than perhaps 2 frames (22ms). At 90 Hz and 11ms you can actually play ping-pong while wearing a pass-through headset.
2. I can relate with the exposure, saturation, and noise issues. The existing well-tuned video ISP pipes aren't suitable for low latency, and building a new one that achieves low latency and gives a good picture in indoor lighting from an essentially smartphone sensor with exposure time < 11ms and doesn't add multi-frame latency is HARD.
3. Above points to one possible explanation for the 30Hz capture; maybe they used a cheap sensor that can't produce usable picture in < 11ms exposure time (in dimly lit indoor environment) and thus had to go with 30Hz capture. Would be interesting to repeat the experiment in well-lit (outdoor) environment to see if the capture pipe kicks into a higher framerate.
4. I wonder if they do any warping to correct for the camera positions and how well that works. For AR-focused headset the ideal pass-through camera positions would approximate eye positions to minimize need for distortion. Even shifting capture positions a few centimeters forward from the eyes (which you'd need to do if you don't replace eyes with cameras or play tricks with optics) creates a subtle but noticeable effect unless corrected.
1. 30 Hz frame rate and 50ms photon-to-photon latency sound so bad that I'm surprised they launched this at all. I would be very interested in trying this out to see how (un)comfortable the experience is. What I would have expected is pass-through feed rate synchronized to display frame rate (i.e., 90 Hz) and photon-to-photon latency no more than perhaps 2 frames (22ms). At 90 Hz and 11ms you can actually play ping-pong while wearing a pass-through headset.
2. I can relate with the exposure, saturation, and noise issues. The existing well-tuned video ISP pipes aren't suitable for low latency, and building a new one that achieves low latency and gives a good picture in indoor lighting from an essentially smartphone sensor with exposure time < 11ms and doesn't add multi-frame latency is HARD.
3. Above points to one possible explanation for the 30Hz capture; maybe they used a cheap sensor that can't produce usable picture in < 11ms exposure time (in dimly lit indoor environment) and thus had to go with 30Hz capture. Would be interesting to repeat the experiment in well-lit (outdoor) environment to see if the capture pipe kicks into a higher framerate.
4. I wonder if they do any warping to correct for the camera positions and how well that works. For AR-focused headset the ideal pass-through camera positions would approximate eye positions to minimize need for distortion. Even shifting capture positions a few centimeters forward from the eyes (which you'd need to do if you don't replace eyes with cameras or play tricks with optics) creates a subtle but noticeable effect unless corrected.