3D camera: calibration and image sequence analysis
Project title:
Development, implementation and verification of methods for processing 3D camera data
Funding: IPF
3D cameras (range imaging cameras) are digital cameras that use modulation techniques to provide distance information for each pixel in addition to the image information. The cameras therefore simultaneously record intensity and range images and enable a spatio-temporally resolved representation of the object space without the need for stereoscopic mapping. With a sensor format of currently approx. 25,000 pixels and a temporal resolution of 20-50 images per second, these depth image sensors represent an interesting alternative for 3D data acquisition in the fields of robotics and navigation, automotive, surveillance, human-computer interaction (HCI) and human-robot interaction (HRI) as well as human movement and behavior analysis. Since 2006, a prototype of a SwissRanger SR-3000 distance-measuring camera has been available at the IPF for research and development work in the areas of sensor calibration and image sequence analysis.
Photogrammetric calibration using intensity and distance information simultaneously
With conventional cameras, the reference between image and object space is established via the model of the central perspective image using the orientation parameters and image coordinates. An ideal model of the central projection is initially also assumed for the on-chip calculation of the Cartesian coordinates for 3D cameras. Deviations from this ideal model must also be corrected for this sensor using suitable calibration approaches.
The reference field was designed taking into account the currently low resolution of 3D cameras. A 3D calibration field can be simulated by sequentially positioning a calibration plate signaled with four white spheres in the camera's field of view. This approach allows an adequate number of spatially distributed control points to be recorded from a single 3D camera viewpoint. An additional static reference with small coded as well as uncoded control points enables the transformation of the sphere centers into a superordinate coordinate system. The 3D coordinates of all object points of the static reference and the dynamic calibration plate can be estimated as part of a bundle adjustment and 3D Helmert transformation. The images required for this were taken with a conventional high-resolution DSLR camera.
Due to the nature of the data from a distance-measuring camera, it seems desirable to integrate the additional distance information as well as the determination of control point image coordinates from distance images into a simultaneous calibration strategy. The following observation types are introduced for an integrated 3D camera calibration approach:
- 1. image coordinates from intensity images
The measurement of the control point coordinates in the image space is carried out as a conventional image point measurement using least square template matching. The image coordinates obtained with sub-pixel accuracy (&sigma x,y = 1/30px) are included in the adjustment as the first type of observation.
- 2. image coordinates from distance images
- 3. distances between projection center and object points
2.5D Least Squares Tracking
Automatic methods of photogrammetric motion analysis in image sequences represent an established sub-area of close-range photogrammetry and enable the highly accurate and reliable extraction of geometric image information. The use of 3D cameras can provide decisive advantages in 3D motion analysis compared to the use of multi-camera systems. They compete with other methods of depth image generation such as laser scanners, stereo camera systems or camera projector systems. Their decisive advantage over stereophotogrammetric systems lies in the elimination of the stereoscopic mapping step.
With regard to the nature of 3D camera data, an integrated 2.5D least squares tracking approach was developed and verified at the IPF. Based on the well-known method of 2D least squares matching, 3D camera intensity and distance information is used to solve the correspondence problem on the time axis. This closed-form determination of all transformation parameters
- Translation in x-, y- and depth direction (a0, b0, d0),
- rotation and shear (a2, b1),
- Scale change in the image plane caused by depth variations (a1=b2=f(d0))
can significantly increase the accuracy and reliability of the assignment step, especially under non-ideal conditions in the intensity or distance channel. In addition to a-posteriori accuracy information of the distance measurement, information about the a-priori accuracy of the original observations is available by integrating a variance component estimation to determine optimal weights of the two observation classes.
To evaluate the established functional model as well as the potential of 2.5D least squares tracking in 3D camera image sequences, several test series with synthetic and real data sets with (i) high intensity contrast, (ii) high range contrast and (iii) balanced contrast between both channels were performed.
The following results were obtained:
| No contrast | No convergence of the solution vector |
| Contrast in one channel | Slight increase in accuracy of translation (5%) and scale parameters (20%) compared to single channel calculation |
| Significant depth variations | Accuracy increase of the scale adjustment by 50% |
| Contrast in both channels | &sigma a0,b0 = 1/50...1/25px &sigma d0 = 0.25% of the shift in depth direction |
The quality of the introduced functional model can be derived from the standard deviations of the balanced observations. For the intensity channel, these are in the range of 100 conts (16-bit) or 0.3 gray values (8-bit). The accuracies of the equalized distance measurements are in the range of 8 conts (16-bit) or 1 mm.
An iterative variance component estimation is used to estimate the variances of both types of observations. The standard deviations of the original observations result in 1600 conts (16-bit) or 6 gray values (8-bit) for the intensity channel and 121 conts (16-bit) or 1.4 cm (integration time: 20.2ms; modulation frequency: 20.0MHz) for the distance measurements. The use of only two groups of observations to build the stochastic model - as opposed to weighting each individual observation - was validated by the additional estimation of a robust variance-covariance matrix.
Future work will address aspects of (i) outlier handling, (ii) extended parameterization (esp. tilting in depth direction), (iii) evaluation and (iv) computational time optimization.
Human motion analysis
Student research projects and diploma theses
Topic: Evaluation of the application potential of a distance-measuring camera in the field of mobile robot navigation
Supervisors: Dipl.-Ing. P. Westfeld, Dipl.-Ing. Ch. Mulsow
Topic: Validation of the accuracy and application potential of a distance measuring camera
Supervisor: Dipl.-Ing. P. Westfeld
Topic: Photogrammetric calibration of the SwissRanger SR-3000 from intensity and range images
Supervisor: Dipl.-Ing. P. Westfeld
Topic: Monitoring an industrial robot by processing three-dimensional image data from the SwissRanger SR-3000 distance-measuring camera
Supervisor: Dipl.-Ing. P. Westfeld
- Publications
- Westfeld, P.; Mulsow, C.; Schulze, M. (2009):
Photogrammetric calibration of range imaging sensors using intensity and range information simultaneously. Grün, A.; Kahmen H. (Eds.): Optical 3-D Measurement Techniques IX. Vol. II, pp. 129, Research Group Engineering Geodesy, Vienna University of Technology ( 950 KB) - Hempel, R.; Westfeld, P. (2009):
Statistical Modeling of Interpersonal Distance with Range Imaging Data. In: A. Esposito, A. Hussain, M. Marinaro and R. Martone (eds.), Multimodal Signals: Cognitive and Algorithmic Issues. COST Action 2102 and euCognition International School Vietri sul Mare, Italy, April 21-26, 2008 Revised Selected and Invited Papers, Lecture Notes in Computer Science, Vol. 5398/2009, ISBN 978-3-642-00524-4, Springer Berlin/Heidelberg, pp. 137-144, DOI: 10.1007/978-3-642-00525-1_14 ( Link: SpringerLink) - Westfeld, P.; Hempel, R. (2008):
Range Image Sequence Analysis by 2.5-D Least Squares Tracking with Variance Component Estimation and Robust Variance Covariance Matrix Estimation. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B5. Beijing, China. pp. 933-938. ( 940 KB) - Westfeld, P. (2007):
Approaches to calibrating the SR-3000 range imaging sensor using simultaneous intensity and range images. Photogrammetrie - Laserscanning - Optische 3D-Messtechnik (Beiträge Oldenburger 3D-Tage 2007, Hrsg. Th. Luhmann), Verlag Herbert Wichmann ( 1,2 MB) - Westfeld, P. (2007):
Development of Approaches for 3-D Human Motion Behavior Analysis Based on Range Imaging Data. Grün, A.; Kahmen H. (Eds.): Optical 3-D Measurement Techniques VIII. Vol., pp., Institute of Geodesy and Photogrammetry, ETH Zurich ( 4,6 MB)
contact
Management:
Editing: