Face tracking and pose estimation with automatic 3D model construction (T.O. 004)

A research group from the Electronic Department of Alcalá University has developed a method for robustly tracking and estimating the face pose of a person using stereo vision. The method is invariant to identity and does not require previous training. A face model is automatically initialised and constructed on-line. This method could serve as the basis of a monitoring system for driving or behavior analysis of drivers and it has been tested on sequences registered in a naturalist simulator and a moving car. The group is interested in commercial agreements with companies from the automotive components sector, simulators designers and electronic entertainment (interfaces for PCs, TVs, video games, etc).

Description:
The model is formed by a set of 3D tri-dimensional points of the face. These points are automatically selected in the first image obtained from the cameras. The face is located using Viola&Jones method, and points in the face that present adequate characteristics for tracking are found with Harris detector. Up to 30 points are used.

The image patches around the 2D projections of these points on each camera are tracked on each frame, using the Simultaneous Modelling and Tracking (SMAT) algorithm. This algorithm builds a model of the changes of the appearance or texture around each point.

The 3D pose is obtained from the 2D points using POSIT, redundantly for both cameras to improve robustness.

Tracking may fail for some points on each frame. RANSAC is used to discard erroneous points from the estimation of the pose. After a set of correctly tracked points (inliers) is obtained, the position of the outlier points is reset accordingly to the estimated pose.

Points become occluded as the head turns and can not be tracked. The system is able to robustly estimate the pose of the face in presence of turns of up to ±90º. It uses a novel technique that completes the model as the face rotates and employs the method of bundle Adjustment to adjust the model.

The system is able to track a driver’s face robustly in real conditions. Experimental results and an analysis of the performance are ready to be presented.

Innovative aspects:

For the first time, this system of computer vision is able of recognizing the orientation of the human face, with no need of previous identification of the person or offline training.

This system is innovative in the way of joining the three algorithms that uses for its functioning: SMAT, RANSAC and POSIT. To the best of our knowledge they had never been used before in that way.
Current stage of development of the research result:
Developed, available for demonstration

Funding Research:
National project

Competitive advantages:

This method works in real time (30 images per second) and it takes 33 milliseconds to execute the algorithms.

It is a very robust system that keeps on working even in situations for which the model has not been designed (sudden turns of face, strong shadows, etc). The system works with increased estimation error but tracking of the face is not lost.

Current state of industrial and intellectual property:
Secret know how

Type of collaboration sought:
Commercial agreement with technical assistance

Comments:
The group is looking for commercial agreements with technical assistance with companies specialized in components for the automotive sector, companies of video games, virtual reality, interface in control of robots, driving simulation...