Business needs

The customer’s agency had 100+ in-house 3D designers to process every image. In the past, the flow used to involve designers processing images from clients. The pictures could be of poor quality, from social networks or family albums. 

Then designers took stock prop of heads, hands, torsos, etc., and sculpted a 3D model that suited the sent image as much as possible. The image was overlaid on top of the created model as a texture. Every image took 15 minutes to process.  

• Help designers save time on creating 3D models of the person’s head, i.e. programmatically generate a 3D model using one photographic shot.   
• The 3D head model includes a face, skull shape, ear shape, teeth, tongue, and an emotional expression, without hairline, beard, eyebrows, or mustache.   
• The client was looking for a fast solution, within a few months. 
• We got a task to research if per-pixel accuracy was possible.

Provided solution

We did not take data from designers because they considered the visible side of the head only, thus their 3D models were not symmetrical and had a non-constant number of vertices.   

We had to use something other than the state-of-the-art solutions like 2D to 3D neural networks because: 

• we were missing a proper dataset,  
• the neural network had a huge size and an enormously long training time,  
• even the state-of-the-art solution had quite low-quality results for the provided examples.  

3D face reconstruction  

Step 1 

• Retrain the FAN2D network;
• Fix the heatmap generation; 
• Annotate cheeks landmarks on the 3D head;
• Generate several samples of synthetic 3D heads with different orientations, genders, and skin colors (textures);
• Final fine-tuning of the existing FAN2D model. 

Step 2

Principal Component Analysis (PCA) is the most widely used tool in exploratory data analysis and in Machine Learning (ML) for dimensionality reduction. In our case, one PCA component is a set of vectors per vertex. The factor/scalar for one component is the same for all vectors belonging to this component.  

Hence, if we found a proper factor for one vertex of the component, we found it for all of the other vertices of the same component. 

It helped us to limit the vertices for landmarks only and significantly reduced computational cost. 

Step 3

The final 3D reconstruction includes the following:  

• Applies rotation to the reference 3D model;
• Estimates the 3D model scale by 2D landmarks;
• Constructs a loss function (distances between 2D landmarks and projected 3D landmarks);
• Uses SGD to find the best scalar for each PCA component and to make the final fit of 3D and 2D landmarks. 

Client values

1. We conducted profound research to find the best-suited solution to the client's problems.
2. Our developers offered an out-of-the-box face detection system that helps reduce time to process images and the time of designers’ work.
3. We managed to do the research within estimated 2 months.
4. We reduced the image processing time from 15 minutes to 10 minutes with high accuracy.