MediaPipe Face Landmarker

Our new model ZOO works with DepthAI V3. Find out more in our documentation.

1+ Likes

Model Details

Model Description

MediaPipe Face Mesh is a lightweight model that predicts 3D face surface geometry. The model operates on cropped face images with additional padding at the borders. The resulting keypoints can be used for different VR/AR applications, emotion recognition, face recognition, and many more.

Developed by: Google
Shared by:
Model type: Computer vision
License:
Resources for more information:

Training Details

Training Data

The dataset consists of around 30K in-the-wild mobile camera photos taken from a wide variety of sensors in changing lighting conditions. For more information about training data check the .

Testing Details

Metrics

The evaluation dataset contains 1700 samples evenly distributed across 17 geographical subregions (e.g. Europe, Central Asia, Eastern Asia,...).

Region	Mean absolute error
Eastern Asia (best)	3.68
Central America (worst)	5.24
Average	4.28

Results are taken from .

Technical Specifications

Input/Output Details

Input:
- Name: input_1
  - Info: NCHW BGR un-normalized image
Output:
- Name: Multiple (please consult NN archive config.json)
  - Info: Face surface represented as 468 3D landmarks and face score

Model Architecture

Straightforward residual neural network architecture for the mesh prediction. Please consult the for more information on model architecture.

Throughput

Model variant: mediapipe-face-landmarker:192x192

• Input shape: [1, 3, 192, 192] • Output shapes: [[1, 1, 1, 1404], [1, 1, 1, 1]]

• Params (M): 0.603 • GFLOPs: 0.041

Platform	Precision	Throughput (infs/sec)	Power Consumption (W)
RVC2	FP16	106.50	N/A
RVC4	INT8	597.77	2.33

* Benchmarked with , using 2 threads (and the DSP runtime in balanced mode for RVC4).

* Parameters and FLOPs are obtained from the package.

Quantization

RVC4 version of the model was quantized using a custom dataset. This was created by taking 40 face-cropped images from different datasets available on the web.

Utilization

Models converted for RVC Platforms can be used for inference on OAK devices. DepthAI pipelines are used to define the information flow linking the device, inference model, and the output parser (as defined in model head(s)). Below, we present the most crucial utilization steps for the particular model. Please consult the docs for more information.

Install DAIv3 and depthai-nodes libraries:

pip install depthai
pip install depthai-nodes

Define model:

model_description = dai.NNModelDescription(
    "luxonis/mediapipe-face-landmarker:192x192"
)

nn = pipeline.create(ParsingNeuralNetwork).build(
    <CameraNode>, model_description
)

Inspect model head(s):

KeypointParser that outputs message (468 detected keypoints of the face).

Get parsed output(s):

while pipeline.isRuning():
    parser_output: Keypoints = parser_output_queue.get()

Example

You can quickly run the model using our script. It automatically downloads the model, creates a DepthAI pipeline, runs the inference, and displays the results using our DepthAI visualizer tool. To try it out, run:

python3 main.py \
    --model luxonis/mediapipe-face-landmarker:192x192

Face landmark model.
License	Apache 2.0 Commercial use
Downloads	657
Tasks	Keypoint Detection
Model Types	ONNX

Model Variants

Name	Version	Available For	Created At	Deploy
		RVC2, RVC3, RVC4	11 months ago