Solution

Each frame, the C++ face beautification module receives a luminance (Y) component of the image captured by the device front camera in a YUV format, as well as facial landmark metadata (contours of eyes, mouth, etc.) obtained from the ARKit/ARCore frameworks. Then, using the points provided, it builds a binary mask indicating areas of the skin on the face, excluding the mouth and eyes. Finally, the module applies a bilateral filter to smooth the skin (due to the use of a bilateral filter, it does not affect glasses, headgear or hair). The processed pixel buffer is then passed back to the iOS app, which uses Metal shaders to convert the YUV values to an sRGB format and render the image on the screen.

Requirements

• The face beautification module stably processes the video stream in real time at 60 FPS (tested on iPhone 12 and iPhone XR). However, some factors can cause frame rate drops:
- Running on an older, slower or overheated device;
- Increasing the filter size parameter (increases the effect of skin smoothing);
- Increasing the size of the face in the frame (due to bringing the face closer to the device).

• The compiled C++ module is less than 1 MB, and the total size of the iOS app is less than 2 MB. However, the submitted .ipa file is a universal application containing resources and binaries for all architectures, so it is slightly larger. In production, thinned IPA files should be exported separately for each architecture.

• The face beautification module is written in pure C++ and does not use any third-party libraries or ML-models.

В работе используется системная функция для определения черт лица, после чего приложение выборочно модифицирует найденные области. Реализована работа на CPU, через GPU только выводится финальный результат.

Размер библиотеки < 200 KB.

Приложение позволяет в автоматическом режиме смягчить тон кожи и управлять глубиной этого параметра. Критическим недостатком приложения является его построение на ARKit, что делает невозможным запуск на устройствах без TrueDepth камеры.