Computer vision in general and facial recognition in particular are rapidly growing fields of computer science that can be beneficial in or even revolutionize some areas, such as transportation or retail. In this article we’re going to look closely at how such system could be built.

The source code for this project is available on GitHub. You can run it locally or you can take a look at the demo app to get an idea on how it works. This service does only one thing: it processes a video from a webcam, detects faces and calculates if people it sees have been seen before. This is a generic computer vision task that many applications nowadays leverage upon, such as facial identification on your phone or security systems with access control.

First, let’s get the bigger picture on how this system operates.

Architecture

The service consists of three distinct parts: a WebRTC server, an application server and a recognition server. Client applications both stream video data and get facial recognition results back over WebRTC. The process can be described with the following sequence diagram:

Now let’s take a closer look on each actor of the system.

Client application

The client in this example is a web application built with React, Redux and Next.js, but it is also possible to create a mobile or desktop application, since there are WebRTC implementations for most modern platforms. Even more, with just some small changes, you can process video from IP cameras over RTSP.

The client application also acts as a temporary store of face data received from the recognition server. In a real-world system, this task will be performed on a server-side and will involve a database, but for the sake of simplicity, this is out of the scope of this article.

WebRTC signaling and streaming is performed by the OpenVidu client over WebSockets, so the WebRTC implementation is trivial.

WebRTC server

The WebRTC server consists of several building blocks. At its core, it is based on Kurento (a media server) and OpenVidu (a signaling server), both are open-source solutions with a paid support for OpenVidu that allows to scale to multiple nodes. The third part of the solution is Coturn – a TURN server, which is used for NAT traversal. There are also Redis and NGINX involved internally to handle the state management and to secure communcation between the system components.

Overall, this solution works out of the box and have unopinionated client libraries available, so we’re not going to dive into the details here. It’s worth mentioning that OpenVidu has recently released a mediasoup support to replace Kurento, which can dramatically improve the overall performance and provide a low-level access for Node.js applications.

Application server

This server is responsible for the following tasks:

• Provide an API to securely initiate WebRTC sessions. It is possible to initiate a session from the client directly, but that entails security risks and should not be used in production.
• Connect to client video broadcasts and take regular snapshots. The server acts as a WebRTC client and saves video frames to canvas for further processing.
• Provide snapshots to the recognition worker and relay the results back to clients. When the worker responses with face data, the server crops a previously stored snapshot to only include regions that contain faces and sends it to the client alongside with face encodings.
• Host the client web application. This is not a must, but rather a neat feature that Next.js provides.

Recognition worker

Last but not least, the server that is responsible for implementing facial recognition tasks. It is a Python web server that utilizes the infamous face_recognition library, which is used to find and to encode faces on images. In this example it’s going to be as simple as that: a stateless server that uses a readily available library to do all the heavy lifting. We’re going to look at possible directions on how to improve upon this solution at the end of the article.

Algorithm

Ultimately, we want to be able to tell if two photos contain a face of the same person. But what we’re going to compare, exactly? In order to perform a comparison, we need to transform a photo, which is merely an array of pixels, to something more meaningful. This process is called face encoding. It can vary based on the approach and algorithm used, but overall you can think of it as a measurement of facial features, such as a distance between the eyes, eyebrows, nose or lips, their size and position. The composition of these features can be represented by an N-dimensional vector (in this case, a 128-dimensional one), sometimes called face vector. By calculating the Euclidean distance between two face vectors, we can measure how close those faces are.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

const newFace = {
  encodings: [/* face vector */],
};

const candidates = [];

// Compare the face with all existing faces
for (const existing of existingFaces) { 
  // Calculate the Euclidean distance, which is a measure
  // of how two faces are similar to each other.
  // The lower the distance, the more similarity they share.
  const sum = existing.encodings.reduce((res, x1, i) => {
    const x2 = newFace.encodings[i];

    return res + ((x1 - x2) ** 2);
  }, 0)

  const distance = Math.sqrt(sum);

  // Only consider faces that are similar enough (distance < 0.6)
  if (distance >= 0 && distance < state.threshold) {
    candidates.push({
      match: existing,
      distance,
    });
  }
}

// Find a face with the shortest euclidean distance 
const candidate = candidates
  .sort((a, b) => a.distance - b.distance)[0];

// Calculate the similarity percentage
if (candidate) {
  const linear = 1.0 - (candidate.distance / (state.threshold * 2.0));
  const score = linear + ((1.0 - linear) * Math.pow((linear - 0.5) * 2, 0.2));
  const similarity = Math.round(score * 100);

  alert(`Found a face with the similarity of ${similarity}%`);
} else {
  existingFaces.push(newFace);

  alert(`A new face detected`);
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

// This is an example of a face vector
[
 -0.08237294852733612, 0.06998157501220703, 0.04017249494791031, -0.04595483839511871,
 -0.20524457097053528, 0.06130267679691315, -0.003034330904483795, -0.034762077033519745,
 0.07720481604337692, 0.0018782643601298332, 0.26475948095321655, -0.06688505411148071,
 -0.2565714120864868, -0.024762030690908432, -0.017667800188064575, 0.08394743502140045,
 -0.1338515281677246, -0.0945650115609169, -0.10690569132566452, -0.10622785240411758,
 0.02856566198170185, -0.002836777362972498, -0.03754136338829994, 0.033261124044656754,
 -0.2024488002061844, -0.25246909260749817, -0.030827492475509644, -0.007754474878311157,
 0.08512576669454575, -0.15835121273994446, 0.09394755959510803, 0.19072295725345612,
 -0.17289769649505615, -0.04181254655122757, 0.04165133833885193, 0.14661748707294464,
 -0.11559691280126572, -0.015472346916794777, 0.19385945796966553, -0.04230993986129761,
 -0.13715803623199463, 0.09014783799648285, 0.09632508456707001, 0.29142847657203674,
 0.16114553809165955, 0.0382876954972744, 0.025807470083236694, 0.0014453493058681488,
 0.08283577114343643, -0.32194820046424866, 0.012574739754199982, 0.18525069952011108,
 0.16502265632152557, 0.0586332231760025, 0.02298889309167862, -0.13721558451652527,
 -0.0279446542263031, 0.14717113971710205, -0.14877578616142273, 0.15434607863426208,
 0.10199335217475891, -0.21128228306770325, -0.08583835512399673, -0.14078442752361298,
 0.15042155981063843, 0.046597909182310104, -0.14368422329425812, -0.1568884551525116,
 0.1736888289451599, -0.088925302028656, -0.11250282824039459, 0.16585050523281097,
 -0.06869833171367645, -0.16567328572273254, -0.25068527460098267, 0.12233469635248184,
 0.4443138837814331, 0.12621524930000305, -0.21290765702724457, -0.00904833059757948,
 -0.028415795415639877, 0.003739515785127878, -0.04296185076236725, 0.032679952681064606,
 -0.06143368408083916, -0.1341024786233902, -0.1555326282978058, 0.04243481904268265,
 0.20805151760578156, -0.0508725643157959, -0.01405704952776432, 0.18905237317085266,
 0.1028638407588005, 0.04386136680841446, 0.0037655732594430447, 0.059780303388834,
 -0.11599305272102356, -0.0248013436794281, -0.13408342003822327, -0.01498719397932291,
 0.040145330131053925, -0.13561423122882843, 6.418908014893532e-05, 0.07700222730636597,
 -0.08549322187900543, 0.2858748733997345, -0.01664814166724682, -0.013422049582004547,
 -0.010783687233924866, 0.019620651379227638, -0.09562011063098907, 0.03741421923041344,
 0.19683292508125305, -0.33195415139198303, 0.1915217787027359, 0.17226657271385193,
 0.07776990532875061, 0.11128266900777817, 0.14693227410316467, 0.09936179965734482,
 0.037961095571517944, 0.017703596502542496, -0.12619523704051971, -0.18041686713695526,
 0.0061234827153384686, -0.09858812391757965, -0.0010826261714100838, 0.04442061856389046
]

Going further

While this example is an MVP and doesn’t cut many corners, it still requires some attention before it can be used in production. There are a few things to consider if you’re going to build a real-world service on top of it.

Machine learning pipeline

If you ran this services yourself, you could have noticed that image processing takes a decent amount of time. One of the reasons behind that is face_recognition algorithm used for face detection. The process of face encoding is preceded by face detection. In order to encode facial features, you need to find them on the image first. Instead of relying on the library’s ability to both detect faces and to encode facial features, you could use a faster ML model to detect faces first, then find and encode facial features in a just small region of the image. This becomes critical when processing higher definition images and handling a lot of requests at the same time.

Moreover, your service could also use specialized ML models to classify faces by age or gender. That opens huge possibilities, but also requires a lot of fine-tuning and performance optimizations.

Hardware acceleration

A model inference can have a much higher performance by using a GPU, but since it’s highly platform-dependent (especially when you put Docker in the mix), it was intentionally disabled. Moreover, depending on your requirements, your application could also benefit from a hardware-accelerated canvas. And even without a discrete GPU, you could still optimize the performance by using the right CPU instructions or CPU architecture that your ML framework is optimized for.

Persistent storage

This demo application stores face data in memory on the client. In a production application, you would likely do that on the server-side and using a persistent storage, such as a database. The main obstacle in doing so is that there aren’t that many databases out there that can efficiently index and query over N-dimensional vectors. You could solve that by writing your own indexing algorithm or by using 3rd party extensions available for some popular RDBMS.