Architecture

In this article we’re going to build a serverless API that can answer any question. But rather than just using an AI chat interface, we’re going to instruct the AI to send its answers to an API in a strictly structured JSON format. This means that the response of the AI is going to have a pre-defined schema and is guaranteed to have a valid syntax, compared to a regular chat prompt response. Effectively, this means that the AI can communicate with another machine when answering a question, not just with a human.

The source code for this project is available at GitHub. It’s based on the Serverless framework and consists of two Lambda functions: one that can relay questions from the user to the AI and the other that the AI can call itself when nececcary. Answers are stored in a DynamoDB table and shared between those two Lambdas:

Implementation

When creating a Bedrock model, we’re going to use a very simple instruction for the AI:

You are an agent that answers questions. You must save the answer to POST::API::answerQuestion API.

And when asking questions, the prompt will be the following:

Save the answer the following question: <question>{question}</question>

Notice the POST::API::answerQuestion part. It is the name of one of the APIs available to the AI. Bedrock agent can be provided with an OpenAPI schema of operations that it can use. The schema for our project looks like this:

{
  "openapi": "3.0.0",
  "info": {
    "version": "1.0.0",
    "title": "AI agent API",
    "description": "APIs available for AI agents."
  },
  "paths": {
    "/answer-question": {
      "post": {
        "summary": "API to save answers to questions",
        "description": "Save answers to questions.",
        "operationId": "answerQuestion",
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "type": "object",
                "required": ["answer"],
                "properties": {
                  "answer": {
                    "type": "string",
                    "description": "Answer to the question."
                  }
                }
              }
            }
          }
        },
        "responses": {
          "200": { "description": "Answer saved successfully." }
        }
      }
    }
  }
}

The schema only provides the structure of available operations, agents can’t call an API directly and instead rely on a Lambda function for execution. Our Lambda function for handling agent API calls has the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

const router = new Router();

const api = async (event) => {
  return await router.handle(event.httpMethod, event.apiPath, event);
};

router.add('POST', '/answer-question', async (request: AgentRequest) => {
  const input = getAgentInput<AnswerQuestionInput>(request); // Parse the request

  await putDocument(answersTable, {
    id: request.sessionId,
    answer: input.answer,
  }); // Save the answer to DB

  return createAgentResponse(request, { status: 'OK' });
});

It saves the answer from the AI in a DynamoDB table so that in can be retrieved later. But in order to process this call, we first need ask the AI to answer a question. We do that via another Lambda function that handles requests from a REST endpoint in API Gateway:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

const api = async (event) => {
  const inputText = textQuestionPrompt // Put question into the prompt template
    .replace('{question}', event.body.question);

  // Call Bedrock agent with the text
  const { completion, sessionId } = await invokeAgent(inputText);

  for await (const response of completion) {
    if (response.trace?.trace) {
      // Output trace info from the agent
      console.log(JSON.stringify(response.trace.trace));
    }
  }

  // Read the saved answer from DB
  const document = await readDocument(answersTable, sessionId);

  return formatJSONResponse(document);
};

Let’s try to call this function with a simple math question:

{
  "question": "Betty had a pack of 25 pencil crayons. She gave five to her friend Theresa. She gave three to her friend Mary. How many pencil crayons does Betty have left?"
}

If tracing is enabled, the AI should log its reasoning and implementation details:

To answer this question, I will:

1. Solve the math problem: Betty originally had 25 pencil crayons. She gave 5 to Theresa and 3 to Mary. So she gave away 5 + 3 = 8 pencil crayons. So she has 25 - 8 = 17 pencil crayons left.
2. Call the POST::m2m-ai-agent-api::answerQuestion function to save the answer.

I have checked that I have access to the POST::m2m-ai-agent-api::answerQuestion function.
<function_call>post::m2m-ai-agent-api::answerQuestion(answer="17")</function_call>
<function_result>{"status":"OK"}</function_result>

And the API response will contain the answer to the original question:

{
  "id": "643dc42c-f492-43ce-bfb8-ac991b2090e5",
  "answer": "17"
}

Conclusion

Agents for Amazon Bedrock is a powerful feature that can bring AI applications to a whole new level. It provides AIs with the tools to solve complex problems and to perform operations on your behalf by making possible the integration with other systems in a scalable and secure way.

Architecture

The application that we’re going to build is a webpage video recorder. The architecture is pretty straightforward: we will deploy an AWS ECS cluster of GPU-powered instances. The system image already has NVIDIA drivers installed and available from within Docker with just a little effort, so we will focus on the application itself. The application contains a pool of virtual displays. Whenever a website recording is requested, an isolated Chrome instance is launched. When it loads the webpage, the server captures the screen and encodes it in realtime to a video file on the GPU.

G5 instances are equipped with a powerful NVIDIA A10G GPU which allows to process several HD video streams simultaneously, so the application can easily scale up on the number of virtual displays, and thanks to Docker, it can run isolated parallel workflows in a secure manner.

The source code for this project is available on GitHub. It contains the application Docker image and the AWS infrastructure definition.

Docker Container

The container is based on the official NVIDIA CUDA image that contains everything that we will need to leverage on the hardware acceleration. As for the AWS instance image, we will use the official ECS-optimized AMI for GPU instances, which contains fresh NVIDIA drivers and libraries, a Docker runtime with GPU support and everything else that we’ll need to start using it in an ECS cluster. The only tricky part is to build FFmpeg from the source and to link it with NVIDIA libraries available in the system. It’s also worth noting that NVIDIA_DRIVER_CAPABILITIES variable controls what NVIDIA features will be available to the container.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

# CUDA 11.4 matches the runtime at the latest AWS ECS-optimized AMI for GPU instances
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html
# Use the development version to be able to reference headers during FFmpeg build
FROM nvidia/cuda:11.4.0-devel-ubuntu20.04

ARG DEBIAN_FRONTEND=noninteractive

# Mount all driver libraries
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#driver-capabilities
ENV NVIDIA_DRIVER_CAPABILITIES=all
ENV PORT=3000
ENV NODE_ENV=develop

RUN apt-get update -y && apt-get install -y curl wget

RUN mkdir /install
WORKDIR /install

# FFmpeg, build from source and include NVIDIA codecs
COPY ./install/ffmpeg.sh .
RUN bash -e ffmpeg.sh

# VirtualGL
COPY ./install/virtualgl.sh .
RUN bash -e virtualgl.sh

# Chrome dependencies
COPY ./install/chrome.sh .
RUN bash -e chrome.sh

# Utilities
COPY ./install/utils.sh .
RUN bash -e utils.sh

# Node.js
COPY ./install/node.sh .
RUN bash -e node.sh

RUN rm -rf /install

# Application
WORKDIR /usr/src/app

COPY package.json .
COPY package-lock.json .
RUN npm i

ADD . .

RUN npm run build && rm -rf src
RUN chmod -R o+rwx node_modules/puppeteer/.local-chromium

CMD npm start

Application

The application is a Node.js API server. It launches virtual displays with Xvfb (which stands for X virtual framebuffer) and manages the resource pooling. Each display contains a window manager, so that applications can draw themselves on it. The displays also utilize OpenGL provided by VirtualGL. While this is not a truly 3D-accelerated environment and Xvfb is also CPU demanding compared to a regular X server, for the sake of simplicity we won’t discuss how to get around that, since it’s more than enough for our use case.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// Start the display server
// https://en.wikipedia.org/wiki/Xvfb
// https://en.wikipedia.org/wiki/VirtualGL
const item = await new Promise<Xvfb>((resolve, reject) => {
  const xvfb = new Xvfb({
    reuse: false,
    xvfb_args: [
      '-screen', '0', `1920x1080x24+32`,
      '+extension', 'GLX', // Enable OpenGL
      '+extension', 'RANDR',
      '-nolisten', 'tcp', '-dpi', '96', '-ac', '-noreset',
    ],
  });
  xvfb.start((error) => {
    if (error) {
      reject(error);
    } else {
      resolve(xvfb);
    }
  });
});
const display = item.display(); // Get the display number, e.g. ":99"

// Start the window manager
// https://en.wikipedia.org/wiki/Fluxbox
const wm = execa('fluxbox', [], {
  env: { DISPLAY: display }, // Run on the new display
});

// Hide the cursor
// http://manpages.ubuntu.com/manpages/trusty/man1/unclutter.1.html
const cursor = execa('unclutter', ['-idle', '0'], {
  env: { DISPLAY: display },
});

Chrome is launched using Puppeteer with quite a few options in order to be able to work inside Docker and to enable the hardware acceleration that is usually disabled in server usage scenarios:

google-chrome-stable \
  --kiosk --start-fullscreen --autoplay-policy=no-user-gesture-required \
  --hide-scrollbars --disable-infobars --no-default-browser-check \
  --no-sandbox --disable-setuid-sandbox \
  --ignore-gpu-blacklist --ignore-gpu-blocklist \
  --enable-features=VaapiVideoDecoder --enable-accelerated-video-decode \
  --enable-gpu-rasterization --enable-oop-rasterization --enable-tcp-fast-open \
  --use-gl=desktop --enable-webgl

The resulting GPU report from chrome://gpu looks this way and is similar to what you can see in Chrome on GPU-powered non-Windows platforms. It can vary greatly depending on the Chrome version used and the driver version installed, but this outcome is perfectly fine for our use case:

Canvas: Hardware accelerated
Canvas out-of-process rasterization: Disabled
Direct Rendering Display Compositor: Disabled
Compositing: Hardware accelerated
Multiple Raster Threads: Enabled
OpenGL: Enabled
Rasterization: Hardware accelerated on all pages
Raw Draw: Disabled
Skia Renderer: Enabled
Video Decode: Hardware accelerated
Video Encode: Software only. Hardware acceleration disabled
Vulkan: Disabled
WebGL: Hardware accelerated
WebGL2: Hardware accelerated

Screen Capture

The screen capture is performed by FFmpeg with X11 as the input source. NVIDIA GPU that is used in G5 instances provides NVENC - a hardware accelerated video encoding module. Since we’ve built a custom version of FFmpeg with CUDA libraries, we can now encode HEVC (H.265) videos with FFmpeg entirely on the GPU, offloading the CPU for other tasks. This provides us with the ability to capture and compress HD videos in realtime at high FPS. The command for FFmpeg to capture the screen is the following:

# https://en.wikipedia.org/wiki/FFmpeg
# https://en.wikipedia.org/wiki/X_Window_System#Key_terms
ffmpeg \
  -f x11grab \ # Capture the X Window System
  -hwaccel nvdec -hwaccel_output_format cuda \ # Enable GPU acceleration with NVDEC
  -thread_queue_size 2048 -probesize 10M -analyzeduration 10M \ # Optimizations
  -framerate 30 -s 1920x1080 \ # Input stream parameters
  -i :99.0 \ # Capture "display" 99 and "screen" 0
  -c:v hevc_nvenc \ # Encode as HEVC (H.265) on GPU with NVENC
  -preset fast -movflags +faststart -g 999999 \ # Optimize for screen capture
  ~/output.mp4

FFmpeg runs for the requested amount of seconds, then stops and the resulting MP4 file is served to the user. The following video has been recorded from this CodePen demo. Thanks to the hardware acceleration, it is smooth and crisp.

Conclusion

As you can see, the toolset for server-side GPU applications has matured enough that building a simple app can take just a few hours. With the power of cloud scalability and the modern software, you can build things that were hard or even unimaginable just a decade ago.

Business Domain

This system operates in the online advertisement domain, which has the following entities:

• Campaign - a sales funnel that includes two or more offers.
• Offer - a product or a service that is advertised to a user by a banner image or a webpage.
• Impression - a single advertisement display to a user. It costs money for the advertiser and one of the goals is to have the highest conversion rate possible.
• Conversion - a successful sale of the offer.
• Conversion rate - the number of conversions divided by the total number of impressions.

The goal of the system is to predict which offer within a campaign has the highest posibility to convert for a specific user. It does so based on the historical data of conversions for each campaign. The assumption is that user metadata such as their geographical location and smartphone or computer traits correlate with the offer that they could be interested in. This is often the case when an offer is targeted for a specific country, mobile carrier or even a phone vendor.

Architecture

The source code for this project alongside with the installation instructions is available in this GitHub repository.

The architecture involves three data flows: historical performance data is uploaded to a storage, realtime prediction and model training requests are routed to an API. Historical data is used to create machine learning models that perform predictions whenever a training request is received.

The whole system is serverless, except for the SageMaker inference endpoint. Amazon has recently announced serverless endpoints as well, but they don’t support multi-model deployments that this system relies upon. Each campaign gets its own ML model and all models are deployed into a single inference endpoint. This allows the system to scale to hundreds of campaigns per server, which makes it more economically feasible compared to a single-model deployment when each model requires its own server.

Data Format

Dataset consists of conversion events in JSON format with the following structure:

{
  "id": "b7e7ec22-5143-48b2-bd40-19b0905e8db4",        // Conversion ID
  "campaign": "97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b",  // Campaign ID
  "offer": "41183105-2ed4-46c6-b654-2f0081aa652d",     // Offer ID
  "country": "PL",                                     // Country code
  "city": "Warsaw",                                    // City name
  "long": "21.0026",                                   // Longitude
  "lat": "52.2484",                                    // Latitude
  "isp": "T-Mobile Polska S.A.",                       // Internet service provider
  "network": "188.146.0.0/15",                         // ISP network range
  "dType": "mobile",                                   // Device type
  "dOs": "Android",                                    // Operating system
  "dOsVersion": "7.0",                                 // OS version
  "dBrowser": "Chrome WebView",                        // Web browser
  "dBrowserVersion": "65.0.3325.109"                   // Web browser version
}

This data is calculated based on client’s user agent and IP address, processed with an IP lookup database to estimate their geographical location and ISP.

There are two offers in the dataset and only one campaign - 97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b.

Data Extraction

In order to start making predictions for a campaign, we need to prepare a ML model for it. When and under what conditions a model preparation should be triggered is outside of the scope of this article. It could be based on a schedule or when a certain advertisement campaign threshold is reached, but we’re going to request it manually. In order to start the ML pipeline, you need call the API with a campaign ID:

curl https://$API_GATEWAY_HOST/train/97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b

{
  "step_machine_url": "https://us-west-2.console.aws.amazon.com/states/home?region=us-west-2#/executions/details/...",
  "model_id": "69ff7307-c1ab-43b5-97b0-a77380a778df" // This is a ML model ID and also a SageMaker Pipeline execution name
}

It will spin up an AWS Step Function, which is responsible for extracting the data from a data lake and starting the SageMaker pipeline. The data is extracted using AWS Athena - a managed service that allows you to run SQL queries over distributed data sources, including traditional relational databases, NoSQL databases and, just like in this case, an S3 data lake with AWS Glue. Using Athena allows to easily switch to any supported DBMS depending on the business needs and to also use plain JSON files without any servers, which is ideal for this tutorial.

The Athena query is pretty straightforward:

1
2
3
4

SELECT country, city, long, lat, network, dos, dtype, 
       dosversion, dbrowser, dbrowserversion, offer
FROM conversions
WHERE campaign = '97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b'

The query results are stored in an S3 bucket and will be provided to the SageMaker pipeline that will be executed on the next step.

SageMaker Pipeline

SageMaker Pipelines is a managed CI/CD service for machine learning. It is defined by the following DAG, each node of which represents a specific ML task.

Data Wrangling

The first step of the pipeline is data preparation (or wrangling) using AWS Data Wrangler, which has its own DAG:

On the first step of the process we’re going to get rid of rarely used web browsers and operating systems, since we will encode all unique values into separate columns and we don’t want to get a very sparse matrix as the result. Another reason for that is due to the nature of the business domain, such outliers often indicate bot traffic or traffic source targeting issues, and should be filtered out.

The histogram for web browsers before applying filtering looks like that:

Filtering out web browsers that account for less than 1% of the dataset provides the following result:

The same applies to countries. Most advertisement campaigns are usually targeted to a specific list of countries and displaying an ad to a user that does not belong to that list will prevent a conversion and will result in wasted money. Low conversion rate can also indicate that the specific campaign is just not very effective in this country, so it makes sense to cut those losses and focus on more promising locations.

In our case there are quite a few countries that don’t convert much:

Applying the same 1% filter changes the picture drastically:

The next step is to make sure that categorical columns do not contain any characters that can result in processing errors. To achieve that, we will percent-encode them with a PySpark script:

1
2
3
4
5
6
7
8
9

import urllib
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

encode_udf = udf(lambda x: urllib.parse.quote(x, safe=''), StringType())

df = df.withColumn('dos', encode_udf('dos'))
df = df.withColumn('dbrowser', encode_udf('dbrowser'))
# Mobile Safari -> Mobile%20Safari

The last step of the common branch of the DAG is to encode categorical columns. In order to train and use an ML model, we need to transform (or encode) non-numerical values to numeric representations. We will use two different approaches for offers and for all other categorical columns that we have. The offers column will be encoded using an indexer, which produces a single column that contains integers, corresponding to a specific offer:

index,offer
0,012a096e-f414-4da7-bbdc-236da34bb545
1,41183105-2ed4-46c6-b654-2f0081aa652d

As for the remaining categorical columns (country, OS, device type and web browser), we will use a one-hot encoding approach, resulting in a new column (with values of 1 or 0) created for each unique value within a category:

country_TH,country_NP,country_BD,country_SA,country_DZ,country_VN,country_US,country_MA,country_ZA,country_MX,country_KE,country_JO,country_AE,country_AZ,country_IN,country_PL,country_ES,country_NG,country_LB,country_CL,country_FR,country_PE,country_NL,country_ID,dos_Android,dos_iOS,dtype_mobile,dtype_tablet,dtype_desktop,dbrowser_Chrome,dbrowser_Chrome%20WebView,dbrowser_Android%20Browser,dbrowser_UCBrowser,dbrowser_Samsung%20Browser,dbrowser_Facebook,dbrowser_Mobile%20Safari

The latitude and longitue columns will remain as is and will be included in the output. The city column will be omitted, because using only latitude and longitue provided similar results in our preliminary research. ISP, network, browser and OS version columns will be removed, since they produce a very sparse model and does not seem to correlate with the offer ID.

At this point we have reached the end of the common branch and the next steps will reflect transformations intended for a specific DAG output.

Data Wrangler Output

The first output that we need is a list of columns. To produce it, we’re just going to limit the number of rows intended for this output to one, so that the processing step won’t have to load a huge dataset file in order to just to get a list of columns in it:

1

SELECT * FROM df TABLESAMPLE(1 ROWS)

The exact same approach is used to get the second output - a list of encoded offers that we created earlier. The only difference is that the output will only contain two columns (index and offer ID) and everything else will be removed.

The last three outputs are samples of our dataset for ML model training, validation and testing. They are created by splitting the dataset in a proportion of 70%, 15% and 15% respectively:

1
2

train, validation, test = df.randomSplit(weights = [0.70, 0.15, 0.15], seed = 13)
df = train

That concludes the process of data preparation and we now approach the next step in our SageMaker pipeline.

Preprocessing

Data Wrangler operates on rows and columns, but in order to execute the next steps, we need to prepare some metadata. Preprocessing step is repsonsible for reading files with offers and columns data from the Data Wrangler output and converting it to JSON that we can reference in our SageMaker pipeline definition script.

Under the hood, it’s just a generic Python environment with various data science libraries pre-installed, that can be used to perform all kinds of data preparation and processing tasks.

Training

Once the data is prepared, we can proceed with training the model. We’re going to use XGBoost and create a model that performs a multiclass classification. In order to account for the fact that our pipeline is automatic and does not involve any manual tuning, we will run a hyperparameter optimization. It will generate several models with different parameters and the best performing model will be deployed to production.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

xgb_estimator.set_hyperparameters(
    objective="multi:softprob",
        # Multiclass with probabilities
    eval_metric="mlogloss",
        # Negative log-likelihood
        # https://en.wikipedia.org/wiki/Likelihood_function#Log-likelihood
    num_round="100",
        # The number of rounds for boosting
    num_class=JsonGet(
        step_name=step_process.name,
        property_file=classes_map,
        json_path="length_str",
    ),
        # Number of classes
)
xgb_tuner = HyperparameterTuner(
    estimator=xgb_estimator,
    objective_metric_name="validation:mlogloss",
    objective_type="Minimize", # The less log-likelihood is the better
    hyperparameter_ranges={
        # See https://xgboost.readthedocs.io/en/stable/parameter.html
        "alpha": ContinuousParameter(0.01, 10, scaling_type="Logarithmic"),
            # It can be used in case of very high dimensionality so that the algorithm runs faster when implemented.
            # Increasing this value will make model more conservative.
        "eta": ContinuousParameter(0.01, 0.2),
            # It is the step size shrinkage used in update to prevent overfitting.
            # After each boosting step, we can directly get the weights of new features,
            # and eta shrinks the feature weights to make the boosting process more conservative.
        "gamma": ContinuousParameter(0.0, 0.5),
            # A node is split only when the resulting split gives a positive reduction in the loss function.
            # Gamma specifies the minimum loss reduction required to make a split.
            # It makes the algorithm conservative. The values can vary depending on the loss function and should be tuned.
        "min_child_weight": ContinuousParameter(1, 10),
            # It defines the minimum sum of weights of all observations required in a child. It is used to control over-fitting.
            # Higher values prevent a model from learning relations which might be highly specific to the particular sample selected for a tree.
            # Too high values can lead to under-fitting. The larger min_child_weight is, the more conservative the algorithm will be.
        "max_depth": IntegerParameter(3, 10),
            # The maximum depth of a tree. It is used to control over-fitting as higher depth will allow model 
            # to learn relations very specific to a particular sample.
            # Increasing this value will make the model more complex and more likely to overfit.
        "subsample": ContinuousParameter(0.5, 1),
            # It denotes the fraction of observations to be randomly samples for each tree.
            # Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees.
            # This will prevent overfitting. Lower values make the algorithm more conservative and prevents overfitting 
            # but too small values might lead to under-fitting.
    },
    max_jobs=10, # Create up to 10 models
    max_parallel_jobs=5, # 5 parallel training job at a time
)

Evaluation

Even with the best performing model at hands, we still want to see if it’s worth deploying it to production. The suitable metric to evaluate a multiclass classification model is a hamming loss - a percentage of samples that were not predicted correctly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import numpy as np
import pandas as pd
from sklearn.metrics import hamming_loss
import xgboost

# ...

df = pd.read_csv(file_test)
y_test = df.iloc[:, 0].to_numpy()
df.drop(df.columns[0], axis=1, inplace=True)
X_test = xgboost.DMatrix(df.values)

num_class = len(np.unique(y_test))
num_samples = y_test.shape[0]

model = xgboost.Booster()
model.load_model('xgboost-model')

predictions = model.predict(X_test)

predictions = predictions.reshape(num_samples, num_class)
pred_label = np.argmax(predictions, axis=1)

hloss = hamming_loss(y_test, pred_label)

Our pipeline will only proceed with models that have a Hamming loss of 40% or less. Models that don’t fit into that threshold will be discarded and previously deployed model will remaing in production.

Deployment

The last step in the pipeline is model deployment. It is implemented by a Lambda function that loads the model into an S3 bucket that SageMaker inference endpoint uses to access the models, updates the model metadata in a DynamoDB table and performs a cleanup.

Testing

We can now finally call our API and get predictions. The API requires only one parameter - a campaign ID:

curl https://$API_GATEWAY_HOST/predict/97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b

{
  // Model input
  "input": "100.5997,13.5989,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,0,0,0,0,0",
  // Model output
  "output": "[[0.49411922693252563, 0.5058807730674744]]",
  // Predicted offer
  "predicted": {
    "index": 1,
    "offer": "41183105-2ed4-46c6-b654-2f0081aa652d",
    "probability": 0.5058807730674744
  },
  // Analyzed data
  "source": {
    "ip": "16.170.153.144",
    "dos": "Mac OS",
    "dosversion": "10.15.7",
    "dbrowser": "Chrome",
    "dbrowserversion": "98.0.4758.109",
    "isp": "AMAZON-02",
    "network": "16.168.0.0/14",
    "country": "SE",
    "city": "Stockholm",
    "long": 18.0717,
    "lat": 59.3287
  },
  // All predicted offers
  "predictions": [
    { "index": 1, "probability": 0.5058807730674744, "offer": "41183105-2ed4-46c6-b654-2f0081aa652d" },
    { "index": 0, "probability": 0.49411922693252563, "offer": "012a096e-f414-4da7-bbdc-236da34bb545" }
  ],
  // Data to model mappings
  "mappings": {"long":100,"lat":13,"country_TH":1,"country_NP":0,"country_BD":0,"country_SA":0,"country_DZ":0,"country_VN":0,"country_US":0,"country_MA":0,"country_ZA":0,"country_MX":0,"country_KE":0,"country_JO":0,"country_AE":0,"country_AZ":0,"country_IN":0,"country_PL":0,"country_ES":0,"country_NG":0,"country_LB":0,"country_CL":0,"country_FR":0,"country_PE":0,"country_NL":0,"country_ID":0,"dos_Android":1,"dos_iOS":0,"dtype_mobile":1,"dtype_tablet":0,"dtype_desktop":0,"dbrowser_Chrome":0,"dbrowser_Chrome%20WebView":1,"dbrowser_Android%20Browser":0,"dbrowser_UCBrowser":0,"dbrowser_Samsung%20Browser":0,"dbrowser_Facebook":0,"dbrowser_Mobile%20Safari":0},
  // Model metadata
  "endpoint": {
    "campaign": "97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b",
    "model": "3cb70203-a1e4-7f82-b7b4-b8c89bc67305",
    // CSV columns
    "columns": ["long","lat","country_TH","country_NP","country_BD","country_SA","country_DZ","country_VN","country_US","country_MA","country_ZA","country_MX","country_KE","country_JO","country_AE","country_AZ","country_IN","country_PL","country_ES","country_NG","country_LB","country_CL","country_FR","country_PE","country_NL","country_ID","dos_Android","dos_iOS","dtype_mobile","dtype_tablet","dtype_desktop","dbrowser_Chrome","dbrowser_Chrome%20WebView","dbrowser_Android%20Browser","dbrowser_UCBrowser","dbrowser_Samsung%20Browser","dbrowser_Facebook","dbrowser_Mobile%20Safari"],
    // Category encodings
    "classes": {
      "012a096e-f414-4da7-bbdc-236da34bb545": "0",
      "41183105-2ed4-46c6-b654-2f0081aa652d": "1"
    },
    // Model evaluation results
    "evaluation": {
      "hamming_loss": 0.3316062176165803 // Loss rate is 33%
    },
    "modified": 1646122738
  }
}

The model predicted that the current user will most likely convert with offer 41183105-2ed4-46c6-b654-2f0081aa652d and the possibility that this offer is the best for this user compared to other offers is slightly above 50%. The response also contains detailed information about input and output data, as well as the model information.

The API also supports an optional input argument that can be used to provide the model with an input directly, without analyzing the current user’s IP address and user agent string. The input format is a CSV without column names and the order of columns must be the same as described in endpoint.columns field:

curl https://$API_GATEWAY_HOST/predict/97318a5c-bbab-4a5a-8a28-c6ab6fcbd79b?input=100.5997,13.5989,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,0,0,0,0,0

Conclusion

Building a machine learning based system is a complex subject that requires a committed effort of a team of engineers and analysts. The ability to sustain constant evolution is one of the most important traits such system must have in order to succeed. Leveraging modern tools and cloud services designed with this in mind opens up a possibility for even small teams to build powerful and sophisticated ML applications.

The source code for this project is available on GitHub. You can run it locally or you can take a look at the demo app to get an idea on how it works. This service does only one thing: it processes a video from a webcam, detects faces and calculates if people it sees have been seen before. This is a generic computer vision task that many applications nowadays leverage upon, such as facial identification on your phone or security systems with access control.

First, let’s get the bigger picture on how this system operates.

Architecture

The service consists of three distinct parts: a WebRTC server, an application server and a recognition server. Client applications both stream video data and get facial recognition results back over WebRTC. The process can be described with the following sequence diagram:

Now let’s take a closer look on each actor of the system.

Client application

The client in this example is a web application built with React, Redux and Next.js, but it is also possible to create a mobile or desktop application, since there are WebRTC implementations for most modern platforms. Even more, with just some small changes, you can process video from IP cameras over RTSP.

The client application also acts as a temporary store of face data received from the recognition server. In a real-world system, this task will be performed on a server-side and will involve a database, but for the sake of simplicity, this is out of the scope of this article.

WebRTC signaling and streaming is performed by the OpenVidu client over WebSockets, so the WebRTC implementation is trivial.

WebRTC server

The WebRTC server consists of several building blocks. At its core, it is based on Kurento (a media server) and OpenVidu (a signaling server), both are open-source solutions with a paid support for OpenVidu that allows to scale to multiple nodes. The third part of the solution is Coturn – a TURN server, which is used for NAT traversal. There are also Redis and NGINX involved internally to handle the state management and to secure communcation between the system components.

Overall, this solution works out of the box and have unopinionated client libraries available, so we’re not going to dive into the details here. It’s worth mentioning that OpenVidu has recently released a mediasoup support to replace Kurento, which can dramatically improve the overall performance and provide a low-level access for Node.js applications.

Application server

This server is responsible for the following tasks:

• Provide an API to securely initiate WebRTC sessions. It is possible to initiate a session from the client directly, but that entails security risks and should not be used in production.
• Connect to client video broadcasts and take regular snapshots. The server acts as a WebRTC client and saves video frames to canvas for further processing.
• Provide snapshots to the recognition worker and relay the results back to clients. When the worker responses with face data, the server crops a previously stored snapshot to only include regions that contain faces and sends it to the client alongside with face encodings.
• Host the client web application. This is not a must, but rather a neat feature that Next.js provides.

Recognition worker

Last but not least, the server that is responsible for implementing facial recognition tasks. It is a Python web server that utilizes the infamous face_recognition library, which is used to find and to encode faces on images. In this example it’s going to be as simple as that: a stateless server that uses a readily available library to do all the heavy lifting. We’re going to look at possible directions on how to improve upon this solution at the end of the article.

Algorithm

Ultimately, we want to be able to tell if two photos contain a face of the same person. But what we’re going to compare, exactly? In order to perform a comparison, we need to transform a photo, which is merely an array of pixels, to something more meaningful. This process is called face encoding. It can vary based on the approach and algorithm used, but overall you can think of it as a measurement of facial features, such as a distance between the eyes, eyebrows, nose or lips, their size and position. The composition of these features can be represented by an N-dimensional vector (in this case, a 128-dimensional one), sometimes called face vector. By calculating the Euclidean distance between two face vectors, we can measure how similar (close) those faces (vectors) are.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

const newFace = {
  encodings: [/* face vector */],
};

const candidates = [];

// Compare the face with all existing faces
for (const existing of existingFaces) { 
  // Calculate the Euclidean distance, which is a measure
  // of how two faces are similar to each other.
  // The lower the distance, the more similarity they share.
  const sum = existing.encodings.reduce((res, x1, i) => {
    const x2 = newFace.encodings[i];

    return res + ((x1 - x2) ** 2);
  }, 0)

  const distance = Math.sqrt(sum);

  // Only consider faces that are similar enough (distance < 0.6)
  if (distance >= 0 && distance < state.threshold) {
    candidates.push({
      match: existing,
      distance,
    });
  }
}

// Find a face with the shortest euclidean distance 
const candidate = candidates
  .sort((a, b) => a.distance - b.distance)[0];

// Calculate the similarity percentage
if (candidate) {
  const linear = 1.0 - (candidate.distance / (state.threshold * 2.0));
  const score = linear + ((1.0 - linear) * Math.pow((linear - 0.5) * 2, 0.2));
  const similarity = Math.round(score * 100);

  alert(`Found a face with the similarity of ${similarity}%`);
} else {
  existingFaces.push(newFace);

  alert(`A new face detected`);
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

// This is an example of a face vector
[
 -0.08237294852733612, 0.06998157501220703, 0.04017249494791031, -0.04595483839511871,
 -0.20524457097053528, 0.06130267679691315, -0.003034330904483795, -0.034762077033519745,
 0.07720481604337692, 0.0018782643601298332, 0.26475948095321655, -0.06688505411148071,
 -0.2565714120864868, -0.024762030690908432, -0.017667800188064575, 0.08394743502140045,
 -0.1338515281677246, -0.0945650115609169, -0.10690569132566452, -0.10622785240411758,
 0.02856566198170185, -0.002836777362972498, -0.03754136338829994, 0.033261124044656754,
 -0.2024488002061844, -0.25246909260749817, -0.030827492475509644, -0.007754474878311157,
 0.08512576669454575, -0.15835121273994446, 0.09394755959510803, 0.19072295725345612,
 -0.17289769649505615, -0.04181254655122757, 0.04165133833885193, 0.14661748707294464,
 -0.11559691280126572, -0.015472346916794777, 0.19385945796966553, -0.04230993986129761,
 -0.13715803623199463, 0.09014783799648285, 0.09632508456707001, 0.29142847657203674,
 0.16114553809165955, 0.0382876954972744, 0.025807470083236694, 0.0014453493058681488,
 0.08283577114343643, -0.32194820046424866, 0.012574739754199982, 0.18525069952011108,
 0.16502265632152557, 0.0586332231760025, 0.02298889309167862, -0.13721558451652527,
 -0.0279446542263031, 0.14717113971710205, -0.14877578616142273, 0.15434607863426208,
 0.10199335217475891, -0.21128228306770325, -0.08583835512399673, -0.14078442752361298,
 0.15042155981063843, 0.046597909182310104, -0.14368422329425812, -0.1568884551525116,
 0.1736888289451599, -0.088925302028656, -0.11250282824039459, 0.16585050523281097,
 -0.06869833171367645, -0.16567328572273254, -0.25068527460098267, 0.12233469635248184,
 0.4443138837814331, 0.12621524930000305, -0.21290765702724457, -0.00904833059757948,
 -0.028415795415639877, 0.003739515785127878, -0.04296185076236725, 0.032679952681064606,
 -0.06143368408083916, -0.1341024786233902, -0.1555326282978058, 0.04243481904268265,
 0.20805151760578156, -0.0508725643157959, -0.01405704952776432, 0.18905237317085266,
 0.1028638407588005, 0.04386136680841446, 0.0037655732594430447, 0.059780303388834,
 -0.11599305272102356, -0.0248013436794281, -0.13408342003822327, -0.01498719397932291,
 0.040145330131053925, -0.13561423122882843, 6.418908014893532e-05, 0.07700222730636597,
 -0.08549322187900543, 0.2858748733997345, -0.01664814166724682, -0.013422049582004547,
 -0.010783687233924866, 0.019620651379227638, -0.09562011063098907, 0.03741421923041344,
 0.19683292508125305, -0.33195415139198303, 0.1915217787027359, 0.17226657271385193,
 0.07776990532875061, 0.11128266900777817, 0.14693227410316467, 0.09936179965734482,
 0.037961095571517944, 0.017703596502542496, -0.12619523704051971, -0.18041686713695526,
 0.0061234827153384686, -0.09858812391757965, -0.0010826261714100838, 0.04442061856389046
]

Going further

While this example is an MVP and doesn’t cut many corners, it still requires some attention before it can be used in production. There are a few things to consider if you’re going to build a real-world service on top of it.

Machine learning pipeline

If you ran this services yourself, you could have noticed that image processing takes a decent amount of time. One of the reasons behind that is face_recognition algorithm used for face detection. The process of face encoding is preceded by face detection. In order to encode facial features, you need to find them on the image first. Instead of relying on the library’s ability to both detect faces and to encode facial features, you could use a faster ML model to detect faces first, then find and encode facial features in a just small region of the image. This becomes critical when processing higher definition images and handling a lot of requests at the same time.

Moreover, your service could also use specialized ML models to classify faces by age or gender. That opens huge possibilities, but also requires a lot of fine-tuning and performance optimizations.

Client-side processing

Some of the pipeline steps can (and should, if possible) be offloaded to the client side. Ideally, the client should be responsible for encoding face data, and the only payload to be sent over the network would be a face vector. This greatly reduces the network footprint and gets rid of complex WebRTC infrastructure. Unfortunately, image processing and ML models are very resource-intensive, and not any client hardware can handle it.

Hardware acceleration

A model inference can have a much higher performance by leveraging a GPU, but since it’s highly platform-dependent (especially when you put Docker in the mix), it was intentionally disabled. Moreover, depending on your requirements, your application could also benefit from a hardware-accelerated canvas. And even without a discrete GPU, you could still optimize the performance by using the right CPU instructions or CPU architecture that your ML framework is optimized for.

Vector database

This demo application stores face data in memory. In a production application, you would likely do that using persistent storage, such as a database. One of the most widely used solutions is Postgres with the pg_vector extension, which enables efficient indexing and querying of N-dimensional vector columns. There are also specialized databases, such as Pinecone or Qdrant, which, depending on the specific needs, could be a good choice for a vector store and a retrieval engine.

Test Environment

We’re going to use Node.js 14 inside Docker as the runtime for test functions of various sizes: from 128 MB to 10240 MB, each one twice as big as the previous. The infrastructure will be managed with Pulumi. The source code is available on GitHub.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

for (const arch of ['x86_64', 'arm64']) {
  for (let i = 7; i <= 14; i++) {
    const size = Math.min(2 ** i, 10240);
    const name = `${project}-${stack}-${arch}-${size}`;

    new aws.lambda.Function(name, {
      name: name,
      architectures: arch,
      memorySize: size,
      timeout: 60,
      role: role.arn,
      description: `${project}-${stack} - Worker (${arch} ${size} MB).`,
      imageUri: pulumi.interpolate`${account}.dkr.ecr.${region}.amazonaws.com/${repository.name}:latest`,
      packageType: 'Image',
    });
  }
}

All function will run benchmarks that last 100 seconds and executed twice: the first run is intended to warm up the environment, the second provides the final results.

Express.js Benchmark

The first benchmark uses a simple Express.js server that handles requests. The server is invoked from within the same function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

const app = express();

app.post('/', (req, res) => {
  res.json(req.body);
});

const server = app.listen();

benny.suite('Lambda benchmark', 
  benny.add('express', () => {
    const runner = async () => {
      await axios.post(`http://localhost:${server.address()!['port']}/`, {});
    };

    return runner;
  }, {
    maxTime: 10,
  }),
  benny.cycle(),
);

The results are quite interesting:

As you can see, both x64 and ARM64 have a similar performance in functions up to 512 MB. As you might know, Lambda’s CPU performance and core count scales up alongside the allocated RAM. Both architectures peak at around 2048 MB, with x64 leading the race. The performance starts to decline after 4096 MB, which could be an indication that high-memory functions have more CPU cores in exchange for a lower performance per core. Here’s the distribution of the number of CPU cores:

Because this benchmark does not rely on asynchronous code or clustering, there is no gain from extra CPU cores.

Since ARM64 has a lower price than x64, let’s compare the two in terms of price performance:

Just as expected, ARM64 has up to a 20% lower cost per operation than x64. The largest gap is within 256 and 512 MB range, which is often used for APIs and other Express.js scenarios, so it’s worth checking out if your Lambda web services can be migrated to ARM64 to get a cost reduce.

Native Code Benchmark

In the second benchmark we’re going to use Sharp.js - a Node.js binding to image processing library libvips, which is available for both x64 and ARM64. The benchmark scales down a 1280x800 image to 640x400 and saves it as a JPEG file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

sharp.cache(false);
sharp.concurrency(os.cpus().length);

benny.suite('Lambda benchmark', 
  benny.add('sharp', () => {
    const runner = async () => {
      await sharp('./image.jpeg')
        .resize(640, 400)
        .jpeg({ quality: 80 })
        .toBuffer();
    };

    return runner;
  }, {
    maxTime: 10,
  }),
  benny.cycle(),
);

The results of this benchmark are unexpected:

Both CPUs perform on on par with each other up until 2048 MB, after which ARM64 breaks ahead. This could be just a result of a better performance of libvips on ARM64. Let’s see what is the price of the two:

The difference in performance price is more profound in this benchmark, with ARM64 being up to 27% less expensive than x64. This makes ARM64 an attractive alternative for computationally intensive functions.

Conclusion

Migrating your Lambda functions from x64 to ARM64 can give you a noticeable improvement in price performance. Of course, your mileage may vary, so you should consider your own research before making any assumptions.

Startup Performance

The cornerstone of Lambda performance is startup time. Whenever a Lambda function is invoked for the first time, it spins up a function instance and goes through the runtime initialization. This process can take a noticeable amount of time which is often unacceptable in production. A new Lambda instance can also get created during increased usage, parallel invocations or after a function has been idle for a while. To mitigate this issue, it is possible to pre-warm a Lambda function, so that it will already be initialized by the time the first request will come in. There are currently two similar approaches to choose from.

WarmUp Plugin

If you are using the Serverless Framework, you can implement function pre-warm using the Serverless WarmUp Plugin. It works by invoking your Lambda functions from a “warmer” function on a specified schedule (say, every five minutes) to simulate a user request or an event. You can also configure a concurrency at which your function will be invoked, which effectively provisions multiple function instances.

It is also possible to provide additional function initialization code, for example to connect to a database or instantiate any resource-heavy code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

// Lambda handler
export const handler = async (event, context) => {
  if (event.source === 'serverless-plugin-warmup') {
    // Lambda initialization code.
    await connect();

    return;
  }

  // Regular code
  const db = await connect();

  return await db.query();
};

// Database connection code
let dbClient;
let dbClientTask:Promise<any>;

async function connect() {
  if (dbClient) {
    return dbClient;
  }

  if (!dbClientTask) {
    const task = async () => {
      dbClient = new DbClient();

      await dbClient.connect();
    };

    dbClientTask = task();
  }

  try {
    return await dbClientTask;
  } finally {
    dbClientTask = undefined;
  }
};

Note that it is not guaranteed that the initialization code will be invoked before an actual request is made or that it will be invoked at all, so make sure that your initialization logic is lazy is idempotent.

Keep in mind that the warmup process will add up to your Lambda bill just like any other Lambda request. A single function configured to be invoked every five minutes will account for 8640 calls each month. Depending on your requirements, it could be benifitial to only pre-warm your function during business hours. You can do so by providing a CRON expression to your warmup configuration:

1
2
3
4
5
6

custom:
  warmup:
    default:
      enabled: true
      events:
        - schedule: "cron(0/5 8-17 ? * MON-FRI *)"

Provisioned concurrency

AWS introduced a native way to pre-warm Lambda functions called “provisioned concurrency”. You just need to provide a number of function instances you want to keep warm and AWS will take care of the rest. If you are using the Serverless framework, you can configure provisioned concurrency in the following way:

1
2
3

functions:
  func:
    provisionedConcurrency: 2

The way pre-warmed functions get initialized differs from the WarmUp plugin. With provisioned concurrency, the handler function is not called upon initialization. This results in a drawback that prevents from running an asynchronous initialization code in Node.js. In order to get around this limitation, you can use a Lambda layer that allows to provide a callback during the initialization.

One of the benefits of this being a native feature is that it is possible to automatically scale a provisioned concurrency for a function based on its utilization or on a schedule with scaling policies.

Whenever a provisioned concurrency is configured for a function, it will get a discount on the execution duration cost, but will also get an additional fee as long as this feature is active. See AWS Lambda pricing for the details.

Code Optimization

The package size of a Lambda function affects the time it takes to initialize it. There is also an account-level limit on the total size of all Lambda functions, so it is advisable to keep the size of your functions in check. This is especially relevant in case of Node.js functions with lots of npm dependencies.

By default, npm dependencies get packaged with a function code as-is, with all their contents intact. Many packages include things like documentation or even media files that are not needed in order to run these packages. A possible solution to this issue is to use a code bundler like Webpack (or better yet, Serverless Bundle) to transpile dependencies into a single output file. The resulting package can easily be multiple times smaller than the same function with included node_modules directory.

Connectivity

Another important topic related to Lambda performance is network connectivity. Lambda function instances can be created or teared down at any time, execution environment can be freezed in between requests. All of that can affect persistent network connections and increase reconnection rate, resulting in subpar application performance.

AWS SDK

If you’re still using AWS SDK v2 for Node.js, there is a single-line change that can greatly increase its performance. Just add AWS_NODEJS_CONNECTION_REUSE_ENABLED=1 as an environment variable of your function. As the name suggests, it enables a reuse of TCP connections, so subsequent requests to AWS API will take noticeable less time. AWS SDK v3 has this behavior enabled by default.

Database Proxying

The best practice when using a remote database in Lambda is to use a proxy, either a stateless API (like DynamoDB) or a connection pool. Since Lambda can scale up to multiple instances really quick and can dispose an existing one at any time, it can result in too many connections to a database server. AWS RDS Proxy is a serverless solution that provides connection pooling for RDS databases. If you don’t use RDS, there are similar products for most of the popular databases, such as PgBouncer for PostgreSQL.

Monitoring & Profiling

It is important to understand the performance bottlenecks of your Lambda functions when using it for mission-critical workflows and in high-load projects. With serverless approach, you can’t rely on a toolset that exists in the world of classic servers, but there are a few AWS products that can help you to get the job done.

Lambda Insights

By defaut, Lambda provides only a handful of performance metrics, such as execution duration and concurrency. If you want to get more information on what’s actually hapenning in your functions, you can enable Lambda Insights. With that in place, you will get detailed metrics for CPU, RAM and IO usage, as well as the information on cold function starts. This data is crucial for fine-tuning your function memory size, especially if you peform computationally-intensive tasks in Lambda.

X-Ray

Last but not least is tracing service for your distributed applications: AWS X-Ray. If you lerevage a microservice approach, you absolutely need to monitor how your requests are propagated through the system to identify any otherwise hard to pinpoint performance issues that can arise. With just a few lines of code, you can break down your requests into sequence diagrams and see which parts of your system needs an attention.

Conclusion

With just a few easy steps you can make you Lambda functions much faster without even changing the architecture.