{"id":3699,"date":"2023-08-23T12:04:02","date_gmt":"2023-08-23T17:04:02","guid":{"rendered":"https:\/\/fgiasson.com\/blog\/?p=3699"},"modified":"2023-08-23T12:04:07","modified_gmt":"2023-08-23T17:04:07","slug":"how-to-deploy-hugging-face-models-in-a-docker-container","status":"publish","type":"post","link":"https:\/\/fgiasson.com\/blog\/index.php\/2023\/08\/23\/how-to-deploy-hugging-face-models-in-a-docker-container\/","title":{"rendered":"How to Deploy Hugging Face Models in a Docker Container"},"content":{"rendered":"\n

In this short tutorial, we will explore how Hugging Face<\/a> models can be deployed in a Docker<\/a> Container and exposed as a web service endpoint.<\/p>\n

The service it exposes is a translation service from English to French and French to English.<\/p>\n

Why someone would like to do that? Other than to learn about those specific technologies, it is a very convenient way to try and test the thousands of models that exists on Hugging Face, in a clean and isolated environment that can easily be replicated, shared or deployed elsewhere than on your local computer.<\/p>\n

In this tutorial, you will learn how to use Docker<\/code> to create a container with all the necessary code and artifacts to load Hugging Face models and to expose them as web service endpoints using Flask<\/code>.<\/p>\n

All code and configurations used to write this blog post are available in this GitHub Repository<\/a>. You simply have to clone it and to run the commands listed in this tutorial to replicate the service on your local machine.<\/p>\n\n\n\n

Installing Docker<\/h2>\n\n\n\n

The first step is to install Docker. The easiest way is by simply installing Docker Desktop<\/a> which is available on MacOS, Windows and Linux.<\/p>\n\n\n\n

Creating the Dockerfile<\/h2>\n\n\n\n

The next step is to create a new Git<\/code> repository where you will create a Dockerfile<\/code>. The Dockerfile is where all instructions are written that tells Docker how to create the container.<\/p>\n

I would also strongly encourage you to install and use hadolint<\/a>, which is a really good Docker linter that helps people to follow Docker best practices. There is also a plugin for VS Code<\/a> if this is what you use as you development IDE.<\/p>\n\n\n\n

Base image and key installs<\/h3>\n\n\n\n

The first thing you define in a Dockerfile is the base image to use to initialize the container. For this tutorial, we will use Ubuntu’s latest LTS:<\/p>\n\n\n\n

# Use Ubuntu's current LTS\nFROM ubuntu:jammy-20230804<\/code><\/pre><\/div>\n\n\n\n
Since we are working to create a Python web service that expose the predictions of a ML model, the next step is to add they key pieces required for the Python service. Let’s make sure that you only include what is necessary to minimize the size, and complexity, of the container as much as possible:<\/p>\n\n\n\n
# Make sure to not install recommends and to clean the \n# install to minimize the size of the container as much as possible.\nRUN apt-get update && \\\n    apt-get install --no-install-recommends -y python3=3.10.6-1~22.04 && \\\n    apt-get install --no-install-recommends -y python3-pip=22.0.2+dfsg-1ubuntu0.3 && \\\n    apt-get install --no-install-recommends -y python3-venv=3.10.6-1~22.04 && \\\n    apt-get clean && \\\n    rm -rf \/var\/lib\/apt\/lists\/*<\/code><\/pre><\/div>\n\n\n\n
This instruct Docker to install Python3<\/code>, pip<\/code> and venv<\/code>. It also ensures that apt get cleaned of cached files, that nothing more is installed and that we define the exact version of the package we want to install. That is to ensure that we minimize the size of the container, while making sure that the container can easily be reproduced, with the exact same codebase, any time in the future.<\/p>\n
Another thing to note: we run multiple commands with a single RUN<\/code> instruction by piping them together with &&<\/code>. This is to minimize the number of layers created by Docker for the container, and this is a best practice to follow when creating containers. If you don’t do this and run hadolint<\/code>, then you will get warning suggesting you to refactor your Dockerfile<\/code> accordingly.<\/p>\n\n\n\n
Copy required files<\/h3>\n\n\n\n
Now that the base operating system is installed, the next step is to install all the requirements of the Python project we want to deploy in the container:<\/p>\n\n\n\n
# Set the working directory within the container\nWORKDIR \/app\n\n# Copy necessary files to the container\nCOPY requirements.txt .\nCOPY main.py .\nCOPY download_models.py .<\/code><\/pre><\/div>\n\n\n\n
First we define the working directory with the WORKDIR<\/code> instruction. From now on, every other instruction will run from that directory in the container. We copy the local files: requirements.txt<\/code>, main.py<\/code> and download_models.py<\/code> to the working directory.<\/p>\n\n\n\n
Create virtual environment<\/h3>\n\n\n\n
Before doing anything with those files, we are better creating a virtual environment where to install all those dependencies. Some people may wonder why we create an environment within an environment? It is further isolation between the container and the Python application to make sure that there is no possibility of dependencies clashes. This is a good best practice to adopt.<\/p>\n\n\n\n
# Create a virtual environment in the container\nRUN python3 -m venv .venv\n\n# Activate the virtual environment\nENV PATH="\/app\/.venv\/bin:$PATH"<\/code><\/pre><\/div>\n\n\n\n
Install application requirements<\/h3>\n\n\n\n
Once the virtual environment is created and activated in the container, the next step is to install all the required dependencies in that new environment:<\/p>\n\n\n\n
    # Install Python dependencies from the requirements file\nRUN pip install --no-cache-dir -r requirements.txt && \\\n    # Get the models from Hugging Face to bake into the container\n    python3 download_models.py<\/code><\/pre><\/div>\n\n\n\n
It runs pip install<\/code> to install all the dependencies listed in requirements.txt<\/code>. The dependencies are:<\/p>\n\n\n\n
transformers==4.30.2\nflask==2.3.3\ntorch==2.0.1\nsacremoses==0.0.53\nsentencepiece==0.1.99<\/code><\/pre><\/div>\n\n\n\n
Just like the Ubuntu package version, we should (have to!) pin (specify) the exact version of each dependency. This is the best way to ensure that we can reproduce this environment any time in the future and to prevent unexpected crashes because code changed in some downstream dependencies that causes issues with the code.<\/p>\n\n\n\n
Downloading all models in the container<\/h3>\n\n\n\n
As you can see in the previous RUN<\/code> command, the next step is to download all models and tokenizers in the working directory such that we bake the model’s artifacts directly in the container. That will ensures that we minimize the time it takes to initialize a container. We spend the time to download all those artifacts at build time instead of run time. The downside is that the containers will be much bigger depending on the models that are required.<\/p>\n
The download_models.py<\/code> file is a utility file used to download the Hugging Face models used by the service directly into the container. The code simply download the models and tokenizer files from Hugging Face and save them locally (in the working directory of the container):<\/p>\n\n\n\n
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\nimport os\n\ndef download_model(model_path, model_name):\n    """Download a Hugging Face model and tokenizer to the specified directory"""\n    # Check if the directory already exists\n    if not os.path.exists(model_path):\n        # Create the directory\n        os.makedirs(model_path)\n\n    tokenizer = AutoTokenizer.from_pretrained(model_name)\n    model = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n\n    # Save the model and tokenizer to the specified directory\n    model.save_pretrained(model_path)\n    tokenizer.save_pretrained(model_path)\n\n# For this demo, download the English-French and French-English models\ndownload_model('models\/en_fr\/', 'Helsinki-NLP\/opus-mt-en-fr')\ndownload_model('models\/fr_en\/', 'Helsinki-NLP\/opus-mt-fr-en')<\/code><\/pre><\/div>\n\n\n\n
Creating the Flask translation web service endpoint<\/h3>\n\n\n\n
The last thing we have to do with the Dockerfile is to expose the port where the web service will be available and to tell the container what to run when it starts:<\/p>\n\n\n\n
# Make port 6000 available to the world outside this container\nEXPOSE 6000\n\nENTRYPOINT [ "python3" ]\n\n# Run main.py when the container launches\nCMD [ "main.py" ]<\/code><\/pre><\/div>\n\n\n\n
We expose the port 6000<\/code> to the outside world, and we tell Docker to run the python3<\/code> command with main.py<\/code>. The main.py<\/code> file is a very simple file that register the web service’s path using Flask, and that makes the predictions (translations in this case):<\/p>\n\n\n\n
from flask import Flask, request, jsonify\nfrom transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\ndef get_model(model_path):\n    """Load a Hugging Face model and tokenizer from the specified directory"""\n    tokenizer = AutoTokenizer.from_pretrained(model_path)\n    model = AutoModelForSeq2SeqLM.from_pretrained(model_path)\n    return model, tokenizer\n\n# Load the models and tokenizers for each supported language\nen_fr_model, en_fr_tokenizer = get_model('models\/en_fr\/')\nfr_en_model, fr_en_tokenizer = get_model('models\/fr_en\/')\n\napp = Flask(__name__)\n\ndef is_translation_supported(from_lang, to_lang):\n    """Check if the specified translation is supported"""\n    supported_translations = ['en_fr', 'fr_en']\n    return f'{from_lang}_{to_lang}' in supported_translations\n\n@app.route('\/translate\/<from_lang>\/<to_lang>\/', methods=['POST'])\ndef translate_endpoint(from_lang, to_lang):\n    """Translate text from one language to another. This function is \n    called when a POST request is sent to \/translate\/<from_lang>\/<to_lang>\/"""\n    if not is_translation_supported(from_lang, to_lang):\n        return jsonify({'error': 'Translation not supported'}), 400\n\n    data = request.get_json()\n    from_text = data.get(f'{from_lang}_text', '')\n\n    if from_text:\n        model = None\n        tokenizer = None\n\n        match from_lang:\n            case 'en':        \n                model = en_fr_model\n                tokenizer = en_fr_tokenizer\n            case 'fr':\n                model = fr_en_model\n                tokenizer = fr_en_tokenizer\n\n        to_text = tokenizer.decode(model.generate(tokenizer.encode(from_text, return_tensors='pt')).squeeze(), skip_special_tokens=True)\n\n        return jsonify({f'{to_lang}_text': to_text})\n    else:\n        return jsonify({'error': 'Text to translate not provided'}), 400\n    \nif __name__ == '__main__':\n    app.run(host='0.0.0.0', port=6000, debug=True)\n<\/code><\/pre><\/div>\n\n\n\n
Building the container<\/h2>\n\n\n\n
Now that the Dockerfile<\/code> is completed, the next step is to use it to have Docker to build the actual image of the container. This is done using this command in the terminal:<\/p>\n\n\n\n
docker build -t localbuild:en_fr_translation_service .<\/code><\/pre><\/div>\n\n\n\n
Note that we specified a tag to make it easier to manage it in between all the other images that may exists in the environment. The output of the terminal will show every step defined in the Dockerfile<\/code>, and the processing for each of those step. The final output looks like:<\/p>\n
\"\"<\/a><\/p>\n\n\n\n
Running and Querying the service<\/h2>\n\n\n\n
Now that we have a brand new image, the next step is to test it. In this section, I will use Docker Desktop’s user interface to show how we can easily do this, but all those step can easily be done (and automated) using the docker<\/code> command line application.<\/p>\n
After you built the image, it will automatically appear in the images<\/code> section of Docker Desktop:<\/p>\n
\"\"<\/a><\/p>\n
<\/p>\n\n\n\n
You can see the tag of the image, its size, when it was created, etc. To start the container from that image, we simply have to click the play arrow<\/code> in the Actions<\/code> column. That will start running a new container using that image.<\/p>\n
Docker Desktop will enable you to add some more parameter to start the container with the following window:<\/p>\n
\"\"<\/a><\/p>\n
<\/p>\n
The most important thing to define here is to Host port<\/code>. If you leave it empty, then the port 6000<\/code> we exposed in the Docker file will become unbound<\/code> and we won’t be able to reach the service running in the container.<\/p>\n\n\n\n
Once you click the Run<\/code> button, the container will appear in the Containers<\/code> section:<\/p>\n
\"\"<\/a><\/p>\n
<\/p>\n
And if you click on it’s name’s link, you will have access to the internal of the container (the files it contains, the execution logs, etc.:<\/p>\n
\"\"<\/a><\/p>\n
<\/p>\n
<\/p>\n\n\n\n
Now that the container is running, we can query the endpoint like this:<\/p>\n\n\n\n
curl http:\/\/localhost:6000\/translate\/en\/fr\/ POST -H "Content-Type: application\/json" -v -d '{"en_text": "Towards Certification of Machine Learning-Based Distributed Systems Behavior"}'<\/code><\/pre><\/div>\n\n\n\n
It returns:<\/p>\n\n\n\n
{\n  "fr_text": "Vers la certification des syst\\u00e8mes distribu\\u00e9s fond\\u00e9s sur l'apprentissage automatique"\n}<\/code><\/pre><\/div>\n\n\n\n
And then for the French to English translation:<\/p>\n\n\n\n
curl http:\/\/localhost:6000\/translate\/fr\/en\/ POST -H "Content-Type: application\/json" -v -d '{"fr_text": "Ce qu'\\''il y a d'\\''admirable dans le bonheur des autres, c'\\''est qu'\\''on y croit."}'<\/code><\/pre><\/div>\n\n\n\n
It returns:<\/p>\n\n\n\n
{\n  "en_text": "What is admirable in the happiness of others is that one believes in it."\n}<\/code><\/pre><\/div>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
As we can see, it is pretty straightforward to create simple Docker containers that turns pretty much any Hugging Face pre-trained models into a web service endpoint.<\/p>\n","protected":false},"excerpt":{"rendered":"
In this short tutorial, we will explore how Hugging Face models can be deployed in a Docker Container and exposed as a web service endpoint. The service it exposes is a translation service from English to French and French to English. Why someone would like to do that? Other than to learn about those specific […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[293,309],"tags":[310,311,312],"class_list":["post-3699","post","type-post","status-publish","format-standard","hentry","category-artificial-intelligence","category-mlops","tag-docker","tag-huggingface","tag-mlops"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3699","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=3699"}],"version-history":[{"count":14,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3699\/revisions"}],"predecessor-version":[{"id":3718,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3699\/revisions\/3718"}],"wp:attachment":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=3699"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=3699"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=3699"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}