Case Study: Python Application

Python is the go-to language for many development teams to build web applications, ranging from Django-based monoliths to lightweight FastAPI-based microservices. Unlike Node.js, which has a single all-purpose runtime, the Python ecosystem is split across several dependency management approaches (pip, poetry, virtualenv) and two server models that are not directly compatible with each other: WSGI for synchronous applications like Django and Flask, and ASGI for asynchronous applications like FastAPI. This chapter covers a complete case study of containerizing Python applications: building multi-stage Dockerfiles for Django and FastAPI, choosing the right dependency management strategy, running applications via Gunicorn or Uvicorn in production, and executing background tasks using Celery and Redis in separate containers.

30.1 Containerizing Django/Flask/FastAPI

Django and Flask are built on top of the WSGI (Web Server Gateway Interface) specification, a synchronous communication model between web servers and Python applications that has been the standard for a long time. Conversely, FastAPI and Starlette are built on ASGI (Asynchronous Server Gateway Interface), a specification that supports asynchronous requests and WebSockets. This difference determines what kind of server is used to run the application inside a container, so Dockerfiles for Django and FastAPI share a similar structure in the dependency section, but differ in the CMD section.

30.1.1 Project Structure and WSGI vs ASGI Differences

A Django project generated by django-admin startproject contains both wsgi.py and asgi.py files inside its configuration directory, but wsgi.py served via Gunicorn is commonly used in production, unless the application specifically requires asynchronous support such as WebSockets. Flask follows a similar pattern, needing only an exported app object from the main module. FastAPI differs because it does not provide a built-in WSGI implementation; its application can only be run using an ASGI server such as Uvicorn.

myproject/
├── config/
│   ├── settings.py
│   ├── urls.py
│   ├── wsgi.py
│   └── asgi.py
├── myapp/
│   ├── models.py
│   ├── views.py
│   └── tasks.py
├── manage.py
├── requirements.txt
├── .dockerignore
└── Dockerfile

The myapp directory contains the Django application code, while manage.py is still used to run administrative commands like database migrations inside the container. FastAPI's structure is much more concise, usually requiring only a single main.py file that exposes a FastAPI() object as its entry point.

30.1.2 Multi-Stage Dockerfile for Django

Just like in other application containerization case studies, Python dependency installation should be separated from the final image using a multi-stage build, ensuring that compilers and temporary installation files are not carried over into the production image.

FROM python:3.12-slim AS builder
WORKDIR /app

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim AS production
WORKDIR /app

RUN groupadd -r django && useradd -r -g django django

COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

COPY . .
RUN chown -R django:django /app

USER django
EXPOSE 8000
CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000"]

The virtual environment pattern at /opt/venv created during the builder stage is copied intact to the production stage via COPY --from=builder. This moves all installed packages over without needing reinstallation, while keeping the Python environment isolated from the base image's system packages. The PYTHONDONTWRITEBYTECODE variable prevents Python from writing .pyc files into image layers, while PYTHONUNBUFFERED ensures that print outputs and logs are immediately forwarded to stdout without buffering, which is crucial so that docker logs displays application logs in real time.

The USER django instruction executes the application process as a non-root user, a fundamental security practice that minimizes the impact if an application vulnerability is exploited by an attacker to gain access inside the container. The python:3.12-slim base image is selected because its size is significantly smaller than standard python:3.12, while still including common system libraries often required by Python packages with native extensions. This contrasts with alpine variants, which sometimes require recompiling these dependencies due to C library differences.

30.1.3 Dockerfile for FastAPI (ASGI)

The FastAPI Dockerfile follows the same multi-stage pattern, differing only in the CMD section because FastAPI is run via an ASGI server rather than WSGI.

FROM python:3.12-slim AS builder
WORKDIR /app

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim AS production
WORKDIR /app

RUN groupadd -r fastapi && useradd -r -g fastapi fastapi

COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

COPY . .
RUN chown -R fastapi:fastapi /app

USER fastapi
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The --host 0.0.0.0 option must be included because Uvicorn by default listens only on connections from 127.0.0.1, which inside a container means it can only be accessed from within the container itself. Without this option, ports mapped via docker run -p or compose.yaml will still be inaccessible from outside the container, even if the Uvicorn process appears to be running normally in the logs.

30.1.4 Static Files and collectstatic in Django

Django includes a static files mechanism that gathers all CSS, JavaScript, and image assets from various apps into a single directory using the collectstatic command. This command requires access to DJANGO_SETTINGS_MODULE and other configuration variables that are typically available only when the container actually runs, not while the image is being built. Thus, running it directly inside the Dockerfile via a RUN instruction risks failure or using incorrect configuration values for the target environment.

A safer approach is to execute collectstatic through an entrypoint script that runs every time the container starts, before Gunicorn actually launches the application.

#!/bin/sh
set -e

python manage.py collectstatic --noinput
python manage.py migrate --noinput

exec gunicorn config.wsgi:application --bind 0.0.0.0:8000

Save this script as entrypoint.sh, copy it into the image, and set it as the Dockerfile ENTRYPOINT.

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

The exec command at the end of the script is essential so that the Gunicorn process replaces the shell process as Process ID 1 inside the container, rather than running as a child process of the sh script. Without exec, the SIGTERM signal sent by Docker when stopping a container will not be forwarded properly to Gunicorn, causing the container to take longer to stop as Docker is forced to wait for the grace period to expire before sending a forced SIGKILL.

Running migrate automatically on every container start is convenient for development, but in production environments with multiple application replicas running concurrently, this practice risks creating race conditions if multiple containers attempt to migrate the database schema simultaneously. For production, consider running migrate as a separate step in the deployment pipeline before new replicas start accepting traffic, rather than automatically on every start.

30.2 Dependency Management in Images

The Python ecosystem offers several dependency management tools, each with different methods for efficient usage inside Docker images. Choosing the right strategy directly impacts image size and build speed, particularly regarding how Docker layer caching is leveraged so that dependency installations do not repeat with every minor code change.

30.2.1 pip and requirements.txt

pip with a requirements.txt file remains the simplest and most common approach. The key to build efficiency lies in the order of COPY instructions: copy requirements.txt first before copying the entire application codebase, so that the dependency installation layer is invalidated only when that file changes, rather than on every code edit.

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

The --no-cache-dir option prevents pip from saving downloaded package caches inside the image layer, as these caches are useless once installation completes and only add bloat to the image size. To strictly lock transitive dependency versions and prevent inconsistent builds over time, use pip-compile from the pip-tools package to generate a fully pinned requirements.txt from a requirements.in file containing only primary dependencies.

30.2.2 Virtual Environments Inside Containers

In non-containerized environments, virtualenv or Python's built-in venv module is used to isolate dependencies between projects on the same machine. Inside a container, such isolation is inherently provided because each container has its own filesystem; thus, venv is no longer mandatory strictly for cross-project isolation.

Nevertheless, venv offers practical benefits in the context of multi-stage builds: all installed packages are cleanly consolidated in a single directory, allowing them to be copied intact from the builder stage to the production stage via a single COPY --from instruction, as demonstrated in the previous Django and FastAPI Dockerfiles. Without venv, packages installed via standard pip install are scattered across default Python site-packages locations in the system, making selective cross-stage copying more complex.

30.2.3 Poetry in Multi-Stage Builds

Poetry manages dependencies using the pyproject.toml file and locks exact versions in poetry.lock, similar to the package-lock.json concept in the Node.js ecosystem. The following Dockerfile uses Poetry to generate a virtual environment inside the project folder, then copies that directory into the final image.

FROM python:3.12-slim AS builder
WORKDIR /app

RUN pip install --no-cache-dir poetry==1.8.3
RUN poetry config virtualenvs.in-project true

COPY pyproject.toml poetry.lock ./
RUN poetry install --no-root --without dev

FROM python:3.12-slim AS production
WORKDIR /app

RUN groupadd -r django && useradd -r -g django django

COPY --from=builder /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

COPY . .
RUN chown -R django:django /app

USER django
EXPOSE 8000
CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000"]

Setting virtualenvs.in-project true configures Poetry to place its virtual environment in a .venv folder inside the project directory, rather than in a hidden location outside the project directory, making it straightforward to locate and copy to the next stage. The --no-root flag during poetry install tells Poetry to install only listed dependencies without installing the project package itself in editable mode, aligning with official Poetry documentation, as application source code is copied via COPY . . after this dependency installation step. The --without dev flag excludes the dev dependency group such as testing tools or linters needed only during development, not during production execution.

Verify installed dependencies against poetry.lock using the following command inside the container.

docker compose exec app /app/.venv/bin/pip list

30.3 WSGI/ASGI Server Setup

Built-in development servers in Django (runserver) or Flask (flask run) are deliberately designed only for development needs, not for handling high concurrency in production. Gunicorn and Uvicorn are two commonly used production servers, selected according to whether the application uses a WSGI or ASGI model.

30.3.1 Gunicorn for WSGI Applications

Gunicorn (Green Unicorn) is a WSGI server utilizing a pre-fork worker model, where a single master process manages multiple worker processes that handle requests independently. Execute Gunicorn by explicitly defining the worker count using the --workers flag.

gunicorn config.wsgi:application \
    --bind 0.0.0.0:8000 \
    --workers 4 \
    --timeout 30 \
    --access-logfile - \
    --error-logfile -

The --access-logfile - and --error-logfile - flags redirect Gunicorn access and error logs to stdout/stderr instead of files, allowing logs to be captured by docker logs and standard container log aggregation systems. The --timeout 30 parameter sets the time limit before Gunicorn considers a worker hung and forcibly restarts it, a value that should be increased if the application contains heavy processing endpoints taking longer than 30 seconds.

In practice, finding the right worker count is a frequent troubleshooting topic for Sysadmins and DevOps Engineers. Official Gunicorn documentation recommends the formula (2 x CPU cores) + 1 as a starting baseline, rather than an arbitrarily large number, because each Gunicorn worker runs as a separate process consuming its own memory memory footprint. Setting too many workers on a container with small resource limits risks running the container out of memory, causing Docker to terminate it forcibly with an OOMKilled status.

30.3.2 Uvicorn and Gunicorn with ASGI Workers

Uvicorn can run standalone to serve ASGI applications like FastAPI, and recent versions also support the --workers flag to manage multiple worker processes from a single command.

uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4

For more complex production requirements, such as requiring zero-downtime graceful restarts during deployments or more robust process management, official Uvicorn documentation recommends running Uvicorn under Gunicorn as a process manager, utilizing the auxiliary uvicorn-worker package which provides a dedicated ASGI worker class for Gunicorn.

pip install uvicorn-worker

gunicorn main:app \
    --bind 0.0.0.0:8000 \
    --workers 4 \
    --worker-class uvicorn.workers.UvicornWorker

This approach combines Gunicorn's process management strengths (such as graceful reloads via SIGHUP signals) with Uvicorn's asynchronous event loop performance for handling ASGI requests. Official Uvicorn documentation marks the worker class implementation previously bundled directly inside its main package as deprecated, instructing users to install the separate uvicorn-worker package for maintained compatibility, even though the --worker-class flag still references the same uvicorn.workers.UvicornWorker path. If Gunicorn fails to recognize this option with a ModuleNotFoundError, verify that the uvicorn-worker package is added to requirements.txt or pyproject.toml, not just uvicorn.

30.3.3 Health Checks and Server Verification

Add a HEALTHCHECK in the Dockerfile or a healthcheck definition in compose.yaml so Docker can detect if a Gunicorn or Uvicorn process is running but unresponsive to incoming requests.

services:
  app:
    build: .
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

The healthcheck command above utilizes Python's built-in urllib module because the python:3.12-slim image does not include curl by default, avoiding extra package installations just for health checks. The /healthz endpoint in this example should be defined within the application code, typically as a simple view or route returning a 200 OK status without checking external dependencies like databases, keeping health checks fast and preventing cascading failures if the database responds slowly.

If a container repeatedly shows an unhealthy status, verify whether the application is listening on 0.0.0.0 rather than 127.0.0.1, then inspect Gunicorn or Uvicorn logs to see if workers are crashing repeatedly during startup.

docker compose logs app
docker compose ps

30.4 Background Tasks with Celery and Redis

Python applications performing heavy operations like sending emails, image processing, or calling slow external APIs typically offload these operations to background tasks to avoid blocking user HTTP responses. Celery is the most popular task queue in the Python ecosystem for this purpose, requiring a message broker like Redis to queue tasks between the main application and worker execution processes.

30.4.1 Redis Service and Celery Workers in Compose

Add the redis and worker services to the previously defined compose.yaml. The Celery worker uses the same image as the main application, differing only in the executed command.

services:
  app:
    build: .
    ports:
      - "8000:8000"
    environment:
      CELERY_BROKER_URL: redis://redis:6379/0
      CELERY_RESULT_BACKEND: redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy

  worker:
    build: .
    command: celery -A config worker --loglevel=info --concurrency=4
    environment:
      CELERY_BROKER_URL: redis://redis:6379/0
      CELERY_RESULT_BACKEND: redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

volumes:
  redis-data:

The -A config flag in the celery worker command points to the Celery application instance module, typically defined in a config/celery.py file in a Django project according to standard Celery integration conventions detailed in its official documentation. The --concurrency=4 option sets the number of child processes used by the worker to execute tasks in parallel, a value that should be adjusted based on task characteristics, depending on whether tasks are CPU-bound or I/O-bound like external API calls.

Redis is chosen as the broker in this example due to its simple setup and sufficiency for typical task queues, though Celery also supports brokers like RabbitMQ for stricter message delivery guarantees, as outlined in official Celery documentation.

30.4.2 Celery Beat for Scheduled Tasks

Celery beat is a separate scheduler responsible for dispatching tasks to the queue at periodic intervals, such as clearing stale data every midnight. Similar to the scheduler pattern in the Laravel case study, beat should run as a standalone container, isolated from the workers executing the tasks.

services:
  beat:
    build: .
    command: celery -A config beat --loglevel=info
    environment:
      CELERY_BROKER_URL: redis://redis:6379/0
      CELERY_RESULT_BACKEND: redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy
    restart: unless-stopped

Official Celery documentation stresses that a beat process must run as a single instance only, as running multiple instances causes periodic tasks to be dispatched multiple times to the queue according to the number of active beat instances. Sysadmins and DevOps Engineers frequently encounter this trap when scaling services via docker compose up --scale without ensuring that the beat service is excluded from scaling operations, unlike worker services which can safely run across multiple replicas.

An alternative is embedding beat into a worker process via the -B flag, but official Celery documentation explicitly states this pattern is suitable only for single-worker setups and is not recommended for production. Thus, a separate container remains the safer choice for production environments.

30.4.3 Monitoring and Troubleshooting Tasks

Verify that Celery workers are active and reachable using the inspect ping command, which sends a signal to all workers connected to the same broker.

docker compose exec worker celery -A config inspect ping

A pong response from a worker confirms that communication between the worker and the Redis broker is functioning properly. If this command times out without a response, first check whether CELERY_BROKER_URL in the worker container correctly points to the redis service name rather than localhost, identical to the database connection trap seen in the Laravel case study.

To inspect active or reserved tasks, use the inspect active and inspect reserved commands.

docker compose exec worker celery -A config inspect active
docker compose exec worker celery -A config inspect reserved

Tasks that fail repeatedly typically leave tracebacks in the worker container logs, making direct log inspection the most practical first step in troubleshooting.

docker compose logs -f worker

Regarding data trustworthiness: if CELERY_RESULT_BACKEND relies on the same Redis instance without adequate persistence configuration, stored task results may be lost if the Redis container restarts without proper volume mounts. The redis-data volume in the compose.yaml example above ensures Redis data persists across container recreations, though it is not a substitute for a comprehensive backup strategy for critical data.