Achieving storage parity in Django with local object storage

Updated

In modern Django projects, uploaded files are often stored in S3-compatible object storage. Using different storage classes for local and production can complicate the codebase and cause subtle production-only issues. A practical solution is to use object storage locally too.

The open source community, through django-storages, has greatly simplified interactions with a wide range of storage backends. The main issue arises when more advanced or fine-grained file operations are required: at a lower level, the django-storages API and Django's native File Storage API are not specified in a uniform way.

And yes -- I'm looking at you: paths, signed URLs, ACLs, overwrites. Both the bane and the boon.

A local object storage is a great workaround for achieving behavioral parity among the environments and for writing reliable automated tests. Specifically, we're going to see how to set up a dockerized local MinIO instance.

Adding MinIO to your Compose file

First, let's add the MinIO service to our Compose file with its health check:

services:

  ...

  minio:
    command: minio server /var/lib/minio/data --console-address ":9001"
    environment:
      - MINIO_ENDPOINT=http://minio:9000
      - MINIO_BUCKET_NAME=my-bucket
      - MINIO_ROOT_USER=minio-admin
      - MINIO_ROOT_PASSWORD=minio-admin
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 3s
      timeout: 3s
      retries: 30
    image: minio/minio:latest
    ports:
      - 9000:9000
      - 9001:9001
    volumes:
      - minio_data:/var/lib/minio/data

If your Django application does not run inside Docker, then you will need to expose the MinIO service through a custom network to make it reachable from outside Docker.

Once the service is running, the MinIO web console will be available at http://localhost:9001, while the S3 API will be exposed on port 9000.

Installing and configuring our MinIO instance

It is also helpful to install the MinIO client to automate bucket creation and configure ACLs. If your Django project runs in a container, you can include something like the following in its Dockerfile:

RUN curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/bin/mc \
    && chmod +x /usr/bin/mc \
    && mc alias set minio http://minio:9000 minio-admin minio-admin \
    && mc mb --quiet --ignore-existing minio/my-bucket \
    && mc anonymous set download minio/my-bucket

With that, we have the MinIO server set up, create the bucket using the mb ("make bucket") command and configure its permissions to be public read-only.

Integrating MinIO with Django

Finally, even out the functional gap by making django-storages your primary storage backend. In your Django settings, specify the following parameters:

  • AWS_ACCESS_KEY_ID -> MINIO_ROOT_USER

  • AWS_SECRET_ACCESS_KEY -> MINIO_ROOT_PASSWORD

  • AWS_STORAGE_BUCKET_NAME -> MINIO_BUCKET_NAME

  • AWS_S3_ENDPOINT_URL -> MINIO_ENDPOINT

  • AWS_S3_REGION_NAME -> your preferred region

  • AWS_LOCATION -> optional subfolder within the bucket

This configuration ensures that your local MinIO setup behaves consistently with your production storage, enabling reliable development and testing.

0