Trivial Mistakes I Made With Docker Compose This Week

Introduction

Does it seem to anyone else that trivial problems tend to slow your workflow down the most? This week, I spent a disproportionate amount of time tracing down simple, avoidable mistakes that I was completely blind to in the moment. Hours were wasted debugging issues that were agonizingly simple once discovered. I share the following so you may be amused by my suffering. Context, the project is a Dockerized Django DRF API deployed to a k8 cluster on a VPC.

Databases & Compose

For development, I'm running two services in my compose.yml file: app & db. The db configuration is a simple setup declaring the postgres image, mounting a volume for data persistence, a health check, and an environment variable import for a .env file. To make things simpler, I use the same .env file to instantiate the variables in my app service. That way, I don't have to worry about credentials ever being wrong with connecting to my dev database. Wrong - bad credentials on docker compose up. How could this be if I'm pulling the variables from the exact same file?

My first though was that the environment variable mapping was incorrect. Off to the Docker Hub docs to see what the configuration should be. After review, everything is set correctly. The docs point to the following environment variable names which I have correctly set and mapped to the database configuration:

#the only required variable
POSTGRES_PASSWORD

POSTGRES_USER
POSTGRES_DB
etc...

Next, I assumed broadly there's a breakdown in the environment variables being brought in. I'm not using a third party package like django-environ so I left the possibility a mistake was made somewhere in configuration. To see if this was the case, I check if I have good data available in my settings.py file with the following:

# settings.py
...
# database configuration
# Print all environment variables
print("All Environment Variables:")
for key, value in os.environ.items():
    print(f"{key}: {value}")

With this, I see I have good data in my logs. The problem must be something else.

Then it hits me: I changed the database password in .env. Not for any specific reason, it's not like this is a production database password. Likely from boredom, I modified the POSTGRES_PASSWORD from something like change-me to my-dev-db-password. I wasn't thinking about how this would affect my development environment since Compose loads your environment variables at run time and they're not built into the container. They're loaded into both the database service and the app service at the same time so they should always be consistent, right? That's true, but while the database service is initialized each time you run docker compose up, the data itself is persistent and mounted to a volume. I wasn't able to connect to my database because I simply wasn't providing correct credentials.

Realizing this silly mistake, all I needed to do was flush the database volume (I didn't have any other data in this volume I would miss) and restart the service. This will initialize a new database with fresh data pulling the changed environment variables. You do this by adding the -v flag to docker compose down. That's it - brutal.

File Permissions, Compose & Gunicorn

I'm running Gunicorn inside the Docker container for app-level concurrency. Consider the following (abbreviated) Dockerfile:

FROM some-python-image-on-linux
...
COPY ./app /app

ARG DEPS="my-dependencies"
RUN apt-get update && apt-get install -y $DEPS --no-install-recommends && \
   # do some clean up
   ...
   # add a user & make entrypoint.sh executable
   adduser \
      --disabled-password \
      --no-create-home \
      my-limited-user && \
   chmod +x /app/entrypoint.sh

USER my-limited-user

CMD ["/app/entrypoint.sh"]

In the Dockerfile, I copy my app into the container which includes a bash file entrypoint.sh. Rather than include the script for Gunicorn directly in the CMD directive, I prefer to separate the script for a more concise, readable Dockerfile. The file, as its name suggests, establishes the entry point for the container, starts the Gunicorn service and binds/exposes a container port. Because this file needs to be executable, I explicitly add the permission in the final line of my RUN directive.

Then in my Docker Compose file, I set up my services for local testing. For the example, assume I'm running two services: app & db. Consider the following abbreviated compose.yml file:

services:
   app:
      build:
         context: .
      ports:
         - "8000:8000"
      volumes:
         - ./app:/app
      command: >
         sh -c "python manage.py migrate &&
         cd /app && ls -l && /app/entrypoint.sh"
      # environments & other stuff
      depends_on:
         - db
   db:
      image: postgres
      volumes:
         -dev-db-data:/var/lib/postgresql/data
      # environments & other stuff

volumes:
   dev-db-data:

If you already see the problem, have mercy. When running the container with docker compose up, I get a failure for bad permissions:

app-1  | sh: 1: /app/entrypoint.sh: Permission denied
app-1 exited with code 126

Permission denied? That doesn't make any sense because I set execute permissions in the Dockerfile. Adding the ls -l to compose.yml, I can clearly see I have bad permissions:

app-1  | -rw-r--r--  1 my-limited-user my-limited-user  136 Jan 24 17:44 entrypoint.sh

My focus is now on why my Dockerfile isn't establishing permissions as expected. I spend hours fumbling around different configurations and researching every directive in detail. No matter what I try, permissions remain unaffected.

Then I remember, I mounted the app directory so I wouldn't need to rebuild the container when making code changes. This is something I often do in a dev environment with Docker Compose to save time by avoiding the need to build a container every time I update code. What's the implication of this? I'm not fighting my container but rather local permissions. I'll I needed to do for docker compose up to run locally was modify the local file permission in terminal, chmod +x entrypoint.sh - brutal. 

Final Thoughts

One thing I'm reminded of as I reflect on my recent mistakes is to not jump to the more complicated suspicions before checking if your basic configurations are correct. It's easy to jump down a rabbit hole assuming you have a nuanced and complex issue to deal with the reality is much simpler: you made a simple error that you're blind to. No amount of Stackoverflow browsing is going to lead to the solution's discovery so you're best to step away for a while and come back fresh when the trivial mistakes are once again visible.

Cheers.