Rotating Docker Logs — Keeping your overlay folder small

TL;DR

Learn from my mistakes. If you are using the default json-file driver, configure max-file and max-size options to have Docker automatically rotate and remove old logs.

The Problem

4:30 pm. Time to go home, figure something out for dinner, relax and — — THE BUILDS ARE FAILING. Classic.

Can’t push or pull to our self-hosted registry. The container is running, so why would it not be able to read/write to the disk unless…the disk is full?

After some quick digging, turns out the Docker overlay folder is big. Like almost 200 GB big. But a quick glance shows that we only have 4 images and a few small volumes, so why would that overlay folder be so big?

The answer: logs. a LOT of logs.

The Fix

By default, Docker uses the json-file log driver and saves those logs in the overlay folder. Docker doesn’t limit the size of the files or how many log files there can be for a single container. If you continue to run that container, the log file will grow as long as it has drive space to do so. Thankfully, it’s pretty easy to set these limits.

If you are running from the CLI, your option looks like:

docker run --log-opt max-size=10m --log-opt max-file=5 my-app:latest

This sets a maximum of five log files with a max size of 10 Mb each. So at most 50 Mb of logs for that container. Tune those numbers as you see fit.

If you are using docker-compose, your options should look like:

my-app:
    image: my-app:latest
    logging:
        driver: "json-file"
        options:
            max-file: 5
            max-size: 10m

You can also configure this as a default in your daemon settings https://docs.docker.com/config/containers/logging/configure/#configure-the-logging-driver-for-a-container

Written by James Quigley, an SRE/DevOps Engineer, and general tech nerd. Views and opinions are my own. Check out my YouTube Channel or follow me on Twitter!