Docker Compose

Overview of Docker ComposeDocker Documentation

Docker Compose

How Compose Works

It helps you configuring and starting multi-container applications.

Compose simplifies the control of the entire application stack, facilitating the management of services, networks and volumes in a single YAML configuration file (usually compose.yaml). And with simple Compose CLI commands you can bring UP or DOWN an entire container network.

docker-compose.yaml and docker-compose.yml are also reconized ONLY for backward compatibility.

Use compose.yaml.

compose.yaml doesn't replace the Dockerfile.

Not all of the configurations in Dockerfile can be done in the compose.yaml.

Compose How-tos

Using multiple Compose .yaml files

You may also use fragments and extensions to keep your Compose files easy to maintain.

These muliple Compose files can be merged together to define the application model.

Merging them with -f

docker compose -f angular-compose.yaml -f node-compose.yaml.

It merges files in the order they are specified on the command line. Subsequent files may merge, override, or add to their predecessors.

• Meaning that if a service with the same name is in two different merging files, their configurations might be overriden or added.

  • For single-value options the value is replaced.

  • For multi-value options the values are added.

  • For <key> <value> options, the values might get added or overwritten.

  • For volumes and devices, values are merged using the mount path in the container.

Some rules for merging files:

• Make sure all paths in the files are relative to the base Compose file (the first file specified with -f on the command line)

  • Use docker compose config to review your merged configuration.

Compose automatically searches on the same folder and parent folder for compose.yaml and compose.override.yaml and merges them by default.

You can also supply configurations from the STDIN with - after -f:

# Reading properties from the STDIN
docker compose -f - <<EOF
webapp:
 image: ubuntu
 # ...
EOF

include other Compose files

To easy modularize complex applications into sub-Compose files. Each path listed in the include section loads as an individual Compose application model, with it's own project directory, in order to resolve relative paths.

include:
  - other-compose-file.yaml
services:
  serviceA:
    build: .
    dependes_on:
      # Just call it as it was in the same file
      - serviceB

Override services

Compose reports an error if any resource from include conflicts with resources from the included Compose file.

So, to tweak configurations from the included file you add an override file to the include directive, for overwriting specific files.

include:
  - path:
    - other-compose-file.yaml
    - override.yaml

Or, if you need to override multiple files and don't want to create multiple override files, you may create a single global compose.override.yaml file.

compose.yaml

include:
  - team-1/compose.yaml
  - team-2/compose.yaml

compose.override.yaml

services:
  service-1:
    ports:
      - 40:40
  service-2:
    volumes:
      - ./src:./src

extends in Compose files

Useful for reusing service options, from other files or from the same file.

services:
  web:
    extends:
      file: common-services.yml
      service: webapp
    # OR
    # IF `webapp` is inside this file
    extends: webabb

volumes_from and depends_on are never shared, to avoid implicit dependencies.

Relative paths are automatically adjusted.

Compose Lifecycle Hooks

When Docker Compose runs a container, it uses ENTRYPOINT and CMD to manage what happens when the containers starts and stops.

In Compose you may handle these separatly with lifecycle hooks - (commands that run right after the container starts or just before it stops).

Lifecycle hooks also have special privileges (like running as the root user), even when the container itself runs with lower privileges for security.

Post-start hooks

Commands that runs right after a container starts.

services:
  APP:
    post_start:
      - command: COMMAND
        user: root

Pre-stop hooks

Commands that runs before the container stops. (docker compose down or manually with Ctrl+C)

These hooks won't work if the container stops by itself or gets killed suddently.

services:
  APP:
    pre_stop:
      - command: COMMAND

Service Profiles

profiles help you adjust your Compose application for different environments or use cases, by selectively activating services.

Services without profiles are always enabled.

Core services of your application shouln't be assigned profiles so they are always enabled and automatically started.

# Running a `docker compose up` here will only start `backend` and `db` services
services:
  frontend:
    image: frontend
    
  phpmyadmin:
    image: phpmyadmin
    depends_on: [db]
    profiles: [debug]
  
  backend:
    image: backend
    profiles: [backend]
    depends_on:
      - db
    
  db:
    image: mysql
    profiles: [backend]

Starting profiles

# Only [frontend] will start
docker compose up

# [frontend, backend, db] will start
docker compose up --profile backend

# Will fail because `phpmyadmin` depends on `db`, but because they are from different profiles, `db` does not get started.
# Possible correction would be to add `debug` profile to `db` OR remove all profiles from `db`.
docker compose up --profile phpmyadmin

# Will call for start only `backend`, but since it depends on `db` it will also start it.
docker compose run backend

# Agina it will call for start only `phpmyadmin`, but this time it will fail, since it depends on `db` but db is in a different profile.
# Possile correction would be to add `debug` profile to `db` OR remove all profiles from `db`.
docker compose run phpmyadmin

Targeted services from depends_on should either:

• Share a common profile.

• Or not have profiles set.

Starting specific profiles (--profile)

# Start a single profile
docker compose --profile debug up

# Start multiple profiles
docker compose --profile debug --profile backend up

# Start with ALL 
docker compose --profile "*" up

Stopping specific profiles

It follows the same logic as starting profiles.

docker compose --profile debug down

Controlling startup order

Compose always starts and stops containers in dependency order, where dependencies are determined by:

• depends_on

• links

• volumes_from

BUT, Compose does not wait until a container is "ready", only until it's running.

This may cause issues if services running inside the container triggers other services that might take some time to start but are critical operationaly speaking.

Detecting "ready" states

Use the condition attribute with one of the following:

• service_started

• service_healthy: A dependency is expected to be "healthy", which is defined by healthcheck.

• service_completed_successfully: A dependency is expected to run to successful completion before starting a dependent service.

services:
  web:
    build: .
    depends_on:
      db:
        condition: service_healthy
        restart: true
      redis:
        condition: service_started
  redis:
    image: redis
  db:
    image: postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 10s
      retries: 5
      start_period: 30s
      timeout: 10s

In this case:

1. db and redis are created first.

2. Then Compose waits for healthchecks to pass on db.

   • Healthcheck is done with pg_isready, retried every 10 seconds up to 5 times.

3. Then it creates web.

Environment Variables

Don't use env variables to pass sensitive information, use secrets instead.

Compose relies on you to resolve the values. Un-resolved values will lead to the variable being unset and removed.

No warning is raised. (Unless interpolation is used)

services:
  app:
    environment:
      # Map syntax:
      # Any boolean values (true, false, yes, no) should be enclosed by quotes to ensure
      # they won't be converted by YAML parser.
      TYPE: development
      DEBUG: "true"
      # Variables without values are expected to be given a value at run-time.
      USER_INPUT:

      # OR
      
      # Array syntax:
      - TYPE=development
      - DEBUG=true
      - USER_INPUT
      
    # OR
    
    # To declare one or more .env files. (If multiple files, they are evaluated in
    # order, and one can overwrite values from the previous)
    # The path is relative to the `compose.yaml` file.
    env_file: "app.env"

When both env_file and environment are set for a service, values set by environment have precedence.

Check the precedence order here.

You can also temporarily set them on.

docker compose run --env USER_INPUT="Some Value" --env DEBUG="false"

Secrets

Are any piece of data, such as password, certificates, or API keys that shouln't be transmitted over a network or stored unencrypted in a Dockerfile or the applications's source code.

Compose grants access to secrets on a per-service basis.

Env variables are often available to all processes, and can be difficult to track access.

They can also be printed in logs.

# Providing secrets inside files
services:
  app:
    image: myapp:latest
    secrets:
      - some_secret
      
secrets:
  some_secret:
    file: ./secret_pass.txt

# Feeding secret from build-time ENV variables
services:
  app:
    image: myapp:latest
    secrets:
      - some_token
      
secrets:
  some_token:
    environment: TOKEN

Network

By default Compose sets up a single network for your application.

• The network's name is given based on the project's name, which is based on the name of the directory it lives in OR the project's name given by --project-name.

Each container for a service joins the default network and is both reachable by other container on that network, and discoverable by the service's name.

• This means instead of using the container's name, like it was done with docker run, with docker compose up you will use the service's name.

When communicating between services (containers), note that you must use the CONTAINER_PORT and not the HOST_PORT.

Avoid referencing containers by IP, since their IPs will change over time when re-created.

Configuring the Default network

services:
  # ...
  
networks:
  default:
    name: custom_name
    driver: host

Aliases with Links

You can alias the service's name for extra ways to connecting to the container.

services:
  app:
    links:
      - "db:database"
  db:
    image: postgres

This way, you can access the db service from app by the either [db, database].

Creating Custom Networks

To create more complex topologies and specify different network drives and options. Also to connect services to externally-created networks which aren't managed by Compose.

Each service may specify one or more networks to connect to.

services:
  angular:
    build: ./angular.dockerfile
    network:
      - frontend
  node:
    build: ./node.dockerfile
    networks:
      - frontend
      - backend
  db:
    build: postgres
    networks:
      - backend
      
networks:
  frontend:
    name: angular_network
    driver: bridge
  # Just specifing the name is enough to create it
  backend: {}

Joining External Networks

services:
  # ...
  
networks:
  network1:
    # Compose will look for a network with this name
    name: the-name-of-the-existing-network
    external: true

Compose CLI

The docker compose has a couple of flags that can be used.

Flag	Description
--dry-run	Execute commands in dry run mode.
--env-file	Specify an alternate environment file.
-f or --file	Compose configuration files.
--progress	Set thype of progress output. (auto, tty, plain, json, quiet)
--project-directory	Specify an alternate working directory. (default: the path of the, first specified, Compose file)
-p or --project-name	Project name. (/^[a-z0-9]+[a-z0-9\-\_]*/)

down

docker compose down [OPTIONS] [SERVICES]

Will stop and remove the following, that were created by up :

• Containers;

• Networks:

  • The ones defined in network section of compose.yaml.

  • Networks defined as external are never removed.

• Volumes:

  • Anonymous volumes are not removed by default. (But are also not mounted for subsequent up)

  • Volumes defined as external are never removed.

• Images if flag --rmi used.

Flag	Description
--remove-orphans	Remove containers for services not defined in the Compose file.
--rmi	Removes Images used by services. ("local" remove only images that don't have a custom tag ("local" | "all"))
-t or --timout	Specify a shutdown timout in seconds.
-v or --volumes	Removes named volumes declared in the volumes section of the Compose file. Removes anonymous volumes attached to containers.

logs

docker compose logs [OPTIONS] [SERVICES]

Display log outputs from services.

ps

docker compose ps [OPTIONS] [SERVICE]

List containers for a Compose project, with current status and exposed ports.

By default, only running containers are shown.

Flag	Description
-a or --all	Show all containers. (Including the ones created without Compose)
--format	Format output with custom template. (table, json, TEMPLATE)
--orphans	Include services not declared by project.

run

docker compose run [OPTIONS] SERVICE [COMMAND] [ARGS...]

Runs a one-time command against a service.

up

docker compose up [OPTIONS] [SERVICES]

Builds, (re)creates, starts and attaches to containers for a service.

If there are existing containers for a service and it's Images or service's configuration were changed, up will detect the changes and recreate the containers (preserving mounted volumes).

To avoid recreation of containers use --no-recreate flag.

To force recreation of all containers use --force-recreate flag.

An exit code 1 is generated if the process encounter an error.

If the process was interrupted using SIGINT (Ctrl + C) or SIGTERM, the containters are stopped, and the exit code is 0.

Flag	Description
--abort-on-container-exit	Stop all containers if any container was stopped. (Incompatible with -d)
--abort-on-container-failure	Stop all container if any container exited with failure. (Incompatible with -d)
--build	Build Images before starting containers.
-d or --detach	Run containers in the background.
--force-recreate	Recreate containers even if their configuration and Image haven't changed.
--no-attach	Do not attach (stream logs) to specified services.
--no-deps	Do not recreate services linked services. Ex.: If you change a service config, and don't need to rebuild the dependent services.
--no-recreate	Don't recreate containers even if their configuration and Image have changed.
--no-start	Don't start the services after creating them.
--quiet-pull	Pull images without printing progress information.
-V or --renew-anon-volumes	Recreate anonymous volumes instead of retrieving data from the previous containers.
--timestamps	Show timestamps.
--wait	Wait for services to be running|healthy. (Implies detached mode)
--wait-timeout	Maximum duration to wait for project to be running|healthy.

Running

• You can run the docker-compose with docker-compose up.

  • Use -d to run in detach mode.

• You can choose the services/container to go up with:

  • docker-compose up [list-of-container], like docker-compose up -d node mysql.

Stopping

• You can stop the containers with docker-compose down.

• Docker compose will stop and remove the containers, networks and others, BUT NOT the volumes that were created.

  • Use -v flag to also delete the volumes.

Choosing a dockerfile from multiple ones

services:
  composer:
    build:
      # The name of the file
      dockerfile: ./dockerfiles/composer.dockerfile

Dependency of Containers

• You can specify that a container should only go up if another one went up with depends_on config.

• This means that if you start with docker-compoase up -d frontend, it will automatically up the backend container.

services:
  backend:
    ...
  frontend:
    ...
    depends_on:
      - backend

Force docker-compose to reevalute the dockerfiles and rebuild if necessary

docker-compose up --build

Front end Containers

• Just like it was needed to use the flag -it in frontend containers, you can config them with:

services:
  frontend:
    ...
    stdin_open: true
    tty: true

YAML Tips & Tricks

Some YAML tips when creating the Compose files.

Fragments (Anchors and Aliases)

When using anchors, use Map Syntax when providing <key> and <values>. Array Syntax only allows <key>.

Are built-in YAML features, to help create re-usable blocks. (They kind of work like variables, not really..)

services:
  first:
    image: my-image:latest
    environment: &env
      # Array syntax can ONLY be used if no values are passed
      - CONFIG_KEY
      - EXAMPLE_KEY
      - DEMO_VAR
  second:
    image: another-image:latest
    environment: *env

From this example, you configure and name the properties from environment as &env (where & symbol states the origin of data), and then reuse them later with *env.

Adding or Overriding values

services:
  first:
    image: my-image:latest
    environment: &env
      # Use Map syntax when providing values along
      CONFIG_KEY: 1h12j4
      EXAMPLE_KEY:
      DEMO_VAR: "false"
  second:
    environment:
      <<: *env
      ADDITIONAL_VAR: value

Pulling more than one anchor

# Using Extensions
x-env1: &env-1
  VAR1: value1
  VAR2: value2
x-env2: &env-2
  VAR3: value3

services:
  app:
    environment: [*env-1, *env-2]

Extensions

Use the prefix x- as a top-level element to modularize configurations to reuse.

Compose ignores any fields that starts with x-.

This means that you can use it to declare properties on the top of the file (like variables), and then use them inside valid properties (services, networks, ...) with anchors.

x-network-config: &net-conf
  bnetwork:
    driver: host
    name: the-b-network
      
services:
  app:
    networks: *net-conf

Interpolation

Values in Compose file can be set by variables and interpolated at runtime.

Both $VARIABLE and ${VARIABLE} are supported.

Interpolation is mostly for <values>, the only properties that allows you to interpolate on <keys> are labels and environment.

Compose Properties

Compose file referenceDocker Documentation

Some of the top-level properties that can be defined in compose.yaml.

name

To define the project's name. Can also be set by COMPOSE_PROJECT_NAME default env variable or by -p option in the command line.

name: project-name

service (required)

A service is an abstract definition of a computing resource, and is backed by a container.

Specify byte values like {amount}{byte unit}, where {byte unit} may be:

b (bytes), k or kb (kilo bytes), m or mb (mega bytes) and g or gb (giga bytes)

Ex.: 300m or 2gb.

Specify durations like {value}{unit}, {unit} may be:

us (microseconds), ms (milliseconds), s (seconds), m (minutes) and h (hours)

Ex.: 40s, 1m30s, 1h30m50s20ms.

Some attributes a service may declare:

Property	Type	Description
build	value	Defines how to create the Docker image for the service. (Usually the path to the Dockerfile)
blkio_config	map	Defines a set of configurations to set IO block limits.
cap_add	array	Additional container capabilities.
command	value	Overrides the default command declared by the container image. (Ex.: By Dockerfile's CMD) (If value is '' or [], default command is overwriten to be empty)
container_name	value	For custom container's name. (If defined, may stop Compose from scaling beyond one container)
cpu_count	value	Defines the number of usable CPUs.
cpu_shares	value	As integer value, defines relative CPU weight versus other containers.
depends_on	array or map	To define dependency to other services. Will also control the order of startup and shutdown.
deploy	map	Set of deployment specifications for managing the behavior of containers across different environments.
domainname	value	Custom domain name for the container.
entrypoint	value or array	Overrides the default entrypoint declared by the container image. (Ex.: By Dockerfile's ENTRYPOINT) (If value is '' or [], default command is overwriten to be empty)
env_file	value, array or map	Specify one or more files that contain env variables.
environment	array or map	To list env variables.
extends	map	Share common configurations among services.
extra_hosts	array or map	Adds hostname mappings to container's network interface configuration. Ex.: (/etc/hosts)
healthcheck	map	Declares a check to determine if the service containers are "healthy".
hostname	value	Custom hostname for the container.
image	value	Another way to define which image to use. (Like if you don't have a Dockerfile)
links	array	Network link aliases to services in another container. (Ex.: SERVICE:ALIAS)
mem_limit	value	Limit the amount of memory a container can allocate. Ex.: (300m, 1gb) (Must be consistent with limits.memory attribute in deploy attribute)
mem_reservation	value	Reserves an amount of memory a container can allocate. Ex.: (300m, 1gb) (Must be consistent with reservations.memory attribute in deploy attribute)
network_mode	value	Sets the container network mode. (none, host, service:{name}, container:{name})
networks	array or map	Defines the networks the service is attached to.
ports	value or array	Define port mapping between host and container (host:container).
privileged	value	Configures the service to run with elevated priviledges.
profiles	array or map	Defines the list of named profiles for the service to be under.
restart	value	Defines the restart policy on container termination.
secrets	array	List of variables for sensitive data. Reference top-evel secrets or create local ones.
tty	value	Configure the service to run with tty. (Same as running the container with -t flag.
user	value	Overrides the user to run the container process. (This overwrites the user defined in Dockerfile)
volumes	array	Define mount host paths or named volumes assessible by the containers. (For the volume to be reused by multiple services, it must also be declared as top-level)
working_dir	value	Overrides the container's working directory. (This overrides WORKDIR from Dockerfile)

networks

Check more info over Network up here.

The top-level networks lets you configure named networks that can be reused across multiple services.

Some attributes a service may declare:

Property	Type	Description
driver	value	Specifies the network driver to use.
external	value	Specifies if the network is maintained outside the application.
ipam	map	To configure more in-depth options for the network.
name	value	Custom name for the network.

volumes

The top-level volumes lets you configure named volumes that can be reused across multiple services.

Some attributes a service may declare:

Property	Type	Description
driver	value	Specifies the volume driver to use.
external	value	Specifies if the volume is maintained outside the application.
name	value	Custom name for the volume.

configs

Lets services to adapt their behavior withtout the need to rebuild the Docker Image.

Check more info on the docs.

secrets

Check more info over Secrets up here.

The top-level secrets lets you define or reference sensitive data that can be accessed across multiple services.

A secret's source can be either a file or environment.