Orchestration on Docker¶

APIs¶

dagster_docker.DockerRunLauncher RunLauncher[source]¶

Config Schema:

image (dagster.StringSource, optional)

The docker image to be used if the repository does not specify one.

registry (strict dict, optional)

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource)

username (dagster.StringSource)

password (dagster.StringSource)

env_vars (List[String], optional)

The list of environment variables names to forward to the docker container

network (dagster.StringSource, optional)

Name of the network to which to connect the launched container at creation time

networks (List[dagster.StringSource], optional)

Names of the networks to which to connect the launched container at creation time

container_kwargs (permissive dict, optional)

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition[source]¶

Config Schema:

image (dagster.StringSource, optional)

The docker image to be used if the repository does not specify one.

registry (strict dict, optional)

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource)

username (dagster.StringSource)

password (dagster.StringSource)

env_vars (List[String], optional)

The list of environment variables names to forward to the docker container

network (dagster.StringSource, optional)

Name of the network to which to connect the launched container at creation time

networks (List[dagster.StringSource], optional)

Names of the networks to which to connect the launched container at creation time

container_kwargs (permissive dict, optional)

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

retries (selector, optional)

Default Value:

{
    "enabled": {}
}

Config Schema:

enabled (strict dict, optional)

Default Value:

{}

disabled (strict dict, optional)

Default Value:

{}

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster import job
from dagster_docker import docker_executor

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.