Executors#

Relevant APIs#

Overview#

Executors are responsible for executing steps within a job run. Once a run has launched and the process for the run (the run worker) has been allocated and started, the executor assumes responsibility for execution. Executors can range from single-process serial executors all the way to managing per-step computational resources with a sophisticated control plane.

Every job has an executor. The default executor is the multiprocess_executor, which executes each step in its own process. You can choose the executor for a job by supplying an ExecutorDefinition to the executor_def parameter of @job or GraphDefinition.to_job.

Executing a job via JobDefinition.execute_in_process, overrides the job's executor and uses in_process_executor instead.

Example executors include:

• in_process_executor: Execution plan executes serially within the [run worker](/deployment/overview#job-execution-flow) itself.
• multiprocess_executor: Each step executes within its own spawned process. Has configurable level of parallelism.
• dask_executor: Executes each step within a dask task.
• celery_executor: Executes each step within a celery task.
• docker_executor: Executes each step within a Docker container.
• k8s_job_executor: Executes each step within an ephemeral kubernetes pod.
• celery_k8s_job_executor: Executes each step within a ephemeral kubernetes pod, using celery as a control plane for prioritization and queuing.
• celery_docker_executor: Executes each step within a Docker container, using celery as a control plane for prioritization and queueing.

Custom Executors#

The executor system is pluggable, and it is possible to write your own executor to target a different execution substrate. This is not well-documented, and the internal APIs continue to be in flux.