Processes like onboarding a new piece of content, transcoding and packaging a video asset, or running a complex studio production workflow don't happen in one service call — they're multi-step sequences that span many microservices, take anywhere from seconds to days to complete, and need to survive individual step failures without losing track of where they are. Before Conductor, that kind of orchestration logic tended to live inside whichever service happened to kick off the process, implemented as ad hoc state machines, retry loops, and status flags scattered across the codebase. Netflix's content and studio engineering teams, dealing with exactly this kind of long-running, many-step process, built Conductor to pull that logic out into a dedicated, visible orchestration layer.
Workflows as data, not buried code
Conductor's central idea is defining workflows declaratively — as JSON documents describing a sequence (or a directed graph) of tasks, their dependencies, retry policies, and timeout behavior — rather than as imperative code embedded inside a triggering service. A workflow definition says explicitly: run task A, then run B and C in parallel, then run D once both finish, retry D up to three times on failure, and time out the whole workflow after a set duration. That declarative structure means the orchestration logic is inspectable and versionable on its own, separate from the business logic of the individual tasks it coordinates.
{
"name": "encode_and_publish",
"tasks": [
{ "name": "validate_source", "type": "SIMPLE" },
{ "name": "transcode", "type": "SIMPLE" },
{ "name": "publish_to_cdn", "type": "SIMPLE" }
]
}Individual tasks in a Conductor workflow are executed by workers — ordinary services that poll Conductor for work matching a task type, execute it, and report the result back. Conductor itself doesn't run the task logic; it tracks state, decides what runs next, handles retries and timeouts, and exposes the current status of every in-flight workflow instance. That separation kept task implementation fully owned by the teams that understood the domain, while giving Netflix a central, queryable view of every long-running process across the company.
Visibility as the actual payoff
The practical win wasn't just correctness — hand-rolled state machines can be made correct too, with enough effort — it was visibility. Before Conductor, answering "where is this specific piece of content in its processing pipeline right now" often meant grepping logs or querying service-specific databases with undocumented schemas. With workflows centralized in Conductor, that question became a straightforward status lookup, and stuck or failed workflows became something operators could see and often retry or restart directly, rather than something that required an engineer familiar with that specific service's internal state machine to diagnose.
That observability mattered enormously for the kind of processes Conductor was built for — content and studio workflows that can span days and involve human review steps — where "is this stuck, and where" is a question that comes up constantly and used to require specialized tribal knowledge to answer.
A concrete failure mode for workflow engines
Conductor-style orchestration looks like a flowchart for encoding, billing, or notifications. The failure mode is putting business logic in the orchestrator until nobody can test a payment path without standing up the whole DAG. Mid-size steal: workflows for long-running, retryable steps with explicit compensation; keep rules in services. If a flow is three HTTP calls and no human wait, a function with retries is enough.
Operational gotcha: stuck workflows. A worker dies after processing but before ack, the task retries, and you double-charge or double-encode. Idempotency keys on workers are mandatory. Another is versioning: a running instance started on definition v3 while you deployed v4 that renamed a task; the instance can never complete. Steal version pinning per execution. The dashboard becomes a second pager: thousands of failed tasks nobody triages. Alert on age of oldest incomplete workflow by type, not only on worker CPU. Conductor is not a replacement for a ledger; do not store money movement only as workflow state. Side effects that cannot be compensated need a human queue. Mid-size teams should start with a managed workflow or even a transactional outbox plus queue, and graduate to a conductor when the graph of waits and retries is too messy for code.
What you can borrow
- Pull multi-step orchestration logic out of individual services into a dedicated, visible layer rather than letting it accumulate as ad hoc state machines.
- Define workflows declaratively so they're inspectable, versionable, and reviewable independent of the task implementations they coordinate.
- Separate "what runs next" (orchestration) from "how a task executes" (worker implementation) so teams retain ownership of their own task logic.
- Prioritize operational visibility for long-running processes — being able to answer "where is this stuck" quickly is worth real engineering investment.
- Build in retry, timeout, and failure-handling as workflow-level configuration, not something each task has to reimplement independently.