AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Infrastructure Dependency Graphs: Release Work Only After Success
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Build a bounded dependency scheduler that releases checks only after success, rejects cycles before execution and preserves actionable failure reports.
TL;DR: Infrastructure dependency graphs are executable contracts: a dependent stage becomes eligible only after its prerequisites succeed. This article builds a bounded Python preflight scheduler that uses a readiness frontier, validates explicit predecessors and rejects cycles before callbacks run. Executed correctness checks cover independent progress, two-worker bounds and failures that hold dependent work. The example validates local manifests, with no cloud provisioning or performance claims. It explains why completed futures are not successful prerequisites, why abort does not undo running work, and why Terraform resource ordering must remain distinct from platform workflow and runtime readiness.
A platform request can be perfectly valid yet fail because work starts in the wrong order. A service needs its database and identity contract; the database needs a subnet; the subnet needs a network. Serializing everything hides this problem behind slow delivery. Starting everything together turns it into intermittent failures.
A dependency graph gives the platform a precise release rule: a task becomes eligible when all its predecessors have succeeded. That rule is useful for provisioning plans, migration preflights and environment checks. It also exposes an uncomfortable distinction: a completed worker is not necessarily a successful prerequisite.
This article implements a bounded, synchronous preflight scheduler using Python 3.11's standard library. It executes local manifest checks, demonstrates failure containment and validates readiness without provisioning infrastructure.
Encode prerequisites, not a preferred ordering
Our graph has two initial tasks, network and identity. Subnet follows network; database follows subnet; service follows both database and identity; probe follows service. The names represent validation stages in this example. They do not imply that a database or service is deployed.
Python's TopologicalSorter accepts a dictionary whose values are predecessors. Thus "service": ("database", "identity") means both prerequisites must finish before service is released. Reversing this relationship can produce a valid but operationally wrong order. The scheduler rejects missing predecessor names instead of accepting an accidental implicit root. Python graphlib documentation, Python 3.11.3 implementation.
Image. Dependency frontier after a database check fails
The diagram shows the separately tested database-failure case: network, subnet and identity succeed; database fails; service and probe never start. Arrows point from prerequisite to dependent. This is a schematic of the local correctness test, with no cloud timing or resource state implied.
Give each edge an explicit meaning. “Network configuration is syntactically acceptable” is weaker than “the subnet is reachable.” “The provider accepted creation” is weaker than “the service can authenticate.” Combining these meanings under a single success flag makes the graph look reassuring while leaving its contract ambiguous.
Keep the platform graph separate from Terraform's graph
Terraform already derives dependencies from references to resource attributes. Prefer those references when they express the actual relationship. Use depends_on for a hidden dependency that cannot be represented by the required value; broad module dependencies can make planning more conservative and leave more values unknown. Expression references, depends_on semantics.
An outer platform graph might order admission checks, independently managed stacks, application deployment and a post-deployment probe. It should not recreate every resource edge inside a Terraform operation. Terraform's execution graph includes provider configuration and can split replacement into distinct destroy and create nodes because their ordering differs. Its concurrency limit also does not directly solve provider API rate limits. Terraform dependency graph.
For an existing configuration, terraform graph produces DOT for inspection; its default output is a simplified resource dependency graph. A visualization is useful evidence when reviewing an unexpected dependency, but it is not an executable platform workflow or a runtime health check. Terraform graph command.
Use a readiness frontier instead of a level barrier
A static topological ordering is sufficient for a serial walk. A worker pool needs a frontier: tasks currently eligible to start. After network finishes, subnet can run even if identity remains busy. Waiting for every task in the first “wave” wastes that opportunity.
The scheduler calls prepare() before invoking a callback, obtains ready nodes, and submits no more than the configured worker limit. Only the scheduler thread calls get_ready() and done(); workers evaluate checks. A future becoming done says it finished or was cancelled, not that the business check passed. Calling result() observes its exception. Concurrent futures.
The callback contract is deliberately simple: any normal return counts as success, including False or None. A failed validator must raise. Callbacks must be synchronous and terminate within their own bounded work; an awaitable result is rejected rather than being mistaken for completed validation. Awaitable inspection.
Complete local preflight example
Prerequisites are Python 3.11 and a writable scratch directory. The example uses only the standard library, with no provider credentials, subprocesses, network access or cloud mutations. Its concrete implementation was checked with Python 3.11.3.
Save the complete code as topic-24-preflight.py. The CIDR checks use ip_network and subnet containment; they validate the supplied strings, not routing or deployment state. Python ipaddress.
Code example. Refer to the written article for the complete code.
Run it without optimized mode, which would disable the example's assertions:
Code example. Refer to the written article for the complete code.
The first run checks a valid manifest. The second changes the image to latest, which violates this example's digest contract. It records service as failed and leaves probe unstarted. A digest-shaped string passing this check does not prove the image exists, is signed or is approved: add those as distinct validators where required.
The graph is bounded to 64 explicit nodes, 256 unique edges and one to eight workers. These are defensive implementation limits, not throughput recommendations. Duplicate predecessor entries are normalized, and input mappings are copied before scheduling so the caller cannot change the dependency declaration mid-run.
Cleanup is limited to the saved script and any locally generated receipts. This example creates no infrastructure to destroy.
Diagnose failure before retrying the graph
The report distinguishes callback success from dependency release. A callback can finish successfully while another callback fails in the same observed batch. The scheduler then aborts conservatively: it starts nothing further, waits for already running callbacks, and does not release additional successors.
Consequently, not_started includes both dependent tasks and unrelated eligible work that was never submitted. Do not label every entry “blocked by database.” Preserve the failed node, its diagnostic evidence, and the input version that produced the result. This implementation records exception types; a production validator needs a safe, structured error with enough detail to locate the invalid field.
Recovery for the bad-image failure is to supply an approved digest and run a fresh preflight over the corrected manifest. Recovery for a database-contract failure is to correct that prerequisite before considering service validation. Neither recovery should mark a failed node done merely to keep the pipeline moving.
Cycle handling is equally strict. prepare() detects a cycle before any callback runs here. The underlying API permits retrieving some independent ready nodes after a cycle error, but this scheduler rejects the whole plan. The reported cycle is one detected cycle, not an inventory of every cycle. Fix the dependency model, then construct a new sorter; do not suppress an edge just to obtain a runnable ordering.
What the executed checks establish
The retained correctness receipt records six released checks for the valid graph. Injected service failure leaves probe unstarted; injected database failure leaves service and probe unstarted. A cyclic graph invokes zero callbacks.
A gated concurrency test holds identity open while network, subnet and database advance. Database then releases the gate, allowing service to proceed. Its observed maximum is two simultaneous callbacks. This establishes bounded readiness behavior without asserting a speedup, production capacity or cloud reliability.
Additional assertions cover unknown predecessors, invalid worker bounds and asynchronous callback rejection. These are functional observations from local fixtures. They do not test provider eventual consistency, permissions, quotas or deployed health.
Define the boundary before using this in a platform
A real platform needs durable operation state, authorization, idempotency and compensation decisions around the graph. A process restart loses this scheduler's state. A hung callback can prevent draining because a Python worker thread cannot be forcibly cancelled safely. Add deadlines inside the actual operation or use an execution system with the required isolation.
Readiness also needs a stable artifact contract. If a predecessor emits configuration, record its verified version before releasing consumers. Reusing a prior success after changing the plan can invalidate the edge's meaning. Treat runtime health as an explicit check with its own timeout and evidence, rather than stretching a configuration dependency into an availability guarantee.
The useful invariant is narrow and reviewable: release dependent work only after its declared prerequisite checks succeed. Everything beyond that invariant needs its own state and failure policy.
Frequently asked questions
Does a topological order prove the infrastructure is ready? No. It orders the declared dependencies. Readiness depends on what each successful callback actually establishes.
Can I call done in a finally block? No. That releases dependents even after validation fails. Call it only after the scheduler has accepted success.
Will abort cancel work already running? No. This example stops new submissions and drains submitted callbacks. It provides no rollback or hard deadline.
Should I copy Terraform's graph into the platform? Usually keep resource dependencies within Terraform. Model the platform stages and cross-operation contracts that Terraform does not own.