The Backend Behind a Developer Portal: Durable Requests, Idempotency, and Dispatch
A green button can hide an unstarted operation.
Your portal accepts a request to create an environment. The backend writes a database row, then crashes before starting the downstream job. The developer retries. Now the platform either creates duplicate work or insists that an operation already exists, even though nothing is running.
The interface is working. The control plane has lost the meaning of the request.
A reliable backend needs three distinct responsibilities: admission, durable operation state, and recoverable dispatch. A catalog card can display the result, but it should not be the sole record of an infrastructure operation.
Start with the admission contract.
An authenticated request carries intent. It does not grant authority. Resolve the caller on the server, verify ownership and target access, apply policy, and normalize the request before admitting work.
Return an operation identifier, its current state, and a status address. Return Accepted only after the database transaction commits. Accepted means the request has been durably admitted. It does not mean Terraform has applied successfully or a workload has become healthy.
There are several identities here, and collapsing them creates bugs.
A caller plus an idempotency key identifies a request replay. A canonical target identifies conflicting active work. The operation identifier anchors dispatch retries. A provider resource or workflow identifier anchors external reconciliation.
Normalize target names centrally. Two strings referring to the same environment must not bypass the active-work guard. Store the action and resolved inputs too. The same target does not make two different actions the same request.
Next, commit intent and dispatch eligibility together.
Writing a database row and starting a remote workflow are separate side effects. Either order leaves a crash window. The transactional outbox pattern writes the operation and a dispatch record in one local transaction. A dispatcher delivers that record later.
This closes the local dual-write gap. It does not guarantee exactly-once provisioning.
The dispatcher can still crash after the workflow engine accepts a job but before the database records delivery. The downstream system therefore needs to recognize the stable operation identifier, or provide a way to reconcile the existing job before repeating the effect.
The PostgreSQL example in the article makes that local contract concrete.
The operations table has a unique constraint on actor and idempotency key. A second, partial unique index permits only one accepted or running operation for each target.
Those constraints solve different problems. The first preserves the original operation when a caller retries. The second prevents another request from starting conflicting active work against the same target.
The admission function inserts the operation. If the caller's key already exists, it compares the stored target and request with the new intent. An identical replay returns the original identifier. A changed request using that key is rejected.
For a new operation, the function inserts an outbox record in the same transaction. If that insertion fails, the operation does not remain committed on its own.
The example uses Read Committed isolation. Stronger isolation levels need deliberate retry handling. Retain the operation record for the promised replay period; deleting it too soon removes your ability to recognize an old retry.
There is also an authorization boundary to understand. This database function does not authenticate the actor string. The trusted backend must derive it from the actual caller. Restrict database roles and schema access; never let a browser supply its own authority through a direct database connection.
Now look at the dispatcher.
Claim one eligible outbox row with a short database statement. The example uses skip locked so competing queue consumers can move past an already locked row. It creates a lease and a fresh claim token, increments the attempt count, and returns the payload.
Commit the claim before making the remote call. Holding database locks while waiting on a network dependency turns downstream slowness into local contention.
Send the operation identifier as the downstream deduplication identity. After confirmed durable acceptance, acknowledge delivery only if the claim token still matches and the lease has not expired.
The token prevents a stale worker from acknowledging another worker's claim. It does not prevent that stale worker from making a remote call. Remote deduplication or reconciliation is still essential.
Delivered means another system accepted responsibility. It does not mean the infrastructure operation succeeded. Completion must come from observed workflow or provider state.
Consider the acknowledgement gap.
The workflow engine accepts operation O. The dispatcher loses its connection before recording delivery. Later, the lease expires and another worker claims the row.
Do not mint a new operation identifier to make the retry look successful.
Inspect the downstream system using O. If a workflow already owns that identity, reconcile it. If the destination supports atomic deduplication, retry the same admission identity. If acceptance is uncertain and neither mechanism is available, expose a repair state for an operator rather than blindly repeating a potentially irreversible effect.
Use bounded retries and backoff. Keep the failure boundary visible.
Finally, treat status and cancellation as contracts.
Update operation state with expected-state or version checks. A delayed callback must not silently overwrite a newer decision. A cancellation request expresses intent; the executor must report whether work stopped, completed, or requires compensation.
Keep authoritative operation state separate from portal projections. Show observation time, external identifiers, and stale or incomplete evidence. Authorize status reads as well as writes: plans and resource names can reveal sensitive infrastructure information.
Record the requester, policy decision, normalized intent, template revision, and external receipts. Keep credentials out of generic request payloads and logs.
The architectural payoff is recoverability. Operators can repair the same durable operation instead of pressing a Retry button that creates another one.
The article contains the complete database example, its commands, cleanup, and source documentation. This listening version explains the design; it does not claim that the example provisioned real infrastructure.
Three distinctions are worth carrying into your next design review: accepted is not completed; idempotency is not authorization; and an outbox is not exactly-once execution.