Idempotency and Webhook Reliability in Payment Systems | RiyadaVenture

In a distributed system, network failures are inevitable. A client makes a payment request, the server executes it, but the TCP connection drops before the HTTP 200 OK is returned. To the client, it looks like a failure. If the client retries, they risk double-charging the customer. This is why idempotency is non-negotiable in payment architecture.

The Idempotency Key

An idempotent operation is one that yields the exact same state, no matter how many times it is executed. In RiyadaVenture's API, every state-mutating request (POST, PUT, DELETE) expects an Idempotency-Key header (usually a v4 UUID generated by the client).

When the API receives a request with an Idempotency Key:

It acquires a distributed lock via Redis using the key.
It checks the primary database to see if the key already exists.
If it exists, the API short-circuits and immediately returns the cached HTTP response of the original successful request.
If it does not exist, the API processes the payment, caches the response against the key, and releases the lock.

Idempotency vs Concurrency

If two identical requests with the same key arrive simultaneously, the distributed lock ensures only one request begins processing. The second request blocks until the first completes, then returns the cached result.

Webhook Reliability and Retries

Webhooks are the inverse of idempotency—they are asynchronous state updates pushed from the PSP to your server (e.g., payment_intent.succeeded). If your server is down, you miss the update.

To handle this safely:

Signature Verification: Validate the webhook signature using a shared HMAC secret to prevent spoofed successful payments.
Exponential Backoff: RiyadaVenture will retry failed webhooks up to 72 hours using an exponential backoff algorithm. Your ingestion endpoint must respond with a 2xx status immediately. Do not perform heavy work synchronously in the webhook handler.
Idempotent Ingestion: Use the event_id included in the webhook payload as an idempotency key on your end to prevent double-processing if RiyadaVenture fires the webhook twice due to network latency.

Proper system boundaries abstract away the chaos of network failures. For further architectural deep-dives, read our guide on building internal payment ledgers.

The Idempotency Key

When the API receives a request with an Idempotency Key:

It acquires a distributed lock via Redis using the key.

It checks the primary database to see if the key already exists.

If it exists, the API short-circuits and immediately returns the cached HTTP response of the original successful request.

If it does not exist, the API processes the payment, caches the response against the key, and releases the lock.

Idempotency vs Concurrency

Webhook Reliability and Retries

Webhooks are the inverse of idempotency—they are asynchronous state updates pushed from the PSP to your server (e.g., payment_intent.succeeded). If your server is down, you miss the update.

To handle this safely:

Signature Verification: Validate the webhook signature using a shared HMAC secret to prevent spoofed successful payments.

Exponential Backoff: RiyadaVenture will retry failed webhooks up to 72 hours using an exponential backoff algorithm. Your ingestion endpoint must respond with a 2xx status immediately. Do not perform heavy work synchronously in the webhook handler.

Idempotent Ingestion: Use the event_id included in the webhook payload as an idempotency key on your end to prevent double-processing if RiyadaVenture fires the webhook twice due to network latency.

Proper system boundaries abstract away the chaos of network failures. For further architectural deep-dives, read our guide on building internal payment ledgers.