Skip to main content

​
Introduction

The Issuing API provides integration partners with a reliable webhook notification system, enabling partners to receive realtime notifications on all events occurring within their account. Such events includes but are not limited to transactions, blockchain confirmations, account balance changes, and payout executions. Webhooks enable partners to build their own copy of historical events throughout the system for various use cases, including reporting, notifications, realtime monitoring, caching and more.

​
Webhook Basics

Webhooks are delivered to the target endpoint in the order they occur, and a event delivery is considered successful only and if only when a HTTP 200 response is received from the webhook target. A subscription can be created using the Create webhook endpoint, and the subscription can be created to listen on all or a subset of events. The target URL must be a SSL enabled https URL.

​
Webhook Integrity Verification

Ensuring the integrity and authenticity of webhooks is paramount. To assist integrators in achieving this, we’ve outlined a method to verify the signature accompanying each webhook. Here’s how you can ensure the validity of the webhooks you receive: Locate the Signature: Every webhook request will have an associated signature in the x-access-signature header. This header contains the value of the payload body signed with our webhook public key. Understand the Signature Mechanism: The signature is generated using the RSA-SHA256 algorithm. This is a public key cryptographic signature method. In essence, the sender (typically our server) uses a private key to sign the payload, and the receiver (you) will use a corresponding public key to verify the signature. Example Python implementation of signature verification 
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization, hashes
from cryptography.hazmat.primitives.asymmetric import padding

def verify_signature(public_key_pem, received_signature, body):
    # Load the public key
    public_key = serialization.load_pem_public_key(
        public_key_pem.encode(),
        backend=default_backend()
    )
    
    try:
        # Verify the received signature
        public_key.verify(
            bytes(received_signature, 'base64'),  # Convert the Base64 encoded signature to bytes
            body.encode(),
            padding.PKCS1v15(),
            hashes.SHA256()
        )
        return True
    except:
        return False

# Usage
public_key_str = "YOUR_RSA_PUBLIC_KEY_IN_PEM_FORMAT"
signature_from_header = "SIGNATURE_FROM_X_ACCESS_SIGNATURE_HEADER"
webhook_body = "WEBHOOK_BODY_CONTENT"

is_valid = verify_signature(public_key_str, signature_from_header, webhook_body)
print(is_valid)  # True if signature is valid, False otherwise

​
Webhook Statuses

Webhooks can transition through various states during their lifecycle. Understanding each state is crucial for effective webhook management, troubleshooting, and monitoring.

​
Status Lifecycle Diagram

The following diagram shows how webhook statuses transition based on delivery success and failure:

​
Status Summary

StatusDescriptionAction Required by Cobrander
HealthyDeliveries successfulNone - continue monitoring
UnhealthyRecent delivery failuresCheck endpoint health
SuspendedMultiple consecutive failuresFix endpoint or wait for auto-recovery
DeletedRemoved from systemMust recreate webhook
The initial state for webhooks upon creation is 🟡 unhealthy and will remain unhealthy until the first successful webhook has been delivered, in which the webhook will transition to a 🟢 healthy state.

​
Status Details

Healthy Description: The webhook is functioning as expected, and deliveries are successful. Implications: The target endpoint is responding appropriately to the webhook notifications. There are no issues detected with the webhook’s functionality. Actions:
  • Continue regular monitoring.
  • No immediate action is required.
Unhealthy Description: The webhook has encountered recent delivery failures. Implications:
  • There might be issues with the receiving endpoint, network connectivity, or other transient problems.
  • Immediate attention might be required to prevent data inconsistencies.
Actions:
  • Check the target endpoint’s health and availability.
  • Review error logs for insights into the failure.
Suspended Description: After multiple consecutive failures, the webhook has been paused to prevent further unsuccessful attempts.
Auto-Recovery Mechanism Enabled: Suspended webhooks are automatically pinged every 5 minutes for up to 24 hours. If a ping succeeds, webhook delivery resumes automatically without manual intervention.
Implications:
  • The system will automatically attempt to recover the webhook.
  • If auto-recovery fails after 24 hours, the webhook will be automatically deleted.
Actions:
  • Investigate and address the root cause of the failures proactively.
  • You can also manually resume using the /webhooks/resume endpoint.
  • Ensure your endpoint is reachable within 24 hours to prevent auto-deletion.
Deleted Description: The webhook has been removed from the system and is no longer active.
Webhooks can be automatically deleted if they remain suspended with failed recovery pings for 24 consecutive hours.
Implications:
  • The webhook will no longer send any notifications.
  • Associated configurations and data are permanently lost.
  • Auto-deleted webhooks cannot be recovered.
Actions:
  • Ensure the deletion was intentional.
  • If unintentional, recreate the webhook with a working endpoint.
  • To prevent auto-deletion, ensure your endpoint is reachable within 24 hours of suspension.

​
Delivery Semantics

​
At-Least-Once Delivery Guarantee

Our webhook notification system provides “At-Least-Once” Delivery Guarantee and ensures that webhook notifications are guarantees to be delivered at least once to the target endpoint. How It Works:
  • If a webhook delivery fails (i.e., any response from the target other than HTTP 200), our system will retry sending the notification after a brief backoff.
  • Retries will continue until a successful HTTP 200 response is received from the target endpoint.
Benefits:
  • Reliability: Guarantees that no notifications are missed due to temporary failures, be it network issues, server downtime, or any other transient problem.
  • Consistency: Helps in maintaining consistent data between systems, as no event notifications are lost.
Considerations:
  • While this semantic ensures that no notification is missed, it’s crucial to handle webhook deliveries idempotently to avoid processing the same event multiple times.

​
Ordered Event Delivery

Definition: Ordered Event Delivery ensures that all events are delivered to the endpoint in the exact sequence they were triggered. How It Works: Events are queued internally within our system per webhook endpoint, ordered based on their triggering sequence. These queued events, are then delivered one by one in their respective order to the target endpoint. Benefits: Data Integrity: Preserving the order of events helps maintain the integrity of operations. For instance, a “create” event followed by an “update” event for the same item will always maintain this order, ensuring correct data updates. Predictability: Systems receiving webhooks can anticipate the order of operations and don’t need complex logic to reorder or check the sequence of events. Considerations: Systems should be designed to handle cases where, due to any unforeseen issues, an event is delayed, and that interruptions to webhook processing is handled promptly. We highly recommend webhook processing to be handled asynchronously.

​
Backoff and Retries for Webhook Delivery

To ensure the reliability of our webhook delivery, we implement a combination of immediate retries and exponential backoff. The strategy is as follows: Immediate Retries: We will make an immediate attempt to redeliver the webhook upon failure. For the first 10 attempts, there will be a delay of 5 seconds between each attempt. Exponential Backoff: Should the immediate retries fail to deliver the webhook, we shift to an exponential backoff strategy:
  • 11th attempt: After the initial 5 retries, we introduce a delay of 30 seconds.
  • 12th attempt: Delay of 60 seconds.
  • 13th attempt: Delay of 120 seconds.
  • 14th attempt: Delay of 240 seconds.
  • 15th and final attempt: Delay of 480 seconds.
After the 15th attempt, if the webhook delivery remains unsuccessful, no further attempts will be made. Ensure to monitor your system’s notifications or logs for any failed webhook deliveries to address them appropriately. Tracking Delivery Attempts: The number of attempts made for each delivery is included in the header x-access-delivery-attempt. This header provides an integer value representing the number of attempts made for that particular delivery.

​
Webhook Delivery Flow

The following sequence diagram illustrates the complete webhook delivery lifecycle, including retry logic and status transitions:

​
Retry Timeline Visualization

​
Handling Webhook Failures

Webhooks are a vital component for ensuring real-time synchronization between systems. However, there are occasions when a webhook may fail to deliver its payload to the designated endpoint. Proper handling of these failures is essential to maintain system reliability and data consistency. Unhealthy Webhooks Criteria for Unhealthy Status:
  • A webhook is marked as unhealthy immediately after a failed delivery attempt.
  • A failed delivery is identified by any response other than a HTTP 200.
Implications:
  • An unhealthy status serves as an early warning system. It indicates that there might be an issue with the receiving endpoint or the network. Immediate attention might be required.
Retrying Failed Webhooks Retry Mechanism:
  • The system will automatically attempt to redeliver the webhook notification if it encounters a failure.
  • The webhook will be retried for up to 10 times, with expoential backoff
Considerations:
  • It’s essential to ensure that the receiving system handles webhook deliveries idempotently to avoid processing the same event multiple times due to retries.
  • Monitoring tools should be set up to alert administrators or developers of repeated failures, allowing for quick intervention.
Suspending Webhooks Criteria for Suspension:
  • If a webhook fails to deliver successfully after 10 retries, it will be marked as suspended.
Implications:
  • A suspended webhook will no longer attempt to deliver its notifications, preventing further system or network strain.
  • Administrators or system maintainers should be alerted when a webhook is suspended to investigate and address the root cause.
Resuming Suspended Webhooks Endpoint for Resumption: Procedure:
  • Identify the webhookId of the suspended webhook.
  • Send a request to /webhooks/resume with webhookId=YOUR_WEBHOOK_ID to initiate the resumption process.
  • Monitor the webhook to ensure successful deliveries post-resumption.
Considerations: Before resuming a suspended webhook, it’s advisable to ensure that the root cause of the failures has been addressed. This might involve checking the receiving server’s health, network connectivity, or the correctness of the webhook’s endpoint URL.

​
Auto-Recovery Mechanism

To reduce manual intervention and prevent accumulation of stale webhooks, the system includes an automatic recovery mechanism for suspended webhooks.

​
How It Works

​
Recovery Timeline

PhaseBehavior
Ping FrequencyEvery 5 minutes
Recovery WindowUp to 24 hours
On SuccessWebhook delivery resumes automatically
On Failure (24 hours)Webhook is permanently deleted

​
Recovery Outcomes

✅ Ping Succeeds

Webhook delivery resumes automatically. No manual action required. All queued events will be delivered.

❌ 24 Hours of Failed Pings

Webhook is automatically deleted. You must create a new webhook subscription to resume receiving events.
Important: Once a webhook is auto-deleted after 24 hours of failed recovery attempts, it cannot be restored. You must create a new webhook subscription.

​
Best Practices

​
Webhook Health Check

Definition: Webhook health monitoring is the practice of proactively tracking the delivery status of your webhook subscriptions to detect and resolve issues before they impact your system. Recommendation:
  • Implement automated monitoring in your system to track webhook health status using the /webhooks/stats endpoint.
  • Set up alerts to notify your team immediately when a webhook status changes to 🟡 unhealthy or 🔴 suspended.
  • Benefits of proactive health monitoring include:
    • Early detection of delivery failures before they escalate to suspension.
    • Reduced downtime by addressing endpoint issues promptly.
    • Preventing auto-deletion by ensuring suspended webhooks are fixed within 24 hours.
Pro tip: Schedule regular health checks (e.g., every 5-15 minutes) and integrate alerts with your incident management system (Slack, PagerDuty, etc.) for immediate visibility.

​
Idempotent Processing

Definition: Idempotent processing means that an operation can be repeated multiple times without changing the result beyond the initial application. Recommendation:
  • Always design your webhook processing logic to handle webhook deliveries idempotently. By doing so, you’ll ensure:
  • Duplicate webhook deliveries, due to retries, don’t result in duplicate data processing.
  • Increased system reliability, as the same event won’t be processed multiple times.

​
Dedicated Webhook subscriptions for High Priority Events

Background: Some events, such as “3DS code received,” are of high importance and need immediate attention. Recommendation:
  • Always create separate webhooks specifically for these high-priority events.
  • By isolating critical events, you can:
    • Ensure that they are processed promptly.
  • Avoid potential delays caused by lower-priority webhook events.

​
Asynchronous Processing

Definition: Asynchronous processing means that the system can continue with other tasks without waiting for the webhook to complete processing. Recommendation:
  • For systems that handle a significant volume of cards and transactions, it’s crucial to process webhook notifications asynchronously.
  • Benefits of asynchronous processing include:
    • Preventing performance bottlenecks.
    • Ensuring that webhook events are delivered and processed in a timely manner.
    • Enhancing the overall system responsiveness.
We recommend dispatching webhook events into a message queue or tempory storage to ensure the process for receiving webhooks is separated from processing, this minimizes webhook delivery latency and maximizes webhook delivery reliability. It also reduces the chance of our delivery system from marking the webhook endpoint as unhealthy due to failure to process webhooks.

​
Conclusion

By building your integration around these delivery semantics, systems can ensure reliable, consistent, and orderly data synchronization through webhooks. Proper handling and processing on the receiving end are essential to make the most out of these semantics.