> ## Documentation Index > Fetch the complete documentation index at: https://developer.utgl.io/llms.txt > Use this file to discover all available pages before exploring further. # Overview ## Introduction The Issuing API provides integration partners with a reliable webhook notification system, enabling partners to receive realtime notifications on all events occurring within their account. Such events includes but are not limited to transactions, blockchain confirmations, account balance changes, and payout executions. Webhooks enable partners to build their own copy of historical events throughout the system for various use cases, including reporting, notifications, realtime monitoring, caching and more. ## Webhook Basics Webhooks are delivered to the target endpoint in the order they occur, and a event delivery is considered successful only and if only when a HTTP 200 response is received from the webhook target. A subscription can be created using the Create webhook endpoint, and the subscription can be created to listen on all or a subset of events. The target URL must be a SSL enabled https URL. ## Webhook Integrity Verification Ensuring the integrity and authenticity of webhooks is paramount. To assist integrators in achieving this, we've outlined a method to verify the signature accompanying each webhook. Here's how you can ensure the validity of the webhooks you receive: **Locate the Signature:** Every webhook request will have an associated signature in the `x-access-signature` header. This header contains the value of the payload body signed with our webhook public key. **Understand the Signature Mechanism:** The signature is generated using the RSA-SHA256 algorithm. This is a public key cryptographic signature method. In essence, the sender (typically our server) uses a private key to sign the payload, and the receiver (you) will use a corresponding public key to verify the signature. Example Python implementation of signature verification ```Text Python theme={null} from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives import serialization, hashes from cryptography.hazmat.primitives.asymmetric import padding def verify_signature(public_key_pem, received_signature, body): # Load the public key public_key = serialization.load_pem_public_key( public_key_pem.encode(), backend=default_backend() ) try: # Verify the received signature public_key.verify( bytes(received_signature, 'base64'), # Convert the Base64 encoded signature to bytes body.encode(), padding.PKCS1v15(), hashes.SHA256() ) return True except: return False # Usage public_key_str = "YOUR_RSA_PUBLIC_KEY_IN_PEM_FORMAT" signature_from_header = "SIGNATURE_FROM_X_ACCESS_SIGNATURE_HEADER" webhook_body = "WEBHOOK_BODY_CONTENT" is_valid = verify_signature(public_key_str, signature_from_header, webhook_body) print(is_valid) # True if signature is valid, False otherwise ``` ### Webhook Statuses Webhooks can transition through various states during their lifecycle. Understanding each state is crucial for effective webhook management, troubleshooting, and monitoring. #### Status Lifecycle Diagram The following diagram shows how webhook statuses transition based on delivery success and failure: ```mermaid theme={null} stateDiagram-v2 [*] --> Unhealthy: Webhook Created Unhealthy --> Healthy: First Successful Delivery Healthy --> Unhealthy: Delivery Failure Unhealthy --> Suspended: 10+ Consecutive Failures Suspended --> Healthy: Auto-Recovery Ping Succeeds Suspended --> Deleted: 24 Hours of Failed Pings Deleted --> [*] note right of Unhealthy Initial state after creation. Remains until first successful delivery. end note note right of Suspended Auto-recovery pings every 5 minutes. Manual resume available. end note note right of Deleted Cannot be recovered. Must create new webhook. end note ``` #### Status Summary | Status | Description | Action Required by Cobrander | | ------------- | ----------------------------- | -------------------------------------- | | **Healthy** | Deliveries successful | None - continue monitoring | | **Unhealthy** | Recent delivery failures | Check endpoint health | | **Suspended** | Multiple consecutive failures | Fix endpoint or wait for auto-recovery | | **Deleted** | Removed from system | Must recreate webhook | The initial state for webhooks upon creation is 🟡 **unhealthy** and will remain unhealthy until the first successful webhook has been delivered, in which the webhook will transition to a 🟢 **healthy** state. #### Status Details **Healthy** Description: The webhook is functioning as expected, and deliveries are successful. Implications: The target endpoint is responding appropriately to the webhook notifications. There are no issues detected with the webhook's functionality. Actions: * Continue regular monitoring. * No immediate action is required. **Unhealthy** Description: The webhook has encountered recent delivery failures. Implications: * There might be issues with the receiving endpoint, network connectivity, or other transient problems. * Immediate attention might be required to prevent data inconsistencies. Actions: * Check the target endpoint's health and availability. * Review error logs for insights into the failure. **Suspended** Description: After multiple consecutive failures, the webhook has been paused to prevent further unsuccessful attempts. **Auto-Recovery Mechanism Enabled**: Suspended webhooks are automatically pinged every 5 minutes for up to 24 hours. If a ping succeeds, webhook delivery resumes automatically without manual intervention. Implications: * The system will automatically attempt to recover the webhook. * If auto-recovery fails after 24 hours, the webhook will be automatically deleted. Actions: * Investigate and address the root cause of the failures proactively. * You can also manually resume using the [`/webhooks/resume`](/issuing/api-reference/webhooks/resume-suspended-webhook) endpoint. * Ensure your endpoint is reachable within 24 hours to prevent auto-deletion. **Deleted** Description: The webhook has been removed from the system and is no longer active. Webhooks can be **automatically deleted** if they remain suspended with failed recovery pings for 24 consecutive hours. Implications: * The webhook will no longer send any notifications. * Associated configurations and data are permanently lost. * Auto-deleted webhooks cannot be recovered. Actions: * Ensure the deletion was intentional. * If unintentional, recreate the webhook with a working endpoint. * To prevent auto-deletion, ensure your endpoint is reachable within 24 hours of suspension. ## Delivery Semantics ### **At-Least-Once Delivery Guarantee** Our webhook notification system provides "At-Least-Once" Delivery Guarantee and ensures that webhook notifications are guarantees to be delivered at least once to the target endpoint. How It Works: * If a webhook delivery fails (i.e., any response from the target other than HTTP 200), our system will retry sending the notification after a brief backoff. * Retries will continue until a successful HTTP 200 response is received from the target endpoint. Benefits: * Reliability: Guarantees that no notifications are missed due to temporary failures, be it network issues, server downtime, or any other transient problem. * Consistency: Helps in maintaining consistent data between systems, as no event notifications are lost. Considerations: * While this semantic ensures that no notification is missed, it's crucial to handle webhook deliveries idempotently to avoid processing the same event multiple times. ### **Ordered Event Delivery** Definition: Ordered Event Delivery ensures that all events are delivered to the endpoint in the exact sequence they were triggered. **How It Works:** Events are queued internally within our system per webhook endpoint, ordered based on their triggering sequence. These queued events, are then delivered one by one in their respective order to the target endpoint. **Benefits:** Data Integrity: Preserving the order of events helps maintain the integrity of operations. For instance, a "create" event followed by an "update" event for the same item will always maintain this order, ensuring correct data updates. Predictability: Systems receiving webhooks can anticipate the order of operations and don't need complex logic to reorder or check the sequence of events. **Considerations:** Systems should be designed to handle cases where, due to any unforeseen issues, an event is delayed, and that interruptions to webhook processing is handled promptly. We highly recommend webhook processing to be handled asynchronously. ### Backoff and Retries for Webhook Delivery To ensure the reliability of our webhook delivery, we implement a combination of immediate retries and exponential backoff. The strategy is as follows: **Immediate Retries:** We will make an immediate attempt to redeliver the webhook upon failure. For the first 10 attempts, there will be a delay of 5 seconds between each attempt. **Exponential Backoff:** Should the immediate retries fail to deliver the webhook, we shift to an exponential backoff strategy: * 11th attempt: After the initial 5 retries, we introduce a delay of 30 seconds. * 12th attempt: Delay of 60 seconds. * 13th attempt: Delay of 120 seconds. * 14th attempt: Delay of 240 seconds. * 15th and final attempt: Delay of 480 seconds. After the 15th attempt, if the webhook delivery remains unsuccessful, no further attempts will be made. Ensure to monitor your system's notifications or logs for any failed webhook deliveries to address them appropriately. **Tracking Delivery Attempts:** The number of attempts made for each delivery is included in the header x-access-delivery-attempt. This header provides an integer value representing the number of attempts made for that particular delivery. ### Webhook Delivery Flow The following sequence diagram illustrates the complete webhook delivery lifecycle, including retry logic and status transitions: ```mermaid theme={null} sequenceDiagram participant Event as Event Occurs participant System as UTGL System participant Endpoint as Your Endpoint Event->>System: Trigger webhook event System->>System: Queue webhook delivery loop Delivery Attempts (up to 15) System->>Endpoint: POST webhook payload
+ x-access-signature header
+ x-access-delivery-attempt header Endpoint->>Endpoint: Process webhook alt HTTP 200 Response Endpoint->>System: 200 OK System->>System: Mark delivery successful Note over System: Webhook status: Healthy break Delivery Successful else Non-200 Response or Timeout Endpoint->>System: Error response / Timeout System->>System: Mark delivery failed alt Attempts 1-10 Note over System: Wait 5 seconds else Attempts 11-15 Note over System: Exponential backoff
(30s, 60s, 120s, 240s, 480s) end alt 10+ Consecutive Failures System->>System: Mark webhook as Suspended Note over System: Start auto-recovery
(ping every 5 minutes) end end end alt All 15 Attempts Failed System->>System: Webhook remains Suspended Note over System: Auto-recovery starts
(24 hour window) end ``` ### Retry Timeline Visualization ```mermaid theme={null} gantt title Webhook Delivery Retry Timeline dateFormat X axisFormat %s section Immediate Retries Attempt 1-10 (5s delay) :0, 50 section Exponential Backoff Attempt 11 (30s delay) :50, 30 Attempt 12 (60s delay) :80, 60 Attempt 13 (120s delay) :140, 120 Attempt 14 (240s delay) :260, 240 Attempt 15 (480s delay) :500, 480 ``` ## Handling Webhook Failures Webhooks are a vital component for ensuring real-time synchronization between systems. However, there are occasions when a webhook may fail to deliver its payload to the designated endpoint. Proper handling of these failures is essential to maintain system reliability and data consistency. **Unhealthy Webhooks** Criteria for Unhealthy Status: * A webhook is marked as unhealthy immediately after a failed delivery attempt. * A failed delivery is identified by any response other than a HTTP 200. Implications: * An unhealthy status serves as an early warning system. It indicates that there might be an issue with the receiving endpoint or the network. Immediate attention might be required. **Retrying Failed Webhooks** Retry Mechanism: * The system will automatically attempt to redeliver the webhook notification if it encounters a failure. * The webhook will be retried for up to 10 times, with expoential backoff Considerations: * It's essential to ensure that the receiving system handles webhook deliveries idempotently to avoid processing the same event multiple times due to retries. * Monitoring tools should be set up to alert administrators or developers of repeated failures, allowing for quick intervention. **Suspending Webhooks** Criteria for Suspension: * If a webhook fails to deliver successfully after 10 retries, it will be marked as suspended. Implications: * A suspended webhook will no longer attempt to deliver its notifications, preventing further system or network strain. * Administrators or system maintainers should be alerted when a webhook is suspended to investigate and address the root cause. **Resuming Suspended Webhooks** Endpoint for Resumption: * Suspended webhooks can be resumed using the [`/webhooks/resume`](/issuing/api-reference/webhooks/resume-suspended-webhook) endpoint Procedure: * Identify the webhookId of the suspended webhook. * Send a request to [`/webhooks/resume`](/issuing/api-reference/webhooks/resume-suspended-webhook) with `webhookId=YOUR_WEBHOOK_ID` to initiate the resumption process. * Monitor the webhook to ensure successful deliveries post-resumption. Considerations: Before resuming a suspended webhook, it's advisable to ensure that the root cause of the failures has been addressed. This might involve checking the receiving server's health, network connectivity, or the correctness of the webhook's endpoint URL. ## Auto-Recovery Mechanism To reduce manual intervention and prevent accumulation of stale webhooks, the system includes an automatic recovery mechanism for suspended webhooks. ### How It Works ```mermaid theme={null} flowchart TD A[🔴 Webhook Suspended] --> B[🔄 Auto-Recovery Starts] B --> C[Ping endpoint every 5 minutes] C --> D{Response OK?} D -->|✅ Yes| E[🟢 Webhook Resumed] D -->|❌ No| F{24 hours passed?} F -->|No| C F -->|Yes| G[⛔ Webhook Deleted] style A fill:#ef4444,color:#ffffff style E fill:#22c55e,color:#ffffff style G fill:#dc2626,color:#ffffff ``` ### Recovery Timeline | Phase | Behavior | | ------------------------- | -------------------------------------- | | **Ping Frequency** | Every 5 minutes | | **Recovery Window** | Up to 24 hours | | **On Success** | Webhook delivery resumes automatically | | **On Failure (24 hours)** | Webhook is permanently deleted | ### Recovery Outcomes Webhook delivery **resumes automatically**. No manual action required. All queued events will be delivered. Webhook is **automatically deleted**. You must create a new webhook subscription to resume receiving events. **Important**: Once a webhook is auto-deleted after 24 hours of failed recovery attempts, it cannot be restored. You must create a new webhook subscription. ## Best Practices ### Webhook Health Check Definition: Webhook health monitoring is the practice of proactively tracking the delivery status of your webhook subscriptions to detect and resolve issues before they impact your system. Recommendation: * Implement automated monitoring in your system to track webhook health status using the [`/webhooks/stats`](/issuing/api-reference/webhooks/get-webhook-stats) endpoint. * Set up alerts to notify your team immediately when a webhook status changes to 🟡 **unhealthy** or 🔴 **suspended**. * Benefits of proactive health monitoring include: * **Early detection** of delivery failures before they escalate to suspension. * **Reduced downtime** by addressing endpoint issues promptly. * **Preventing auto-deletion** by ensuring suspended webhooks are fixed within 24 hours. **Pro tip**: Schedule regular health checks (e.g., every 5-15 minutes) and integrate alerts with your incident management system (Slack, PagerDuty, etc.) for immediate visibility. ### Idempotent Processing Definition: Idempotent processing means that an operation can be repeated multiple times without changing the result beyond the initial application. Recommendation: * Always design your webhook processing logic to handle webhook deliveries idempotently. By doing so, you'll ensure: * Duplicate webhook deliveries, due to retries, don't result in duplicate data processing. * Increased system reliability, as the same event won't be processed multiple times. ### Dedicated Webhook subscriptions for High Priority Events Background: Some events, such as "3DS code received," are of high importance and need immediate attention. Recommendation: * Always create separate webhooks specifically for these high-priority events. * By isolating critical events, you can: * Ensure that they are processed promptly. * Avoid potential delays caused by lower-priority webhook events. ### Asynchronous Processing Definition: Asynchronous processing means that the system can continue with other tasks without waiting for the webhook to complete processing. Recommendation: * For systems that handle a significant volume of cards and transactions, it's crucial to process webhook notifications asynchronously. * Benefits of asynchronous processing include: * Preventing performance bottlenecks. * Ensuring that webhook events are delivered and processed in a timely manner. * Enhancing the overall system responsiveness. We recommend dispatching webhook events into a message queue or tempory storage to ensure the process for receiving webhooks is separated from processing, this minimizes webhook delivery latency and maximizes webhook delivery reliability. It also reduces the chance of our delivery system from marking the webhook endpoint as unhealthy due to failure to process webhooks. # Conclusion By building your integration around these delivery semantics, systems can ensure reliable, consistent, and orderly data synchronization through webhooks. Proper handling and processing on the receiving end are essential to make the most out of these semantics. *** * [Table of Contents](#) * * * [Introduction](#introduction) * [Webhook Basics](#webhook-basics) * [Webhook Integrity Verification](#webhook-integrity-verification) * [Delivery Semantics](#delivery-semantics) * [Handling Webhook Failures](#handling-webhook-failures) * [Auto-Recovery Mechanism](#auto-recovery-mechanism) * [Best Practices](#best-practices) * [Conclusion](#conclusion)