Webhook Worker

Referenced Files in This Document - webhook.py - webhook.py - scanner.py - config.py - init.py - listener.py - sweeper.py - payment.py - README.md

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document describes the Webhook Worker responsible for delivering payment status notifications in the cTrip Payment Gateway. It explains how the worker sends webhook callbacks to merchant endpoints, how payloads are formatted and signed, and how delivery outcomes are handled. It also documents retry behavior, signature verification, security measures, filtering and categorization of events, batch processing, authentication, rate limiting, endpoint validation, monitoring, resubmission workflows, and manual intervention procedures.

Project Structure

The webhook system spans several modules: - Workers: background task actors powered by ARQ and Redis - Services: core logic for sending webhooks and blockchain scanning - Configuration: global settings for webhook URL and secret - API: payment creation endpoints (unrelated to webhook delivery) - Listener and Sweeper: periodic workers that drive scanning and sweeping

graph TB subgraph "Workers" W1["app/workers/webhook.py
send_webhook_task"] W2["app/workers/listener.py
listen_for_payments"] W3["app/workers/sweeper.py
sweep_funds"] end subgraph "Services" S1["app/services/webhook.py
WebhookService"] S2["app/services/blockchain/scanner.py
ScannerService"] end subgraph "Config" C1["app/core/config.py
Settings"] end subgraph "Broker" B1["app/workers/__init__.py
Redis broker"] end subgraph "API" A1["app/api/v1/payments.py
Payment endpoints"] end W2 --> S2 S2 --> W1 W1 --> S1 C1 --> W1 C1 --> S2 B1 --> W1 B1 --> W2 B1 --> W3 A1 -. unrelated .-> W1

Diagram sources - webhook.py - webhook.py - scanner.py - config.py - init.py - listener.py - sweeper.py - payment.py

Section sources - README.md - init.py

Core Components

  • Webhook Actor: a ARQ task that sends webhooks asynchronously and participates in the retry mechanism.
  • Webhook Service: encapsulates HTTP posting, JSON serialization, optional HMAC-SHA256 signing, and logging.
  • Scanner Service: detects and confirms payments, then triggers webhook delivery via the actor.
  • Configuration: exposes webhook_url and webhook_secret used by the actor and service.
  • Broker Initialization: wires Redis as the ARQ Redis backend for reliable task delivery.

Key responsibilities: - Asynchronous webhook delivery with structured logging - Optional payload signing for authenticity verification - Retry handling via ARQ’s built-in retry policy - Event categorization by payment status transitions - Batch processing through periodic scanning and confirmation cycles

Section sources - webhook.py - webhook.py - scanner.py - config.py - init.py

Architecture Overview

The webhook delivery pipeline is event-driven and asynchronous: - Listener periodically scans for detected payments and confirms them - On confirmation, the system constructs a webhook payload and enqueues a task - The Webhook Actor executes the task, signs the payload if configured, and posts to the merchant endpoint - Delivery outcomes are logged; failures trigger ARQ retries

sequenceDiagram participant L as "Listener Actor" participant SC as "ScannerService" participant DB as "Database" participant WA as "Webhook Actor" participant WS as "WebhookService" participant EP as "Merchant Endpoint" L->>SC : "scan_chain()" SC->>DB : "select pending/detected" SC->>DB : "update status to confirmed" SC->>WA : "enqueue send_webhook_task(url, payload, secret)" WA->>WS : "send_webhook(url, payload, secret)" WS->>EP : "POST JSON payload" EP-->>WS : "HTTP 2xx/4xx/5xx" WS-->>WA : "success/failure" WA-->>L : "task completion"

Diagram sources - listener.py - scanner.py - webhook.py - webhook.py

Detailed Component Analysis

Webhook Actor

The Webhook Actor wraps the asynchronous sending logic and integrates with ARQ’s retry mechanism. It: - Receives the merchant URL, payload, and optional secret - Runs an internal async loop to call the Webhook Service - Raises exceptions on failure to trigger retries - Logs errors and forwards them to ARQ

flowchart TD Start(["Actor invoked"]) --> Run["Run async send_webhook_task"] Run --> CallService["Call WebhookService.send_webhook(url, payload, secret)"] CallService --> Success{"Success?"} Success --> |Yes| Done(["Exit"]) Success --> |No| Raise["Raise Exception"] Raise --> Retry["ARQ handles retries"] Retry --> Done

Diagram sources - webhook.py - webhook.py

Section sources - webhook.py

Webhook Service

The Webhook Service performs: - JSON serialization of the payload - Optional HMAC-SHA256 signing with a shared secret - HTTP POST using an async client with a fixed timeout - Structured logging for successes and failures - Returning a boolean outcome for the caller

Security and formatting: - Content-Type header set to application/json - Signature header name is standardized for merchant verification - Timeout configured to bound network latency

Section sources - webhook.py

Payload Structure and Event Categorization

The payload delivered upon confirmation includes: - payment_id: unique identifier - status: current status (confirmed) - address: destination address - amount: payment amount (as string) - chain: blockchain identifier - token_id: associated token identifier if applicable

Event categorization: - Payments move from pending to detected to confirmed during lifecycle - Webhooks are triggered only on confirmation transitions - Filtering occurs implicitly by selecting only confirmed payments

Section sources - scanner.py

Retry Logic and Exponential Backoff

  • The actor is configured with a finite number of retries
  • On failure, the actor raises an exception to signal failure to ARQ
  • ARQ’s default retry policy applies; explicit exponential backoff is not implemented in code
  • Consider adding jitter and backoff configuration for production hardening

Section sources - webhook.py

Dead Letter Queue (DLQ) Handling

  • The code does not configure a DLQ for failed tasks
  • Failed tasks remain in the queue until max_retries is exhausted
  • Production deployments should consider configuring a DLQ for failed messages

Section sources - webhook.py

Signature Verification and Security Measures

  • The service optionally signs payloads using HMAC-SHA256 with a shared secret
  • The header name for signatures is standardized for merchant-side verification
  • Merchant endpoints should verify the signature using the same algorithm and shared secret
  • Additional security measures:
  • Enforce HTTPS endpoints
  • Rotate secrets periodically
  • Limit exposed fields in payloads
  • Validate endpoint URLs before enqueueing tasks

Section sources - webhook.py - config.py

Authentication Methods

  • Shared secret-based HMAC signature for payload integrity
  • Optional endpoint authentication can be implemented by merchants using the provided signature
  • No built-in HTTP authentication (e.g., Basic/Digest) in the worker

Section sources - webhook.py

Rate Limiting and Endpoint Validation

  • Built-in rate limiting is not implemented in the worker
  • The service sets a fixed timeout to avoid long blocking
  • Endpoint validation:
  • Verify webhook_url and webhook_secret are present before triggering tasks
  • Consider validating endpoint reachability or using circuit breaker patterns at the service level

Section sources - config.py - webhook.py

Monitoring and Observability

  • Logging is used for success and failure events
  • Recommendations:
  • Add metrics for delivery success rate, latency, and retry counts
  • Instrument the actor and service with counters and histograms
  • Correlate logs with payment_id for traceability
  • Monitor queue depth and backlog

Section sources - webhook.py

Resubmission Workflows and Manual Intervention

  • Resubmission:
  • Re-enqueue tasks for specific payment_id/status combinations
  • Use administrative endpoints or scripts to trigger resubmissions
  • Manual intervention:
  • Inspect logs and database state
  • Manually post payloads to endpoints for testing
  • Adjust configuration (URL/secret) and restart workers if needed

[No sources needed since this section provides general guidance]

Dependency Analysis

The webhook system depends on: - ARQ with Redis for task queuing and retries - Async HTTP client for outbound requests - Configuration for webhook URL and secret - Database for payment state transitions

graph LR Settings["Settings
webhook_url, webhook_secret"] --> Actor["send_webhook_task"] Actor --> Service["WebhookService"] Service --> HTTP["httpx.AsyncClient"] Broker["Redis Broker"] --> Actor Scanner["ScannerService"] --> Actor DB["Database"] --> Scanner

Diagram sources - config.py - webhook.py - webhook.py - init.py - scanner.py

Section sources - config.py - init.py

Performance Considerations

  • Asynchronous I/O reduces blocking during outbound HTTP calls
  • Fixed timeout prevents long-tail latency issues
  • Consider:
  • Configurable timeouts and retry backoff
  • Connection pooling and keep-alive
  • Batching multiple webhooks per payment if appropriate
  • Circuit breakers to protect downstream endpoints

[No sources needed since this section provides general guidance]

Troubleshooting Guide

Common issues and resolutions: - Webhook delivery fails: - Check merchant endpoint availability and TLS configuration - Verify webhook_url and webhook_secret in settings - Review logs for HTTP status and exceptions - Signature mismatch: - Confirm the merchant endpoint uses HMAC-SHA256 with the correct secret - Ensure payload serialization matches exactly (order, encoding) - Retries exhausted: - Inspect task queue backlog and broker connectivity - Consider increasing max_retries or implementing exponential backoff - Endpoint validation: - Validate URLs before enqueueing tasks - Use health checks for endpoints

Section sources - webhook.py - webhook.py - config.py

Conclusion

The Webhook Worker provides a robust, asynchronous delivery mechanism for payment status notifications. It leverages ARQ for reliability, supports optional payload signing for security, and integrates cleanly with the payment confirmation pipeline. Production deployments should consider adding configurable retry backoff, DLQ handling, rate limiting, and comprehensive observability to ensure high delivery success rates and operability.

Appendices

Webhook Payload Reference

  • Fields: payment_id, status, address, amount, chain, token_id (optional)
  • Content-Type: application/json
  • Signature header (when secret configured): standardized name for HMAC-SHA256

Section sources - scanner.py - webhook.py

Running Workers

  • Start the ARQ workers to process webhook tasks alongside listener and sweeper.

Section sources - README.md