Debts Hub
2025
Centralized debts substrate gating Balady service issuance on outstanding-debt resolution. Replaced REST-based confirmations with RabbitMQ after resilience issues, eliminating that class of failure.
The problem
Balady’s service-issuance flow needs to verify a citizen has no outstanding debt before granting a licence. The original integration relied on REST callbacks from the payment-confirmation flow. Under load, callbacks occasionally did not reflect on the licence — a subtle, recurring incident class with painful manual recovery.
The architecture
A centralized debts substrate sitting between the licence-issuing services and the payments domain. Seven internal containers — eligibility evaluator, event consumer, snapshot store, scheduled-job reconciler, BI exporter, REST API, and admin UI.
After the production incident, drove the revision that replaced REST callbacks with RabbitMQ-based event consumption for the payment-confirmation flow, eliminating that entire failure class.
Trade-offs (ADRs)
- Scheduled-job vs real-time eligibility check — chose hybrid: real-time for normal operation, scheduled-job for self-healing reconciliation
- BI/ETL replication into the national payments BI database — separate pipeline avoids coupling the operational hub to BI deadlines
Outcome
Recurring incident class eliminated. The RabbitMQ-based substrate is now the contract for all consuming services in the Balady ecosystem.