Date and time of incident
10. December 2025, 14:00 CET - 15. December 2025, 01:00 CET
Summary
During the above period, OnlineFundraising experienced significant delays in outgoing webhooks. The cause was an anomaly large spike in Payment API usage, which resulted in late processing of webhook events and in addition impacted the cadence of recurring payment collections.
Impact
- Outgoing webhooks were delayed by several hours to days.
- Recurring payment collections were processed later than scheduled, and the backlog was not fully resolved until 17. December.
- Importantly, the incident did not affect our ability to register new one-time payments or new payment agreements.
Root cause
The incident was caused by an unusual combination of overlapping events:
- Delivery of December status from Betalingsservice (received 10. December 22:40 CET) and an early January collection (scheduled for 18. December 09:00 CET).
- Multiple effective SMS push campaigns increased overall activity.
- A major customer implemented an inefficient merge policy and performed database cleanup with parallel updates in OnlineFundraising. This resulted in old Contacts (including agreements and payment history) being merged into new Contacts, causing a more than tenfold increase in activity compared to the usual December/Christmas peak.
Timeline
- 10. December 14:00 CET: Incident begins; webhook queue starts to grow.
- 10. December 22:40 CET: Delivery from Betalingsservice.
- 15. December 01:00 CET: System update deployed to address webhook bottleneck and prioritize payment agreement collections.
- 17. December: Backlog in payment collections fully resolved.
Resolution
A system update was deployed during the night of 15. December to:
- Alleviate the bottleneck in outgoing webhooks.
- Prioritize the processing of recurring payment agreements.
Next steps
- Review and optimize webhook processing to better handle peak loads.
- Collaborate with key customers to improve data merge policies and avoid similar spikes.
- Enhance monitoring and alerting for webhook queue growth and payment collection delays.