Pingcord - Event processing delays – Incident details

Event processing delays

ResolvedMajor outage3 hours 45 minutes

Affected

Main Services

Major outage from 3:30 AM to 7:15 AM, Degraded performance from 6:45 AM to 7:15 AM

IRIS Ingest

Major outage from 3:30 AM to 7:15 AM, Degraded performance from 6:45 AM to 7:15 AM

Posting Service

Major outage from 3:30 AM to 7:15 AM, Degraded performance from 6:45 AM to 7:15 AM

  • ResolvedUTC

    This incident has been resolved. Our investigation revealed an issue with a CloudFlare API led to internal service discovery problems which caused services to stop talking to each other. This has now been resolved by eliminating sole reliance on this API. Thank you for your patience.

  • MonitoringUTC

    We have identified the cause of the issue and are in the process of recovering pings. Pings will start to come through over the next hour or so. Any pings where the event occurred more than 2 hours ago will have their notifications silenced but will still be posted. Thank you for your patience while this was resolved.

  • InvestigatingUTC
    We are currently investigating this incident.