Pingcord - Event processing delays – Incident details

CloudFlare Infrastructure experiencing degraded performance

Event processing delays

Resolved
Major outage
Started 4 months agoLasted about 4 hours

Affected

Main Services

Major outage from 3:30 AM to 7:15 AM, Degraded performance from 6:45 AM to 7:15 AM

IRIS Ingest

Major outage from 3:30 AM to 7:15 AM, Degraded performance from 6:45 AM to 7:15 AM

Posting Service

Major outage from 3:30 AM to 7:15 AM, Degraded performance from 6:45 AM to 7:15 AM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. Our investigation revealed an issue with a CloudFlare API led to internal service discovery problems which caused services to stop talking to each other. This has now been resolved by eliminating sole reliance on this API. Thank you for your patience.

  • Monitoring
    UTC
    Monitoring

    We have identified the cause of the issue and are in the process of recovering pings. Pings will start to come through over the next hour or so. Any pings where the event occurred more than 2 hours ago will have their notifications silenced but will still be posted. Thank you for your patience while this was resolved.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.