Pingcord - Ongoing database issues – Incident details

All systems operational

Ongoing database issues

Resolved
Under maintenance
Started about 1 year agoLasted about 11 hours

Affected

Main Services

Degraded performance from 5:18 PM to 6:50 PM, Under maintenance from 6:50 PM to 7:04 PM, Operational from 7:04 PM to 7:05 PM, Major outage from 7:05 PM to 7:06 PM, Operational from 7:06 PM to 4:30 AM

IRIS Ingest

Degraded performance from 5:18 PM to 6:50 PM, Under maintenance from 6:50 PM to 7:04 PM, Operational from 7:04 PM to 7:05 PM, Major outage from 7:05 PM to 7:06 PM, Operational from 7:06 PM to 4:30 AM

Posting Service

Degraded performance from 5:18 PM to 6:50 PM, Under maintenance from 6:50 PM to 4:30 AM

Integrations

Degraded performance from 5:18 PM to 6:50 PM, Under maintenance from 6:50 PM to 4:30 AM

Instagram

Degraded performance from 5:18 PM to 6:50 PM, Under maintenance from 6:50 PM to 4:30 AM

Twitch

Degraded performance from 5:18 PM to 6:50 PM, Under maintenance from 6:50 PM to 4:30 AM

Updates
  • Resolved
    Resolved

    As of now the service is restored. We will continue to monitor the service throughout the weekend.

  • Monitoring
    Monitoring

    Emergency database maintenance has been completed successfully.

    Unfortunately, due to the nature of the database crash, any changes made to pre-existing integrations will be rolled back to this morning. New servers that joined after the crash should have their integrations preserved.

    Thank you for your continued patience.

  • Update
    Update

    We have now started the database backup process. During this time, we are reconciling any changes made from this morning's outage. This may take some time to process all of the records.

    We will attempt to save any data received from Twitch, YouTube, etc. to recover pings after the downtime.

    Please keep an eye out here, or join our Discord server at https://discord.gg/QKwDHGy for further information.

  • Investigating
    Investigating

    We are currently investigating issues related to our outage earlier this morning. It appears our database cluster has not recovered properly and as a result some integrations may be missing.

    We are in the processing of confirming if this is the case. If so, we will reset our servers using a backup from before the outage this morning. To do this, we will take Pingcord offline temporarily to carry this out. We plan to do this by 7pm UTC (around 2 hours from this message at 8pm UK time, 3pm ET, 12 noon PT). The downtime will last a few hours but we will try and queue any incoming data and send the pings out afterwards.

    Thank you for your continued patience while we sort out these issues.