Duration: 3 hours and 45 minutes

Affected Components: IRIS Ingest, Posting Service, , Main Services →

Apr 4, 03:30:00 GMT+0
Investigating - We are currently investigating this incident..

Apr 4, 06:45:00 GMT+0
Monitoring - We have identified the cause of the issue and are in the process of recovering pings. Pings will start to come through over the next hour or so. Any pings where the event occurred more than 2 hours ago will have their notifications silenced but will still be posted. Thank you for your patience while this was resolved..

Apr 4, 07:15:00 GMT+0
Resolved - This incident has been resolved. Our investigation revealed an issue with a CloudFlare API led to internal service discovery problems which caused services to stop talking to each other. This has now been resolved by eliminating sole reliance on this API. Thank you for your patience..

Duration: 15 minutes

Affected Components: Website, IRIS Ingest

Mar 20, 04:19:42 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Mar 20, 04:34:48 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 9 minutes

Affected Components: Website, IRIS Ingest

Mar 15, 04:05:50 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Mar 15, 04:15:19 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 14 minutes

Affected Components: Website, IRIS Ingest

Feb 13, 03:50:45 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Feb 13, 04:04:53 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 14 minutes

Affected Components: Website, IRIS Ingest

Dec 27, 03:35:20 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Dec 27, 03:49:28 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 10 minutes

Affected Components: IRIS Ingest, Posting Service

Dec 9, 17:25:29 GMT+0
Investigating - We are currently investigating this incident..

Dec 9, 17:35:31 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 29 minutes

Affected Components: Website, IRIS Ingest

Oct 4, 22:21:23 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Oct 5, 00:50:18 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 44 minutes

Affected Components: Website, IRIS Ingest

Oct 4, 22:05:26 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Oct 4, 21:21:53 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Duration: 14 minutes

Affected Components: Website, IRIS Ingest

Jun 23, 03:20:11 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Jun 23, 03:34:06 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 3 hours and 2 minutes

Affected Components: YouTube

Jun 12, 18:41:22 GMT+0
Identified - We are seeing multiple reports from other apps & services that Google Cloud APIs (incl. YouTube API) are experiencing an outage at this time. We will save YouTube events and will hold them in our event queue until Google resolve their outage..

Jun 12, 18:34:48 GMT+0
Investigating - The YouTube API is returning a server error which is preventing Pingcord from processing YouTube events. Please note this is not an issue with Pingcord and we are attempting to identify a workaround until YouTube API services are working again..

Jun 12, 20:31:34 GMT+0
Monitoring - The YouTube API is currently working as expected however Google services are still experiencing widespread downtime issues. Pingcord is continuing to process YouTube events at this time but we will monitor the situation closely in case of any further downtime from YouTube..

Jun 12, 21:37:05 GMT+0
Resolved - This incident has been resolved..

Duration: 19 minutes

Affected Components: IRIS Ingest

Jun 5, 20:38:31 GMT+0
Investigating - We are currently reduced performance in the event processing pipeline. Pingcord is still online however pings may be delayed by up to 30 minutes..

Jun 5, 20:58:00 GMT+0
Resolved - Event processing has returned to normal service without intervention. We will monitor the situation closely..

Duration: 4 hours and 9 minutes

Affected Components: Website, TikTok, IRIS Ingest

Jun 3, 13:50:13 GMT+0
Identified - We have resolved the root cause of the problem and are bringing Pingcord services back online..

Jun 3, 14:12:22 GMT+0
Monitoring - Pingcord is back online, and we are currently monitoring the situation. TikTok pings will remain offline for the time being to ensure full system stability..

Jun 3, 15:50:57 GMT+0
Resolved - This incident has been resolved. We are monitoring the service closely and will be investigating the root cause leading to the downtime. Thank you for your patience!.

Jun 3, 11:41:33 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Jun 3, 15:14:46 GMT+0
Monitoring - TikTok services are now being brought back online. Thank you for your patience..

Duration: 7 hours and 16 minutes

Affected Components: Posting Service, IRIS Ingest

Apr 12, 10:12:58 GMT+0
Identified - The issue has been identified and events are now processing normally. Events will be cleared from our queue system within the next 20 minutes. Thank you for your patience..

Apr 12, 03:26:00 GMT+0
Investigating - We are investigating this issue..

Apr 12, 10:42:05 GMT+0
Resolved - The event processing system has returned to normal. We are continuing to monitor the situation closely. We will investigate the root cause of this problem shortly..

Duration: 2 days, 7 hours and 36 minutes

Affected Components: Facebook Pages

Apr 8, 22:36:29 GMT+0
Monitoring - The issue has been resolved and we are now receiving events from Facebook Pages as normal. Pingcord will attempt to recover any missed pings during the down period though some may be missed unfortunately. We are monitoring the situation closely for any further changes. This issue is a recurrence of a previous issue in March. We are continuing discussion with Meta to prevent this issue from happening in the future. Thank you for your patience!.

Apr 9, 00:38:00 GMT+0
Resolved - This incident has been resolved..

Apr 6, 17:01:51 GMT+0
Identified - We have identified a recurrence of an issue last month with Facebook Pages connections. Instagram remains unaffected. We are working with Meta to resolve these issues promptly..

Duration: 11 hours and 31 minutes

Affected Components: Facebook Pages

Mar 14, 18:28:49 GMT+0
Investigating - We are currently investigating this incident..

Mar 15, 06:00:00 GMT+0
Resolved - This incident has been resolved..

Mar 15, 00:29:21 GMT+0
Identified - We have identified the problem and are working with Meta to resolve a connection issue with our application. Thank you for your patience while we work on resolving this issue..

Mar 15, 05:02:48 GMT+0
Monitoring - The problem has been resolved and we are now receiving events from Facebook as usual. We will monitor the situation closely. Thank you for your patience!.

Duration: 10 hours and 1 minute

Affected Components: Reddit

Jan 22, 20:00:00 GMT+0
Identified - The cause of the issue has been found and work is underway to resolve the problem..

Jan 23, 00:18:50 GMT+0
Monitoring - The issue has been resolved and we are monitoring closely for any recurrences..

Jan 22, 15:00:00 GMT+0
Investigating - We are currently investigating an issue with Reddit pings from earlier today..

Jan 23, 01:01:00 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours and 27 minutes

Affected Components: Posting Service, IRIS Ingest

Jan 1, 04:15:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 1, 03:57:52 GMT+0
Monitoring - Service has been restored and our system is now processing the backlog of events. Pings should not be missed. Pingcord will post them unless they are related to livestreams which have already ended. We are monitoring the situation closely, thank you for your patience and happy new year!.

Jan 1, 02:03:00 GMT+0
Investigating - The Pingcord service is experience severe delays to notification processing. We are investigating the root cause of the issue and will update this page with further updates..

Jan 1, 08:30:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 11 hours and 14 minutes

Affected Components: Facebook Pages, Instagram

May 14, 01:01:00 GMT+0
Resolved - This incident has been resolved..

May 12, 13:46:43 GMT+0
Investigating - We are currently investigating this incident..

May 13, 21:01:59 GMT+0
Monitoring - We have resolved the issue with our access to Meta's APIs. Facebook and Instagram integrations should begin to work again as usual. We will monitor the situation closely to ensure it has been resolved..

Duration: 10 hours and 12 minutes

Affected Components: YouTube

Apr 16, 03:40:15 GMT+0
Identified - We have identified an issue with tracker connections to YouTube for some channels. We have deployed a fix and are currently renewing all trackers as a precautionary measure. This process will take a few hours to get through all of the trackers. Once your channel's tracker is fixed, YouTube may send us any queued events which will cause Pingcord to post the alerts. However, we appreciate you may have already posted a message yourself so Pingcord will "suppress" the @ mention tags for any video uploads over 3 hours ago. This means they will still appear in Discord like a mention but it has not notified your users..

Apr 16, 13:52:00 GMT+0
Resolved - We are pleased to confirm all trackers have been renewed. Please report any issues as usual in our support server. Thank you for your continued patience!.

Duration: 11 hours and 12 minutes

Affected Components: TikTok, Timer Alerts, Reddit, Facebook Pages, Mastodon, Kick.com, RSS Feeds, Instagram, Twitch, Spotify, YouTube, Posting Service, IRIS Ingest, Website, X / Twitter

Apr 12, 17:18:04 GMT+0
Investigating - We are currently investigating issues related to our outage earlier this morning. It appears our database cluster has not recovered properly and as a result some integrations may be missing. We are in the processing of confirming if this is the case. If so, we will reset our servers using a backup from before the outage this morning. To do this, we will take Pingcord offline temporarily to carry this out. We plan to do this by 7pm UTC (around 2 hours from this message at 8pm UK time, 3pm ET, 12 noon PT). The downtime will last a few hours but we will try and queue any incoming data and send the pings out afterwards. Thank you for your continued patience while we sort out these issues..

Apr 13, 04:30:00 GMT+0
Resolved - As of now the service is restored. We will continue to monitor the service throughout the weekend..

Apr 13, 03:00:19 GMT+0
Monitoring - Emergency database maintenance has been completed successfully. Unfortunately, due to the nature of the database crash, any changes made to pre-existing integrations will be **rolled back** to this morning. New servers that joined after the crash _should_ have their integrations preserved. Thank you for your continued patience..

Apr 12, 19:00:00 GMT+0
Investigating - We have now started the database backup process. During this time, we are reconciling any changes made from this morning's outage. This may take some time to process all of the records. We will attempt to save any data received from Twitch, YouTube, etc. to recover pings after the downtime. Please keep an eye out here, or join our Discord server at <https://discord.gg/QKwDHGy> for further information..

Duration: 3 hours and 26 minutes

Affected Components: IRIS Ingest, Website

Apr 12, 06:45:46 GMT+0
Investigating - The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates..

Apr 12, 10:11:29 GMT+0
Resolved - The affected issues have been resolved. We will continue to monitor the services closely. We apologise for any inconvenience caused..

Apr 12, 07:10:04 GMT+0
Monitoring - Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened > 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues. We are monitoring the situation closely and will provide updates soon..

Duration: 3 hours and 47 minutes

Affected Components: Posting Service, IRIS Ingest

Jan 3, 19:00:00 GMT+0
Resolved - This incident has been resolved..

Jan 3, 15:13:19 GMT+0
Investigating - We are currently investigating an increased event delay rate..

Jan 3, 18:28:01 GMT+0
Monitoring - The root issue has been identified and events are now processing normally. We expect all events to be cleared from our queue system within the next few minutes. Thank you for your patience..

Duration: 1 day, 21 hours and 1 minute

Affected Components: IRIS Ingest, Website

Dec 29, 21:19:00 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Dec 29, 22:04:01 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Duration: 4 hours and 59 minutes

Affected Components: YouTube, IRIS Ingest

Oct 12, 01:40:57 GMT+0
Investigating - We are currently investigating an increased event error rate..

Oct 12, 06:40:00 GMT+0
Resolved - This issue has been resolved. Some pings may have been missed or delayed. The root cause of the issue was identified and we are working towards implementing a solution to prevent it happening again. We apologise for the inconvience..

Duration: 13 hours and 24 minutes

Affected Components: IRIS Ingest, Website

Aug 9, 10:06:39 GMT+0
Investigating - <p>The Pingcord website is currently down. We are investigating the root cause of the issue and will update this page with further updates.</p>.

Aug 9, 11:20:00 GMT+0
Resolved - <p>Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened &gt; 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.</p> <p>We are monitoring the situation closely and will provide updates soon.</p>.

Aug 10, 14:59:28 GMT+0
Resolved - A networking issue with our cloud server provider resulted in connection drops between our own services. The issue was resolved quickly but recurred later that day for a few hours. Since the second fix, networking has remained stable and we are not expecting any further impact. Please see: https://status.digitalocean.com/incidents/q4b09b022nmh for further information.