Pingcord - Notice history

Pingcord is beginning the recovery process, and pings should slowly start to come through. Pingcord will suppress mentions for events that have happened > 2 hours ago, to prevent annoying users. Thank you for your patience while we work through these issues.

We are monitoring the situation closely and will provide updates soon.

This issue has been resolved.

Following a fix by Google Engineering last week, all YouTube trackers have been re-established over the last few days.

We will continue to run the backup system while monitoring traffic to ensure we are seeing a similar level of activity from before the issues started. Once we're happy, we'll switch off the backup system.

Update: We apologise that the backup system had malfunctioned since earlier today. The backup system's issue has now been resolved and pings are recovering.

We are *still* waiting on updates from Google Engineering regarding the main issue that has caused us to rely on the backup system.

Original issue:

We have identified an issue relating to trackers for YouTube channels. The issue is caused by an internal Google service. Google Engineering are aware of the issue and are actively looking into it.

Existing channels tracked by Pingcord have now been moved to a backup system until the issue is resolved. Newly-added channels are also now being tracked by the backup system. Pings may be delayed by up to 10 minutes as a result of this.

Update: We apologise that the backup system had malfunctioned since earlier today. The backup system's issue has now been resolved and pings are recovering.

We are *still* waiting on updates from Google Engineering regarding the main issue that has caused us to rely on the backup system.

Original issue:

We have identified an issue relating to trackers for YouTube channels. The issue is caused by an internal Google service. Google Engineering are aware of the issue and are actively looking into it.

Existing channels tracked by Pingcord have now been moved to a backup system until the issue is resolved. Newly-added channels are also now being tracked by the backup system. Pings may be delayed by up to 10 minutes as a result of this.

We have identified an issue relating to trackers for YouTube channels. The issue is caused by an internal Google service. Google Engineering are aware of the issue and are actively looking into it.

Existing channels tracked by Pingcord have now been moved to a backup system until the issue is resolved. Newly-added channels are also now being tracked by the backup system. Pings may be delayed by up to 10 minutes as a result of this.

We will provide further updates when we receive them from Google.

Current estimated impact: 5% of YouTube channels affected

While we await updates from Google, we are continuing to identify and setup mitigations to reduce any impact to notifications. Any affected existing YouTube channel trackers are being moved to a temporary system. Once this is complete, we will be able to support new YouTube channel registrations.

We are closing this incident but will continue to monitor ongoing. Our incident monitoring stack will also be reconfigured to generate an urgent alert to the team for the specific issue that occurred this evening.

A critical internal service handling communication to Discord restarted due to a fault. The restart was successful but it did not report to our other services that it was back online. An internal incident was raised automatically by our monitoring stack but was misclassified as a minor severity issue.

We have restored service currently. Pingcord is now recovering pings and data from prior to the incident.