Incident Summary

On November 25, 2024, from 16:01 to 19:44 UTC, the “Uploading from 3rd Party Services” feature of our platform was unavailable. This feature allows users to upload files from social networks and storage services like Dropbox, Facebook, Google Drive, and Instagram.

Timeline

• 15:50 UTC – A server configuration update was initiated to streamline deployments in our Kubernetes environment.
• 16:01 UTC – The outage began as requests to the “Uploading from 3rd Party Services” feature failed to be processed.
• 19:00 UTC – Investigation revealed that the application networking was incorrectly configured.
• 19:15 UTC – A fix was implemented to re-configure the application networking.
• 19:44 UTC – The incident was fully resolved, and service was restored.

Root Cause

The outage was caused by a synchronization issue between repositories during a server configuration update. This misalignment between repositories, compounded by a lack of explicit configuration, led to the service outage.

Impact

For approximately 3 hours and 43 minutes, users were unable to upload files from third-party services.

Challenges During Resolution

This type of issue is difficult to catch in a staging environment because the traffic profile does not reflect production-level demand. Although the alert system worked as intended, delays in responding to the alert contributed to the resolution time.

Resolution

The application configuration was updated and service functionality was fully restored at 19:44 UTC.

Action Items

Short-term

• Improve feature status monitoring and alerting to detect outages faster.

Long-term

• Improve synchronization processes between repositories to avoid dependency misalignments.

‌

We sincerely apologize for the disruption this caused to our users. We take this incident seriously and are committed to implementing the above action items to prevent similar occurrences in the future.

Thank you for your patience and continued trust in our platform. If you have any questions or need further details, please don’t hesitate to reach out.