Postmortem -
Read details
Dec 17, 17:55 UTC
Resolved -
The incident is resolved.
Dec 16, 03:32 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
Dec 16, 03:06 UTC
Investigating -
We experience more DB lateny issues.
Investigating, looking for the source of the latency.
Dec 16, 01:15 UTC
Update -
We experience more DB lateny issues.
Dec 16, 01:11 UTC
Update -
The Upload API is still experiencing degradation.
You may see longer-than-usual upload processing times, as well as intermittent 5XX responses.
We’re actively working to resolve the underlying issue and will continue to share updates.
Dec 16, 00:30 UTC
Update -
We are continuing to monitor for any further issues.
Dec 16, 00:10 UTC
Update -
DB performance is recovered along with REST API and Dashboard.
Dec 15, 23:15 UTC
Update -
Extensive processing of Upload API backlog affects DB and therefore latency of REST API and Dashboard.
Dec 15, 22:45 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
The upload "finish" wait times are still outside of norm.
Dec 15, 21:59 UTC
Update -
Currently there is a very big task queue accrued during initial production issue phase.
We've fixed original issue and are working through this queue.
All uploaded files should be accounted for eventually.
For end users it still looks like uploads are not working.
Affected endpoints:
- `/from_url/`
- `/base/`
Dec 15, 20:55 UTC
Update -
We are continuing to work on a fix for this issue.
Dec 15, 20:29 UTC
Identified -
The issue has been identified and a fix is being implemented.
Dec 15, 19:24 UTC
Update -
We are continuing to investigate this issue.
Dec 15, 18:50 UTC
Update -
We're experiencing performance degradation for Upload API.
Uploads are finished with significantly higher latency.
Dec 15, 18:07 UTC
Investigating -
We are currently investigating this issue.
Dec 15, 18:05 UTC