Some paginated queries in Forge hosted storage kept repeating the last page
Incident
Report for Compass
Resolved
We have identified and resolved a problem with Forge hosted storage, where some paginated queries kept repeating the last page. The incident was detected by our internal monitoring and was resolved quickly after detection by reverting the deployment.
Activating changes recently made to the query cursors for paginated queries introduced a bug that impacted some apps. A small number of requests were impacted over a 16-minute window, while the incident lasted.
Timeline:
- 25/Mar/24 10:08 p.m. UTC - Impact started, when the changes were deployed to production
- 25/Mar/24 10:09 p.m. UTC - Incident was detected
- 25/Mar/24 10:24 p.m. UTC - Incident was resolved and impact ended
The impact of this incident has been completely mitigated and our monitoring tools confirm that query operations are back to the pre-incident behaviour. We have also resolved the underlying bug and deployed the fix to production, completely eliminating the cause of this incident.
We apologise for any inconvenience this may have caused to our customers and our developer community.
Activating changes recently made to the query cursors for paginated queries introduced a bug that impacted some apps. A small number of requests were impacted over a 16-minute window, while the incident lasted.
Timeline:
- 25/Mar/24 10:08 p.m. UTC - Impact started, when the changes were deployed to production
- 25/Mar/24 10:09 p.m. UTC - Incident was detected
- 25/Mar/24 10:24 p.m. UTC - Incident was resolved and impact ended
The impact of this incident has been completely mitigated and our monitoring tools confirm that query operations are back to the pre-incident behaviour. We have also resolved the underlying bug and deployed the fix to production, completely eliminating the cause of this incident.
We apologise for any inconvenience this may have caused to our customers and our developer community.