Koala - Delayed delivery across all automations and search experience – Incident details

All systems operational

Delayed delivery across all automations and search experience

Resolved
Degraded performance
Started 6 months agoLasted about 10 hours

Affected

app

Degraded performance from 3:00 PM to 12:28 AM, Operational from 12:28 AM to 12:29 AM

api

Degraded performance from 3:00 PM to 12:28 AM, Operational from 12:28 AM to 12:29 AM

Updates
  • Resolved
    Resolved

    We've fully processed all automation and indexing backlog across our systems and all customer workspaces. We've implemented several safeguards to prevent this situation and dramatically speed up processing under heavy load (5-10x backlog)

  • Monitoring
    Monitoring

    We've implemented several fixes to speed up processing and start working through the backlog.

  • Identified
    Identified

    We've detected several bottlenecks with our background processing that is contributing to the delayed automations.

  • Investigating
    Investigating

    We are currently investigating this incident after a 2-3x increase in processing.