During a scheduled rollout of a backend API service, the influx of frontend client reconnection requests led to a sharp increase in load on some backend services, causing room data to fail to load. We were able to slowly resolve the issue by increasing capacity to catch up with the incoming requests.
We've identified some follow-up work to improve the way Growlink apps communicate with our servers and make these types of regular updates more seamless. While frontend rooms were intermittently unavailable, no services used by controllers at customer sites were impacted, and those controllers were operating normally.
Resolved
During a scheduled rollout of a backend API service, the influx of frontend client reconnection requests led to a sharp increase in load on some backend services, causing room data to fail to load. We were able to slowly resolve the issue by increasing capacity to catch up with the incoming requests.
We've identified some follow-up work to improve the way Growlink apps communicate with our servers and make these types of regular updates more seamless. While frontend rooms were intermittently unavailable, no services used by controllers at customer sites were impacted, and those controllers were operating normally.