Resolved -
This incident has been resolved. Our async processing uses GCP "spot" instances, and GCP unexpectedly started denying the creation of these spot instances in our kubernetes cluster. We have now created non-spot instances that will allow us to serve async tasks as normal, and have both spot and non-spot instances in place for future
Jul 16, 18:55 UTC
Monitoring -
New node pools are up & running, tasks are running again. We are recovering.
Jul 16, 18:44 UTC
Identified -
GCP is denying "spot instances" for our async processing kubernetes cluster. This means that no async tasks are being processed. We are creating a new node pool that does not rely on spot instances to get us back up & running
Jul 16, 18:41 UTC
Investigating -
We are having issues with GCP provisioning compute instances for async processing. Async processing is impacted
Jul 16, 18:38 UTC