AWS has resolved the API Gateway issue and we have switched back to using the same region for enrichment. We are at last starting to see the event pipelines make headway on the backlog of events. We have spun up more machines to help us catch up with backlog.
Posted Sep 28, 2022 - 14:33 PDT
Update
AWS still has API Gateway for us-west-2 marked as disrupted. We are now using an API Gateway in another region, and the event backlog is recovering.
Posted Sep 28, 2022 - 13:36 PDT
Update
AWS added API gateway to list of affected areas. This means even more delayed data enrichments. We are monitoring the situation. We are receiving data sent to us, AWS latency and errors are causing delays in processing.
Posted Sep 28, 2022 - 10:23 PDT
Update
AWS us-west-2 is reporting increased API latencies leading to slowdown.
Posted Sep 28, 2022 - 10:04 PDT
Monitoring
Slower than expected scale up to deal with increased traffic.
Posted Sep 28, 2022 - 09:47 PDT
This incident affected: Data Pipeline (SnowcatCloud Data Pipeline (US)).