The normal system for running background processes has been restored. Our workaround has been successfully running these processes in the interim. The AWS infrastructure which hosts Upscribe is still reporting degraded performance. We are continuing to monitor Amazon's progress on mitigating their outage, but do not see any significant impact to Upscribe's customer or admin interfaces at this time.
Upscribe's internal monitoring and reporting systems are still impaired, but showing signs of improvement.
Their latest statement as of this status update is as follows:
Latest Update (6:23 PM PST): We’d like to provide an update on the issue affecting the Kinesis Data Streams API, and other dependent services, within the US-EAST-1 Region. We have now fully mitigated the impact to the subsystem within Kinesis that is responsible for the processing of incoming requests and are no longer seeing increased error rates or latencies. However, we are not yet taking the full traffic load and are working to relax request throttles on the service. Over the next few hours we expect to relax these throttles to previous levels. We expect customers to begin seeing recovery as these throttles are relaxed over this timeframe.
CloudWatch metrics remain delayed in the US-EAST-1 Region. Once we have restored the throttles for Kinesis to previous levels, we will be restoring CloudWatch metrics functionality. We expect to see recovery of CloudWatch metrics at that stage for new incoming metrics, but the backlog of metrics may take additional time to populate.
We will continue to keep you updated on our progress.
We have implemented a work around to run background processes. Operations seem normal.
Internal monitoring and log systems are still impacted.
AWS team has posted this message:
"12:15 PM PST: We continue to work towards recovery of the issue affecting the Kinesis Data Streams API in the US-EAST-1 Region. We also continue to see an improvement in error rates for Kinesis and several affected services, but expect full recovery to still take up to a few hours.
The issue continues to also affect other services, or parts of these services, that utilize Kinesis Data Streams within their workflows. While features of multiple services are impacted, some services have seen broader impact and service-specific impact details are below. We continue to work towards full recovery."
Posted Nov 25, 2020 - 12:27 PST
Identified
We are investigating the issue of background processes not running. These processes include subscription creation, email sending, webhook sending, and others. These are tied to issues with Amazon Web Service which is currently experiencing a substantial outage. See https://status.aws.amazon.com/ and https://i.imgur.com/CTQ7O0T.png
We are developing a work around.
This AWS outage is also causing inaccurate metrics on our status page.