Previous incidents
September 2024
No incidents reported
August 2024
No incidents reported
July 2024
Jul 26, 2024
1 incident
Automatos Mode Downtime Post Mortem
Resolved Jul 26 at 11:40pm UTC
What happened
- There was a brief 7-minute period where Mode transactions failed to be submitted:
- Between 16:35 PST and 16:43 PST, July 25, 2024.
Why this happened
- Some of our RPC providers went down.
- Other RPC providers lagged behind the latest block.
Estimated costs
- Automatos was down on Mode for about 7 minutes.
- Adrastia Oracles suffered minor staleness:
- Ionic's utilization and error oracle had one update that was delayed by up to 7 minutes.
How to prevent ...
Jul 13, 2024
1 incident
Web App Rootstock Connectivity Downtime Post Mortem
Resolved Jul 13 at 11:59pm UTC
What happened
- At 11:32 PM PST, all of our web app's Rootstock RPC providers become unavailable.
- At 11:37 PM PST, we were notified of downtime and responded immediately.
- At 11:47PM PST, our fix went live.
Why this happened
- Our web app was only using 2 RPC providers for Rootstock, and they both went down.
Estimated costs
- Users were unable to view Automatos worker balances for Rootstock via our web app for about 15 minutes. This did not negatively affect anyone.