Previous incidents

August 2024
No incidents reported
July 2024
Jul 26, 2024
1 incident

Automatos Mode Downtime Post Mortem

Resolved Jul 26 at 11:40pm UTC

What happened

  • There was a brief 7-minute period where Mode transactions failed to be submitted:
    1. Between 16:35 PST and 16:43 PST, July 25, 2024.

Why this happened

  • Some of our RPC providers went down.
  • Other RPC providers lagged behind the latest block.

Estimated costs

  • Automatos was down on Mode for about 7 minutes.
  • Adrastia Oracles suffered minor staleness:
    1. Ionic's utilization and error oracle had one update that was delayed by up to 7 minutes.

How to prevent ...
Jul 13, 2024
1 incident

Web App Rootstock Connectivity Downtime Post Mortem

Resolved Jul 13 at 11:59pm UTC

What happened

  • At 11:32 PM PST, all of our web app's Rootstock RPC providers become unavailable.
  • At 11:37 PM PST, we were notified of downtime and responded immediately.
  • At 11:47PM PST, our fix went live.

Why this happened

  • Our web app was only using 2 RPC providers for Rootstock, and they both went down.

Estimated costs

  • Users were unable to view Automatos worker balances for Rootstock via our web app for about 15 minutes. This did not negatively affect anyone.

How to p...
June 2024
Jun 20, 2024
1 incident

Automatos Arbitrum One Downtime Post Mortem

Resolved Jun 20 at 09:52pm UTC

What happened

  • There were two periods where many Arbitrum One transactions failed to be submitted:
    1. Between 04:27 PST and 06:00 PST.
    2. Between 09:59 PST and 11:04 PST.

Why this happened

  • Severe network congesion led to extreme gas costs — about 2000 times more costly than usual.
  • Our workers used hardcoded gas limits of 20M, meanwhile average transactions were consuming up to 1B gas since the network uses dynamic gas accounting. Therefore, our worker transactions were ...