Previous incidents

What happened

• There were two periods where many Arbitrum One transactions failed to be submitted:
  1. Between 04:27 PST and 06:00 PST.
  2. Between 09:59 PST and 11:04 PST.

Why this happened

• Severe network congesion led to extreme gas costs — about 2000 times more costly than usual.
• Our workers used hardcoded gas limits of 20M, meanwhile average transactions were consuming up to 1B gas since the network uses dynamic gas accounting. Therefore, our worker transactions were ...

What happened

1. At 9:50PM PST, all of our worker nodes lost blockchain connectivity.
2. At 9:54PM PST, we were notified of downtime and immediately responded.
3. At 10:03PM PST, we changed RPC configurations to bring Automatos back online. Incident resolved.

Why this happened

1. All of the RPC providers we use with our Automatos workers went offline.

Estimated costs

1. Automatos was down on Blast for about 13 minutes.
2. Automatos did not miss any work that needed to be ...