What happened

• There were two periods where many Arbitrum One transactions failed to be submitted:
  1. Between 04:27 PST and 06:00 PST.
  2. Between 09:59 PST and 11:04 PST.

Why this happened

• Severe network congesion led to extreme gas costs — about 2000 times more costly than usual.
• Our workers used hardcoded gas limits of 20M, meanwhile average transactions were consuming up to 1B gas since the network uses dynamic gas accounting. Therefore, our worker transactions were rejected due to insufficient gas.

Estimated costs

• Automatos was down on Arbitrum One for about 2 hours and 38 minutes.
• Adrastia Oracles suffered minor staleness:
  1. Between 04:27 PST and 06:00 PST: Up to about 0.5% price inaccuracy.
  2. Between 09:59 PST and 11:04 PST: Up to about 0.1% price inaccuracy.

How to prevent this in the future

• We'll use dynamic gas estimations on Arbitrum One.
• We'll improve our error detection and escalation processes to ensure high response times to incidents like these.