According to an recent publication by Coinbase, the exchange experienced yet another brief outage on last Monday, June 1st.
The issue impacted coinbase.com, pro.coinbase.com, and Coinbase mobile applications. Trading through API remained functional throughout the outage period. The Coinbase team able to quickly determine the root cause and restore the services.
According to the details, on Monday, June 1st, at around 16:05 PDT, the price of Bitcoin [BTC] reached $10,000. In connection with the rising price, the platform experienced a 5 times more traffic spike for over 4 minutes. Coinbase said that its autoscaling system was unable to keep pace with this sudden spike in traffic.
This affected a number of Coinbase’s internal services, increasing latency between services. According to Coinbase:
“This led to process saturation of the web servers responsible for our API, where the number of incoming requests was greater than the number of listening processes, causing the requests to either be queued and timeout, or fail immediately. Our request error rate spiked to 50%, causing customers to experience errors when interacting with coinbase.com and our mobile apps.”
The health check is also served by the web servers responsible for Coinbase API. Since they were affected by the problem, this caused health check to mark some instances as unhealthy and “taken out of the load balancer, further exacerbating this issue.”
To mitigate this process saturation of web servers responsible for the Coinbase API, the team redeployed the API at 16:20 PDT to increase the machines serving the traffic. The publication further reads:
“Once this deploy completed, the previous deployed instances were taken out of rotation, leading to another 2 minute outage due to instances saturating and being marked unhealthy. This was handled automatically by our autoscaling.”
In response to this outage, Coinbase has “fixed the health endpoint to ensure that saturated instances don’t get taken out of rotation.” The technical teams are working on some pre-scaling and caching mechanisms to reduce the impact of price-related traffic spikes in future. Coinbase said that they were also working on their API deployment process to avoid autoscaling issues.
As Crypto Economy reported, San Francisco-based crypto exchange also suffered outages on April 29th and May 9th, 2020. These outages also involved problem with Coinbase API.
Coinbase is a prominent name in crypto exchanges list. According to the exchange, the aim of making these details public is to image Coinbase as the most trusted platform in crypto space.
