Uniswap Subgraph Halting Post-Mortem
Early this morning, The Graph’s hosted service hit an error that caused the Uniswap subgraph to get stuck. This impacted the data shown on http://uniswap.info and any other apps that query the Uniswap subgraph, but did not impact http://app.uniswap.org, the Uniswap trading pages. The issue was fixed around 6am PT but because the subgraph is so data intensive, it took several hours to re-sync our nodes and process blocks.
Once we recognized the issue, we communicated with developers in our Discord, and tweeted about the incident to ensure our community was informed.
We care greatly about our developers and we recognize that applications that query The Graph are handling valuable assets that must rely on accurate data. Today’s incident is another example of why decentralizing every layer of the stack is so critical and why we are focused on building and launching The Graph Network as soon as we can.
What Happened
The hosted service experienced two errors that led to delayed data in the Uniswap subgraph or no query responses.
Uniswap Subgraph Error
The Uniswap subgraph saw an inconsistency with Ethereum contract calls. This resulted in the Uniswap subgraph halting and data being behind for several hours until the nodes resynced and all blocks were processed.
When operating against the head block, contract calls by block number can lead to calling different blocks due to reorgs or Ethereum node inconsistencies. When inconsistent data is returned, Graph Node halts processing new blocks for the affected subgraph to favor safety over liveness.
10 Minute Outage
Additionally, between 8:53 - 9:03 am PST (10 minutes) the hosted service went down due to server overload - processing over 6K queries per second just before the incident. After our nodes resynced, the hosted service has continued to process between 4,000-6,000 queries per second, throughout the day.
How We’re Improving
Since our last outage when DeFi began eating the world, we have made significant improvements to our infrastructure and internal processes. However, we still have more work to do to ensure reliability of the hosted service until we launch The Graph Network later this year.
Leaning on 3rd Party Indexers
Now that we’ve started our testnet we have over 200 Indexers running an early version of the network. It’s going to take more time for us to stress test the network, conduct security audits, and prepare for mainnet launch, but we can start leaning on these Indexers to provide failover and backup support now. Over the coming weeks we’re going to start leaning on this diverse community of Indexers to help us increase Ethereum node diversity and maintain a higher quality of service.
Improving Robustness of Contract Calls
We have dedicated an engineer to implementing an improvement to Graph Node to make contract calls by block hash instead of block number. This will eliminate the possibility of subgraphs receiving inconsistent contract call results. Any node that supports the latest hard fork of Ethereum supports this.
Doubling Down on DevOps
As we ramp up towards mainnet launch of the network, we will be hiring a Site Reliability Engineer to support Graph Node infrastructure and supporting uptime of the hosted service.
The Graph’s users are spread around the world and we need more talent across timezones to monitor the hosted-service systems. We are seeking an experienced Site Reliability Engineer located in Europe that is eager to join The Graph on our mission of making open data easily accessible. Please apply here if you think you could be a good fit.
Onwards and Upwards
The Graph is focused on providing public infrastructure and a reliable indexing layer to enable decentralized and serverless applications.
As DeFi and Web3 usage continues to grow, so will the need for decentralized infrastructure to support these applications, making them unstoppable and eliminating single points of failure. The Graph Network will be a critical data layer for building new applications that is upheld by a network of Indexers and Curators who are aligned with the incentives of the network.
We are currently deep into Phase 1 of Mission Control, our incentivized testnet with over 200 Graph Nodes deployed, you can subscribe to the newsletter to stay up to date on the network. Learn more about The Graph, find live subgraphs in the Graph Explorer, or learn how to deploy a subgraph in our docs. You can also follow The Graph’s journey:
Website: https://thegraph.com
Blog: https://thegraph.com/blog/
Telegram: https://t.me/GraphProtocol
Discord: https://discord.com/invite/vtvv7FP
Linkedin: https://www.linkedin.com/company/thegraph/
Everest: https://everest.link
Developer Documentation: https://thegraph.com/docs/
Twitter: https://twitter.com/graphprotocol
GitHub: https://github.com/graphprotocol
Contact: [email protected]
