Substreams-powered subgraphs
Reading time: 5 min
is a new framework for processing blockchain data, developed by StreamingFast for The Graph Network. A substreams modules can output entity changes, which are compatible with Subgraph entities. A subgraph can use such a Substreams module as a data source, bringing the indexing speed and additional data of Substreams to subgraph developers.
This cookbook requires , , and the latest version of Graph CLI (>=0.52.0):
npm install -g @graphprotocol/graph-cli
graph init --from-example substreams-powered-subgraph
A Substreams package is composed of types (defined as ), modules (written in Rust), and a substreams.yaml
file which references the types, and specifies how modules are triggered. , and check out and the for more examples.
The Substreams package in question detects contract deployments on Mainnet Ethereum, tracking the creation block and timestamp for all newly deployed contracts. To do this, there is a dedicated Contract
type in /proto/example.proto
():
syntax = "proto3";package example;message Contracts {repeated Contract contracts = 1;}message Contract {string address = 1;uint64 blockNumber = 2;string timestamp = 3;uint64 ordinal = 4;}
The core logic of the Substreams package is a map_contract
module in lib.rs
, which processes every block, filtering for Create calls which did not revert, returning Contracts
:
#[substreams::handlers::map]fn map_contract(block: eth::v2::Block) -> Result<Contracts, substreams::errors::Error> {let contracts = block.transactions().flat_map(|tx| {tx.calls.iter().filter(|call| !call.state_reverted).filter(|call| call.call_type == eth::v2::CallType::Create as i32).map(|call| Contract {address: format!("0x{}", Hex(&call.address)),block_number: block.number,timestamp: block.timestamp_seconds().to_string(),ordinal: tx.begin_ordinal,})}).collect();Ok(Contracts { contracts })}
A Substreams package can be used by a subgraph as long as it has a module which outputs compatible entity changes. The example Substreams package has an additional graph_out
module in lib.rs
which returns a substreams_entity_change::pb::entity::EntityChanges
output, which can be processed by Graph Node.
The substreams_entity_change
crate also has a dedicated Tables
function for simply generating entity changes (). The Entity Changes generated must be compatible with the schema.graphql
entities defined in the subgraph.graphql
of the corresponding subgraph.
#[substreams::handlers::map]pub fn graph_out(contracts: Contracts) -> Result<EntityChanges, substreams::errors::Error> {// hash map of name to a tablelet mut tables = Tables::new();for contract in contracts.contracts.into_iter() {tables.create_row("Contract", contract.address).set("timestamp", contract.timestamp).set("blockNumber", contract.block_number);}Ok(tables.to_entity_changes())}
These types and modules are pulled together in substreams.yaml
:
specVersion: v0.1.0package:name: 'substreams_test' # the name to be used in the .spkgversion: v1.0.1 # the version to use when creating the .spkgimports: # dependenciesentity: https://github.com/streamingfast/substreams-entity-change/releases/download/v0.2.1/substreams-entity-change-v0.2.1.spkgprotobuf: # specifies custom types for use by Substreams modulesfiles:- example.protoimportPaths:- ./protobinaries:default:type: wasm/rust-v1file: ./target/wasm32-unknown-unknown/release/substreams.wasmmodules: # specify modules with their inputs and outputs.- name: map_contractkind: mapinputs:- source: sf.ethereum.type.v2.Blockoutput:type: proto:test.Contracts- name: graph_outkind: mapinputs:- map: map_contractoutput:type: proto:substreams.entity.v1.EntityChanges # this type can be consumed by Graph Node
You can check the overall "flow" from a Block, to map_contract
to graph_out
by running substreams graph
:
To prepare this Substreams package for consumption by a subgraph, you must run the following commands:
yarn substreams:protogen # generates types in /src/pbyarn substreams:build # builds the substreamsyarn substreams:package # packages the substreams in a .spkg file# alternatively, yarn substreams:prepare calls all of the above commands
These scripts are defined in the package.json
file if you want to understand the underlying substreams commands
This generates a spkg
file based on the package name and version from substreams.yaml
. The spkg
file has all the information which Graph Node needs to ingest this Substreams package.
If you update the Substreams package, depending on the changes you make, you may need to run some or all of the above commands so that the spkg
is up to date.
Substreams-powered subgraphs introduce a new kind
of data source, "substreams". Such subgraphs can only have one data source.
This data source must specify the indexed network, the Substreams package (spkg
) as a relative file location, and the module within that Substreams package which produces subgraph-compatible entity changes (in this case map_entity_changes
, from the Substreams package above). The mapping is specified, but simply identifies the mapping kind ("substreams/graph-entities") and the apiVersion.
Currently, Subgraph Studio and The Graph Network support Substreams-powered subgraphs which index mainnet
(Mainnet Ethereum).
specVersion: 0.0.4description: Ethereum Contract Tracking Subgraph (powered by Substreams)repository: https://github.com/graphprotocol/graph-toolingschema:file: schema.graphqldataSources:- kind: substreamsname: substream_testnetwork: mainnetsource:package:moduleName: graph_outfile: substreams-test-v1.0.1.spkgmapping:kind: substreams/graph-entitiesapiVersion: 0.0.5
The subgraph.yaml
also references a schema file. The requirements for this file are unchanged, but the entities specified must be compatible with the entity changes produced by the Substreams module referenced in the subgraph.yaml
.
type Contract @entity {id: ID!"The timestamp when the contract was deployed"timestamp: String!"The block number of the contract deployment"blockNumber: BigInt!}
Given the above, subgraph developers can use Graph CLI to deploy this Substreams-powered subgraph.
yarn install # install graph-cliyarn subgraph:build # build the subgraphyarn subgraph:deploy # deploy the subgraph
That's it! You have built and deployed a Substreams-powered subgraph.
In order to serve Substreams-powered subgraphs, Graph Node must be configured with a Substreams provider for the relevant network, as well as a Firehose or RPC to track the chain head. These providers can be configured via a config.toml
file:
[chains.mainnet]shard = "main"protocol = "ethereum"provider = [{ label = "substreams-provider-mainnet",details = { type = "substreams",url = "https://mainnet-substreams-url.grpc.substreams.io/",token = "exampletokenhere" }},{ label = "firehose-provider-mainnet",details = { type = "firehose",url = "https://mainnet-firehose-url.grpc.firehose.io/",token = "exampletokenhere" }},]