Data Catchups/Backfill
Learn how you can backfill older data into your program.
It's common in a contracts-centered development workflow to first deploy the contract and make sure things just work vs deploying the indexer first. We understand this crucial requirement and thus have set up a fast and efficient data catchup infrastructure that builds artificial ledger transitions.
This allows to backfill older data into your program ensuring that you index also actions performed before your indexer's deployment.
Note that you'll only be allowed to catchup with contract events of a subset of contracts that you arbitrarily specify in the CLI when running the catchup. You cannot catchup will all of the emitted events for instance, you must know the contract's addresses beforehand.
1. Subscribe to the contract's events.
After the last CLI update it's no longer needed to subscribe to the relevant events before the catchup: the CLI will perform the task automatically.
Before you can request a catchup with events from a given contract or contract, you must be subscribed to these events first. See Subscriptions. Contract event subscriptions can be easily performed through the Mercury webapp (testnet, mainnet).
The subscription is only required for catchups, not standard execution. Additionally, it doesn't currently mean higher resource usage or higher costs.
2. Deploy your program
Define your indexing logic beforehand and deploy the program. When you run the catchup, Zephyr will generate artificial ledger transitions to execute that same logic even for past data.
3. Call the catchup
To initiate a catchup you can use the following mercury CLI command.
Mainnet:
Testnet:
Executing this command will yield a catchup id which can be used to retrieve the status of the catchup.
We almost always recommend to use scoped event catchups in this step instead of the above standard catchup. Learn more below in the Scoped Event Catchups section.
3. Reading catchup status.
To query the catchup status:
You can see the status of all catchups. This is intentional and not an IDOR.
Reading catchup logs
Catchups logs, as well as other Mercury logs, can be found by clicking on the "User Logs" section in the dashboard's sidebar.
Scoped Event Catchups
Sometimes we don't need to catchup all the events of a contract or at all past ledgers. For example, assuming that we're indexing a pool contract's deposits/withdrawals, but the pool also emits many airdrop or claim events, with a standard catchup we'd end up spinning a vm, and executing our program for artificial close metas that don't hold any of the information we're looking for.
To trigger a catchup scoped by event topics (this shows for mainnet but it's the same for testnet, just with --mainnet false ):
Note that this was used to catchup Blend's YieldBlox pool on mainnet.
Alternatively, imagine a scenario where you need to index events from after a certain ledger because a new token was added to the pool and your indexer wasn't written with that in mind. In such a scenario, with standard catchups you'd need to ingest again all of your data, however scoped catchups allow to catchup starting from a certain ledger.
To trigger a catchup after a certain ledger (this shows for mainnet but it's the same for testnet, just with --mainnet false ):
Note that you can combine both the topics and the ledger start in scoped event catchups
Resources
Last updated