Data Catchups/Backfill
Learn how you can backfill older data into your program.
Last updated
Learn how you can backfill older data into your program.
Last updated
It's common in a contracts-centered development workflow to first deploy the contract and make sure things just work vs deploying the indexer first. We understand this crucial requirement and thus have set up a fast and efficient data catchup infrastructure that builds artificial ledger transitions.
This allows to backfill older data into your program ensuring that you index also actions performed before your indexer's deployment.
Before you can request a catchup with events from a given contract or contract, you must be subscribed to these events first. See Subscriptions. Contract event subscriptions can be easily performed through the Mercury webapp (, ).
The subscription is only required for catchups, not standard execution. Additionally, it doesn't currently mean higher resource usage or higher costs.
Define your indexing logic beforehand and deploy the program. When you run the catchup, Zephyr will generate artificial ledger transitions to execute that same logic even for past data.
To initiate a catchup you can use the following mercury CLI command.
Mainnet:
Testnet:
Executing this command will yield a catchup id which can be used to retrieve the status of the catchup.
To perform the catchup of a RetroShades program just add the --retroshades true flag:
We almost always recommend to use scoped event catchups in this step instead of the above standard catchup. Learn more below in the Scoped Event Catchups section.
To query the catchup status:
Catchups logs, as well as other Mercury logs, can be found by clicking on the "User Logs" section in the dashboard's sidebar.
Sometimes we don't need to catchup all the events of a contract or at all past ledgers. For example, assuming that we're indexing a pool contract's deposits/withdrawals, but the pool also emits many airdrop or claim events, with a standard catchup we'd end up spinning a vm, and executing our program for artificial close metas that don't hold any of the information we're looking for.
To trigger a catchup scoped by event topics (this shows for mainnet but it's the same for testnet, just with --mainnet false ):
Note that this was used to catchup Blend's YieldBlox pool on mainnet.
Alternatively, imagine a scenario where you need to index events from after a certain ledger because a new token was added to the pool and your indexer wasn't written with that in mind. In such a scenario, with standard catchups you'd need to ingest again all of your data, however scoped catchups allow to catchup starting from a certain ledger.
To trigger a catchup after a certain ledger (this shows for mainnet but it's the same for testnet, just with --mainnet false ):
