Data Catchups/Backfill
Learn how you can backfill older data into your program.
It's common in a contracts-centered development workflow to first deploy the contract and make sure things just work vs deploying the indexer first. We understand this crucial requirement and thus have set up a fast and efficient data catchup infrastructure that builds artificial ledger transitions.
This allows to backfill older data into your program ensuring that you index also actions performed before your indexer's deployment.
1. Subscribe to the contract's events.
Before you can request a catchup with events from a given contract or contract, you must be subscribed to these events first. See Subscriptions. Contract event subscriptions can be easily performed through the Mercury webapp (testnet, mainnet).
The subscription is only required for catchups, not standard execution. Additionally, it doesn't currently mean higher resource usage or higher costs.
2. Deploy your program
Define your indexing logic beforehand and deploy the program. When you run the catchup, Zephyr will generate artificial ledger transitions to execute that same logic even for past data.
3. Call the catchup
To initiate a catchup you can use the following mercury CLI command.
Mainnet:
mercury-cli --jwt $MERCURY_JWT --local false --mainnet true catchup --contracts "CONTRACT_1" --contracts "CONTRACT_2" --project-name "YOUR ZEPHYR PROJECT NAME"Testnet:
mercury-cli --jwt $MERCURY_JWT --local false --mainnet false catchup --contracts "CONTRACT_1" --contracts "CONTRACT_2" --project-name "YOUR ZEPHYR PROJECT NAME"Executing this command will yield a catchup id which can be used to retrieve the status of the catchup.
RetroShades Catchup
To perform the catchup of a RetroShades program just add the --retroshades true flag:
mercury-cli --jwt $MERCURY_JWT --local false --mainnet true/false catchup --retroshades true --contracts "YOUR_CONTRACT" --project-name "YOUR_PROJECT_NAME"We almost always recommend to use scoped event catchups in this step instead of the above standard catchup. Learn more below in the Scoped Event Catchups section.
3. Reading catchup status.
To query the catchup status:
curl -X GET https://api.mercurydata.app/catchups/IDReading catchup logs
Catchups logs, as well as other Mercury logs, can be found by clicking on the "User Logs" section in the dashboard's sidebar.
Scoped Event Catchups
Sometimes we don't need to catchup all the events of a contract or at all past ledgers. For example, assuming that we're indexing a pool contract's deposits/withdrawals, but the pool also emits many airdrop or claim events, with a standard catchup we'd end up spinning a vm, and executing our program for artificial close metas that don't hold any of the information we're looking for.
To trigger a catchup scoped by event topics (this shows for mainnet but it's the same for testnet, just with --mainnet false ):
./mercury-cli --jwt $MERCURY_JWT --mainnet true catchup --contracts "CBP7NO6F7FRDHSOFQBT2L2UWYIZ2PU76JKVRYAQTG3KZSQLYAOKIF2WB" --topic1s "AAAADwAAABFzdXBwbHlfY29sbGF0ZXJhbAAAAA==" --topic1s "AAAADwAAABN3aXRoZHJhd19jb2xsYXRlcmFsAA==" --topic1s "AAAADwAAAAZib3Jyb3cAAA==" --topic1s "AAAADwAAAAVyZXBheQAAAA==" --project-name "ybx-pool"Note that this was used to catchup Blend's YieldBlox pool on mainnet.
Alternatively, imagine a scenario where you need to index events from after a certain ledger because a new token was added to the pool and your indexer wasn't written with that in mind. In such a scenario, with standard catchups you'd need to ingest again all of your data, however scoped catchups allow to catchup starting from a certain ledger.
To trigger a catchup after a certain ledger (this shows for mainnet but it's the same for testnet, just with --mainnet false ):
./mercury-cli --jwt $MERCURY_JWT --mainnet true catchup --contracts "CBP7NO6F7FRDHSOFQBT2L2UWYIZ2PU76JKVRYAQTG3KZSQLYAOKIF2WB" --start 52810830 --project-name "ybx-pool"Resources
Last updated