Quickstart

Get started with Zephyr

Opening Zephyr in production is in BETA.

If you're looking to deploy Zephyr programs and are not already running one (early testers), once you've deployed the program, if you'd like to keep it running you should reach out to the xyclooLabs team either on xyclooLabs' discord server or on the SDF dev server Mercury channel.

Anyone can deploy, but the xyclooLabs team will currently take the liberty of removing programs that haven't been claimed by the creators as described above, especially if such programs have high resource consumption or heavy, large, and high-frequency database writes.

Writing large amounts of data for each ledger will currently result in your program being stopped. Beware that Mercury's cloud environment is still BETA. Once Mercury gets out of BETA release you will be charged according to the amount of data you're writing.

Note that Zephyr's SDK is still a work in progress, if you have any feature requests please let us know at https://github.com/xycloo/rs-zephyr-toolkit/issues

In this quickstart, we will build a very simple hello world Zephyr program that writes to a database table for every new ledger close with the message: "World at ledegr sequence {}", sequence.

Prerequisites

Before starting with Zephyr, make sure that you have a Mercury account and have a Mercury API key at hand.

Also, you should have Rust installed. Being familiar with rust is not a strict requirement as working with Zephyr is generally quite easy, but is recommended for a better experience. We also hope to be able to bring more SDKs for other languages in the future!

Setting up Zephyr.

Before starting, make sure to load your mercury API token as a variable in your current shell. For example:

export MERCURY_JWT="ey..."

Installing the Mercury CLI

The Mercury CLI (which currently only has Zephyr functionalities), is an essential tool to interact with Mercury's cloud execution environment. Technically, you could also use the API, but we recommend working with the CLI for ease of development.

To install the CLI simply run:

cargo install mercury-cli

This should install the CLI, you can verify with

mercury-cli --version

Setting up the project

The mercury-cli takes care of setting up the project for us:

mercury-cli new-project --name zephyr-hello-world

This will create the starting point for our Zephyr program: set up the Cargo.toml, add some compiler flags, a starting point in the lib.rs and create the zephyr.toml.

Zephyr.toml

As the last step for setting up our Zephyr program, we need to create and define a zephyr.toml configuration file.

This configuration mainly defines the tables your program will read from/write to and their structure. A more detailed explanation about zephyr.toml files can be found at Understanding Zephyr.toml And Database Interactions, but here's the configuration that we will use for this quickstart:

name = "zephyr-hello-world"

[[tables]]
name = "test"

[[tables.columns]]
name = "hello"
col_type = "BYTEA"

The above file means that we're declaring that we may write to/read from a table "test" with "hello" as the only column during the execution of our Zephyr program.

Note that col_type = "BYTEA" means that we can only store bytes in the database. Currently, this is the only supported data type (even though it will support built-in types in the future). That said, we recommend writing XDR structures to the database to ease decoding the client side with the existing Stellar tooling.

Writing the program

You're now all set to head to the src/lib.rs and start building your custom indexer!

As you'll notice, our lib.rs comes with prelude imports. These are required by the DatabaseDerive macro which we use to describe into a rust type the structure of our table:

#[derive(DatabaseDerive, Clone)]
#[with_name("test")]
struct TestTable {
    hello: ScVal,
}

Here we use the with_name attribute to assign the structure to the same table ("test") name we defined in the zephyr.toml, and with the hello column as struct field.

Entry point

Now that we've defined the rust-level table that we'll be using, we can start writing the actual ingestion logic. To do so, we move to the already-existing entry point function, i.e. the function the VM will call when executing our program. Inside the function, we also define a new EnvClient object, which is the object that will help us communicate with the VM.

#[no_mangle]
pub extern "C" fn on_close() {
    let env = EnvClient::new();
    
    ...
}

The env variable now enables us to:

• retrieve XDR data.

• log messages and data.

• interact with the database.

• send web requests.

• read from the ledger.

For this quickstart, we will only be writing to the database, reading raw XDR data, and logging some messages to ensure our program is running correctly.

Obtaining the ledger sequence

Our program is called for every new ledger close and is provided with all the changes, transactions, etc that occurred in the ledger. This also includes the ledger sequence kept in the header, which we can easily access thanks to the SDK's meta reader:

let sequence = env.reader().ledger_sequence();

Logging

Logging is highly recommended in the current state where local testing is unavailable to all users. Logging can help you understand where the code might be breaking and isolate the cause of the errors. For example, we can now log that we have retrieved a certain ledger sequence:

env.log().debug(format!("Got sequence {}", sequence), None);

In Zephyr, the possible log levels are debug(), warning(), and error(). Each of these methods takes two arguments:

• A string message.

• An optional Vec<u8>, designed to log serialized data.

Writing to the database

To write to the database (and log accordingly in the meantime) we use:

...

// Craft the message as ScVal
let message = {
    let message = format!("World at ledegr sequence {}", sequence);
    ScVal::String(ScString(message.try_into().unwrap()))
};

// Create an abstract representation of a
// table row.
let table = TestTable {
    hello: message.clone(),
};

env.log().debug(
    "Writing to the database",
    Some(bincode::serialize(&message).unwrap()),
);
// Insert the row into the table.
table.put(&env);
env.log().debug("Successfully wrote to the database", None);

First, we craft the ScVal that we will store in the database, then create a new TestTable object that represents a row within the table, and insert it into the table using table.put(&env).

Complete code

This is the final code:

use zephyr_sdk::{prelude::*, soroban_sdk::xdr::{ScString, ScVal}, EnvClient, DatabaseDerive};

#[derive(DatabaseDerive, Clone)]
#[with_name("test")]
struct TestTable {
    hello: ScVal,
}

#[no_mangle]
pub extern "C" fn on_close() {
    let env = EnvClient::new();

    let sequence = env.reader().ledger_sequence();
    env.log().debug(format!("Got sequence {}", sequence), None);

    let message = {
        let message = format!("World at ledegr sequence {}", sequence);
        ScVal::String(ScString(message.try_into().unwrap()))
    };

    let table = TestTable {
        hello: message.clone(),
    };

    env.log().debug(
        "Writing to the database",
        Some(bincode::serialize(&message).unwrap()),
    );
    table.put(&env);
    env.log().debug("Successfully wrote to the database", None);
}

Deployment

To deploy your program to testnet, you can run

mercury-cli --jwt $MERCURY_JWT --local false --mainnet false deploy

This will automatically create or replace the tables needed and deploy the program.

Monitoring the Execution

Now that you've deployed the program, you can head to https://test.mercurydata.app/custom-ingestion. You should see something similar to

As you can see from the table size, storing information for each ledger execution as we're doing in this program is not recommended. If you're storing new data for each ledger close, then it's likely that what you're doing is replicating a part of ledger functionality and we recommend relying on serverless functions.

From the dashboard, you can currently:

• pause/resume the execution of your program.

• see your tables and their space growth.

• start/end logs streaming.

Soon, you'll also be able to manage the tables from the dashboard to add filtering queries or modify their structure on the database (which currently can only be done by reaching out to us). Also, once the projects feature is out, you'll be able to deploy multiple programs with the same account.

That said, to monitor that our program is executing correctly, you can click on "start streaming" and the logs will appear under the "program logs" tab:

Logs should only be used to test the correct execution of the program and should not be kept streaming indefinitely. Mercury will automatically empty your logs after a certain threshold.

Querying

Our program is writing to a Zephyr table, more specifically in my case the table is zephyr_85b036892719b0a99aa987b1f62e9b10 . To query the table, you can run the query on Mercury's GraphQL API (in this case, the testnet api):

query Test {
  allZephyr85B036892719B0A99Aa987B1F62E9B10S {
    edges {
      node {
        hello
      }
    }
  }
}

For example:

curl 'https://api.mercurydata.app/graphql' \
  -H 'authorization: Bearer YOUR_JWT' \
  -H 'content-type: application/json' \
  --data-raw '{"query":"query Test {\n  allZephyr85B036892719B0A99Aa987B1F62E9B10S {\n    edges {\n      node {\n        hello\n      }\n    }\n  }\n}\n","operationName":"Test"}' \
  --compressed

As a response, you should see something similar to:

{
  "data": {
    "allZephyr85B036892719B0A99Aa987B1F62E9B10S": {
      "edges": [
        {
          "node": {
            "hello": "AAAADgAAACBXb3JsZCBhdCBsZWRlZ3Igc2VxdWVuY2UgMTMxMTY5Nw=="
          }
        },
        {
          "node": {
            "hello": "AAAADgAAACBXb3JsZCBhdCBsZWRlZ3Igc2VxdWVuY2UgMTMxMTY5OA=="
          }
        },
        ...
      ]
    }
  }
}

If you decode the contents, you should see the message we encoded in the scval string:

---

You've now created your first Zephyr ingestion program! From now on the possibilities are endless, and we recommend checking out the next sections of the docs.