How to Add a Custom Index in Maestro Symphony

the maestro symphony https //github com/maestro org/maestro symphony is a fast, mempool aware, and extensible bitcoin indexer and api server it provides a framework for indexing utxos, metaprotocols, and any other onchain activity this guide explains how to construct and integrate custom logic to be extracted and persisted during transaction processing by the maestro symphony indexer steps 1\ create a new indexer project and initialize it after forking https //github com/maestro org/maestro symphony/fork the repo, create a file or module in the /custom / /src/sync/stages/index/indexers/custom/ subdirectory setup a new project directory mkdir p src/sync/stages/index/indexers/custom/my proj create the files (plus any extra that you need) touch mod rs indexer rs tables rs generate the mod rs (and add your related files) pub mod indexer; pub mod tables; it is recommended to split logic across multiple files (see the runes / /src/sync/stages/index/indexers/custom/runes/ indexer for reference) 2\ register a new enum variant create a new transactionindexertype enum variant src/sync/stages/index/indexers/custom/my proj/mod rs add your variant only at the end example /// unique u8 for each transaction indexer, used in the key encodings do not modify, only add new /// variants \#\[derive(clone, copy, encode, decode, partialeq, eq, std hash hash, debug)] \#\[repr(u8)] pub enum transactionindexertype { txcountbyaddress = 0, runes = 1, utxosbyaddress = 2, myprojindexer = 3, // my proj indexer } ⚠️ do not reorder or delete existing variants they are encoded as u8 and reused in key encodings any change will corrupt existing indexed data unless starting from a clean state 3\ define the indexer object implement a struct that represents your indexer and implements the processtransaction trait pub struct myprojindexer { start height u64, track inputs bool, } impl processtransaction for myprojindexer { fn process tx( \&mut self, task \&mut indexertask, tx \&transaction, ctx \&indexercontext, ) > result<(), error> { } } where task read/write interface to storage tx the transaction being processed ctx context with input resolver, block height, hash, network, arbitrary data to outputs, etc a resolver lets you provide a transcation input utxo reference and receive the corresponding transaction output utxo reference runes/indexer rs#l41 l61 https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/runes/indexer rs#l41 l61 4\ implement and handle config your indexer should expose a new() function that takes a configuration struct and returns an instance of the indexer this enables configuration driven behavior such as start height, track inputs, or custom logic impl myprojindexer { pub fn new(config myprojindexerconfig) > result\<self, error> { let start height = config start height; let track inputs = config track inputs; ok(self { start height, track inputs, }) } } \#\[derive(clone, debug, deserialize)] pub struct myprojindexerconfig { \#\[serde(default)] pub start height u64, \#\[serde(default)] pub track inputs bool, } the myprojindexerconfig struct should define fields relevant to your indexer reference https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/runes/indexer rs#l41 l72 5\ define storage tables define custom key value tables for storing the new data using the define indexer table! macro src/sync/stages/index/indexers/custom/my proj/tables rs example define indexer table! { name myprojindexerkv, key type scriptpubkey, value type u64, indexer transactionindexer myprojindexer, table 0 } // key value address => tx count each table must have a unique table id use your new enum variant use key/value types that implement encode and decode reference tx count by address rs#l20 l26 https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/tx count by address rs#l20 l26 6\ implement processtransaction process each transaction by iterating over inputs, ouputs, resolving utxos, etc reading/writing to storage with impl processtransaction for myprojindexer { fn process tx( \&mut self, task \&mut indexertask, tx \&transaction, ctx \&indexercontext, ) > result<(), error> { let transactionwithid { tx, } = tx; // retrieve value from kv store task get \<myprojindexerkv>(\&key)?; // set value in kv store task put \<myprojindexerkv>(\&key, \&value)?; } } example tx count by address rs#l38 l76 https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/tx count by address rs#l38 l76 7\ register module and add to factory add your my proj module in custom/mod rs pub mod id; pub mod runes; pub mod tx count by address; pub mod utxos by address; pub mod my proj; // my proj indexer reference https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/mod rs#l11c1 l14c26 next, add your variant to the transactionindexerfactory enum the create indexer function \#\[derive(clone, debug, deserialize)] \#\[serde(tag = "type")] pub enum transactionindexerfactory { txcountbyaddress, runes(runesindexerconfig), utxosbyaddress, myprojindexer(myprojindexerconfig), } impl transactionindexerfactory { pub fn create indexer(self) > result\<box\<dyn processtransaction>, error> { match self { self txcountbyaddress => ok(box new(txcountbyaddressindexer new())), self runes(c) => ok(box new(runesindexer new(c)?)), self utxosbyaddress => ok(box new(utxosbyaddressindexer new())), self myprojindexer(c) => ok(box new(myprojindexer new(c)?)), } } } 8\ (optional) attach metadata to utxos to persist data across transactions using utxos (e g , inscriptions, runes), you can attach metadata during output processing task attach metadata to output(vout, \&data)?; later, during input resolution let meta = ctx resolver resolve input(input)? metadata \<yourtype>()?; examples attach metadata https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/runes/indexer rs#l256 l260retrieve metadata https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/runes/indexer rs#l318 l321 this saves storage and avoids manually tracking utxos elsewhere for working examples, refer to runes https //github com/maestro org/maestro symphony/tree/main/src/sync/stages/index/indexers/custom/runestx count by address https //github com/maestro org/maestro symphony/blob/main/src/sync/stages/index/indexers/custom/tx count by address rs 🎉 you’re done! you have now walked through a guide on how to index a custom piece of data and add an api endpoint using the maestro symphony https //github com/maestro org/maestro symphony be sure to check out maestro's additional services https //www gomaestro org/chains/bitcoin for further assisting your development of building on bitcoin support if you are experiencing any trouble with the above, reach out on discord https //discord gg/es2rdhbjt3