Intermediate
How to Set Up a Private indexer
Welcome to our tutorial on how to set up a private indexer for the Tezos blockchain network. In this tutorial, we will cover the process of setting up a private indexer to index the Tezos blockchain and provide fast, efficient, and secure access to its data. We will explore the various tools and methods that can be used to set up a private indexer, such as Tezos-Indexer and TzScan.
By the end of this tutorial, you will have the knowledge and hands-on experience needed to set up your own private indexer for the Tezos blockchain. You will also learn how to configure and maintain your private indexer to ensure optimal performance and security. This tutorial is ideal for developers, researchers, and anyone interested in setting up a private indexer for the Tezos blockchain to access its data in a fast, efficient, and secure way.
Prerequisites
We will simulate a private network that will be indexed by TzIndex. Therefore, this private network should hold the basic expected elements on any network:
-
a few funded accounts
-
a smart contract
-
some operations
Let’s start by setting up a private network. The process is quite difficult as it requires starting and configuring Tezos nodes (please refer to the Private Blockchain module for more info). Instead, let’s choose an easier way by only simulating a private network using Ganache, a node module used for tests and simulations of Tezos and Ethereum blockchains. Ganache launches the Flextesa docker image, which runs Tezos nodes, bakers and endorsers in a sandbox environment. It also provides ten funded accounts from the start.
Let’s use the Raffle smart contract from the LIGO module and migrate it onto our private network using the Truffle configuration from the Build a Dapp module.
Installing the prerequisites
The complete source code of the raffle contract can be found here.
It contains a ganache configuration (with predefined accounts), three smart contracts, and their associated migrations:
1. The first contract holds a big map and does not do anything. This dummy contract is used to bypass a TzStats bug regarding big maps: the first big map (whose index is 0) is not fetched by the frontend.
2. A raffle contract using a map
3. A raffle using a big map
The second and third smart contracts will highlight the difference between the way maps and big maps are handled by indexers.
$ cd examples/explorer $ npm install -g truffle@tezos $ npm install
TzIndex is written in Go, a statically typed compiled programming language developed by Google. Install it using the instructions at golang.org/doc/install.
The next action is to install the docker images of TzIndex. Install Docker using the instructions at docs.docker.com/get-docker/.
Launching Ganache
In package.json
, one script is defined:
"scripts": { "start-sandbox": "ganache-cli --flavor tezos --seed alice" }
This script starts a simulated private Tezos blockchain with ganache:
$ npm run start-sandbox Ganache CLI v6.12.1-tezos.0 (ganache-core: 2.13.2-tezos.2) Available Accounts ================== alice 100 TEZ pk: edpkvGfYw3LyB1UcCahKQk4rF2tvbMUk8GFiTuMjL75uGXrpvKXhjn pkh: tz1VSUr8wwNhLAzempoch5d6hLRiTh8Cjcjb bob 100 TEZ pk: edpkurPsQ8eUApnLUJ9ZPDvu98E8VNj4KtJa1aZr16Cr5ow5VHKnz4 pkh: tz1aSkwEot3L2kmUvcoxzjMomb9mvBNuzFK6 eve 100 TEZ pk: edpku9qEgcyfNNDK6EpMvu5SqXDqWRLuxdMxdyH12ivTUuB1KXfGP4 pkh: tz1MnmtP4uAcgMpeZN6JtyziXeFqqwQG6yn6 mallory 100 TEZ pk: edpkujwsG5JMrWVXQwmRMBoR9yJkokRbn6wy3monpQKBpnZTs1ogRR pkh: tz1R2oNqANNy2vZhnZBJc8iMEqW79t85Fv7L trent 100 TEZ pk: edpkukjpYWLPEavoTXid8KqfYAY4J1rqtbWHWrzp22FgdRQqzaRkDD pkh: tz1TfRXkAxbQ2BFqKV2dF4kE17yZ5BmJqSAP marketa 100 TEZ pk: edpktiaGKkt8Yu6m4Gse2GhMdJCVdCtcrjtqATn3y3sf7xTBDj5g2a pkh: tz1fhigafd7PQAh3JBvq7efZ9g6cgBkaimJX eulalie 100 TEZ pk: edpkvCvic2obeedM7oMJaeyapEafg4dSdYuWvkZigKbcvc64q6ZKM7 pkh: tz1fEqJ6rD3mfQjVat7G6XJP522V6V8wWTP2 stella 100 TEZ pk: edpkvRuciP6vZeoXn1KJtBuEEtaD2SpHW59wbbCGt1SEDBL94W7AUE pkh: tz1i3eqdPNs9zjpavVBFcF8JarJUgEErfsUK carline 100 TEZ pk: edpktxefxf3dtJEQLVb72MjV8yMiLh6DfCgNJQUV81rnsYJoZhbnK8 pkh: tz1PQP815EzZRktgLaEhtDCR22kiRrcQEurw tabbie 100 TEZ pk: edpkvXobE6tQLdsm3kPu3MYT2z2XrgVchfkK2WPB9tniNXdWSRyud3 pkh: tz1WP3xUvTP6vUWLRnexxnjNTYDiZ7QzVdxo <...>
Notice all the funded accounts created by Ganache.
Migrating the Raffle contracts
The contracts can be migrated onto the private network. More details can be found about the Truffle tool in the Build a Dapp module.
First, the private network has to be defined in the truffle-config.js file. A development
subsection should be found under the networks
section:
development: { host: "http://localhost", port: 8732, network_id: "*", secretKey: alice.sk, type: "tezos" }
It defines a localhost network, matching the ganache private network.
We’re going to use the accounts from the scripts/sandbox/account.js
file. Our three contracts can now be migrated with this command:
$ truffle migrate --network development
Three contracts are now deployed onto our private network:
The migration files also include some operations to open a raffle and buy a ticket automatically once the contracts are deployed.
Setting up a private indexer (TzIndex)
TzIndex is an open-source indexer: it can freely be used and modified. The source code is available at github.com/blockwatch-cc/tzindex
To start the application, proceed as follows:$ git clone https://github.com/blockwatch-cc/tzindex
$ cd tzindex
$ make build
$ ls tzindex
$ git clone https://github.com/blockwatch-cc/tzindex $ cd tzindex $ make build $ ls tzindex
The last command should output “tzindex” (to check if a tzindex has indeed been built) A TzIndex binary is created in the same directory. The indexer can now watch the private network with this command:
$ ./tzindex run --rpcurl 127.0.0.1:8732 --notls --enable-cors
Note the following options:
-
–rpcurl: url of the ganache private network rpc node
-
–notls: option to use http instead of https
-
–enable-cors: TzIndex API will be queried by Tzstat (exposed on a different port): CORS need to be enabled
TzIndex now exposes its API on http://localhost:8000.
A db/
folder is created when launching TzIndex for the first time: it contains all the data indexed about the private network.
Setting up a private explorer (TzStats)
TzStats is the open-source frontend made to display the data from TzIndex. The source code is available at github.com/blockwatch-cc/tzstats
TzStats can interact with TzIndex by setting the TZSTATS_API_URL variable to the url of the TzIndex API.
The application can be launched with:$ git clone https://github.com/blockwatch-cc/tzstats
$ cd tzstats
$ echo ‘TZSTATS_API_URL=http://localhost:8000’ > development.env
$ npm install
$ yarn start
$ git clone https://github.com/blockwatch-cc/tzstats $ cd tzstats $ echo 'TZSTATS_API_URL=http://localhost:8000' > development.env $ npm install $ yarn start
TzStats will now launch in your web browser.
Interacting with the private explorer
At this point, the working context is:
-
a simulated private network running through Ganache
-
three migrated contracts
-
two contract calls
-
TzIndex indexing the private network
-
TzStats connected to TzIndex, and running
This means that there has already been some activity that we can watch on our network.
Watching activity on TzStats
TzStats is running at http://localhost:3000:
Notice that the frontend is different from its public version on TzStats.com but the inner workings are the same.
On top, the search bar allows you to search for transactions, blocks, addresses, etc. The left panel contains various information displayed, such as the number of baked blocks.
Click on it to get more info on that block:
Many block details are displayed; most of them can be clicked on for even more info. The block above only contains one operation, which is a call to our smart contracts. Click the hash to open the operation page:
The sender does match Alice’s pkh
from scripts/sandbox/account.js. pkh
means public key hash: it is an address on a Tezos network. Notice the contract address matches the returned address from the migration output logs.
Click on the address to inspect the smart contract:
You can see the history of calls, the entrypoints, the storage, etc. Notice the two calls made by Truffle: the origination of the contract and the purchase of a ticket.
The storage page displays the storage definition, the type of each field, and their associated current value. Note that one participant is registered in the raffle and the contract holds one Tez (from selling one ticket).
You can see the history of calls, the entrypoints, the storage, etc. Notice the two calls made by Truffle: the origination of the contract and the purchase of a ticket.
The storage page displays the storage definition, the type of each field, and their associated current value. Note that one participant is registered in the raffle and the contract holds one Tez (from selling one ticket).
Regular maps are meant to be used with limited data size as the data is directly retrieved from the storage section. You can see an example of such a map in the second migrated smart contract (baked in the fourth block):
The TzStats interface is user-friendly and every piece of information can easily be read by clicking on each element. Remember that all this information is coming from the data retrieved by the indexer: TzStats just displays them.
Watching activity from TzIndex
The same pieces of information can be retrieved without the frontend by just using the TzIndex API. For each page opened on TzStats, an API call is made to TzIndex. Each API call can be seen in the network explorer by pressing F12 in the network section.
The request URL shows the called API endpoints:
$ GET http://127.0.0.1:8000/explorer/contract/KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF? { "address": "KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF", "manager": "tz1VSUr8wwNhLAzempoch5d6hLRiTh8Cjcjb", "delegate": "", "height": 6, "fee": 0.003939, "gas_limit": 12826, "gas_used": 12726, "gas_price": 0.30952, "storage_limit": 2654, "storage_size": 2397, "storage_paid": 2397, "is_funded": true, "is_vesting": false, "is_spendable": false, "is_delegatable": false, "is_delegated": false, "first_in": 7, "first_out": 0, "last_in": 7, "last_out": 0, "first_seen": 6, "last_seen": 7, "delegated_since": 0, "first_in_time": "2021-04-17T17:06:27Z", "first_out_time": "0001-01-01T00:00:00Z", "last_in_time": "2021-04-17T17:06:27Z", "last_out_time": "0001-01-01T00:00:00Z", "first_seen_time": "2021-04-17T17:06:26Z", "last_seen_time": "2021-04-17T17:06:27Z", "delegated_since_time": "0001-01-01T00:00:00Z", "n_ops": 1, "n_ops_failed": 0, "n_tx": 1, "n_delegation": 0, "n_origination": 0, "token_gen_min": 1, "token_gen_max": 1, "bigmap_ids": [ 1 ], "op_l": 3, "op_p": 0, "op_i": 0, "iface_hash": "d30a2146", "call_stats": [ 1, 0, 0 ] }
A lot of information is displayed, such as the contract’s address, but some data is also missing, like the storage. Another API call has to be made for that:
$ GET http://127.0.0.1:8000/explorer/contract/KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF/storage? { "meta": { "contract": "KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF", "time": "2021-04-17T17:06:27Z", "height": 7, "block": "BLwCojKb2fv7Ph88kxMoYXeTRYdtavvzCNh3ZzsJNpfFgqq5ey8" }, "value": { "admin": "tz1cGftgD3FuBmBhcwY24RaMm5D2UXLr5LHW", "close_date": "1618679185827", "contract_name": "Raffle smart contract with big map", "description": "", "jackpot": "100", "players": [ "tz1VSUr8wwNhLAzempoch5d6hLRiTh8Cjcjb" ], "raffle_is_open": "true", "sold_tickets": "1", "winning_ticket_number_hash": "00" } }
All the available endpoints can be found here: tzstats.com/docs/api#explorer-endpoints
Setting up a private indexer for a public network
If you don’t want your infrastructure to rely on third-party public indexers to monitor public networks, you can also use a local indexer. It just has to monitor a node: either a local node or a public node (listed at tezostaquito.io/docs/rpc_nodes).
$ ./tzindex run --rpcurl <node_url> --notls --enable-cors
There are three operation modes, which retrieve more or less data:
-
Full regular: all indexes are built (default mode)
-
Light light-weight: consensus and governance indexes are not built (use –light in the CLI)
-
Validate: state validation mode for checking accounts and balances at each block/cycle (use –validate in the CLI)
Note that whichever mode your choose, indexing a public network will take a significant amount of time due to the size of the data to be fetched.
According to the Tzstats team, it is possible to index 13 000 blocks per minute from a local node connected to the mainnet (Cost and benefice section from this tzstats post). When indexing the mainnet from a public node (Smartpy node in our case), we could index about 10 blocks per second.
The mainnet was about 1 400 000 blocks, so to index the mainnet, it should take:
-
about an hour and fifty minutes from a local node
-
about a day and a half from a public node.
Of course, these figures depend on your network connectivity and hardware.