Intermediate

Node Setup – Setting up BLS Keys and syncing DB

This tutorial provides instructions on setting up a validating node on the Harmony blockchain by generating BLS keys, configuring them, and setting up a standby node. It also includes information on syncing the node’s database using Rclone, a tool for synchronizing files and directories to and from various cloud storage services. The tutorial covers the following steps:

Creating new BLS keys and providing a passphrase to encrypt the BLS key file

Configuring the BLS keys by creating a folder called .hmy/blskeys, copying the BLS key files to this folder, and creating a corresponding .pass file for each BLS key with the passphrase inside it.

Setting up a standby node with the same BLS key

Syncing the node’s database using Rclone by installing Rclone, configuring it, and creating a rclone.conf file.

Creating new BLS keys

You will need to generate one or more BLS keys in order to run a validating node. When generating a BLS key, the CLI will ask you to provide a passphrase to encrypt the BLS key file.‌

./hmy keys generate-bls-keys --count 1 --shard 1 --passphrase

On the command above --count defines the number of BLS keys you want to generate and --shard the shard associated. On this example, we are generating 1 BLS key on shard 1.

Remember your passphrase. You will need it to decrypt the BLS key file in order to create a validator & start a node with that key.

Create a backup of your BLS key file or save the BLS private key (optional).

The BLS public key is the same as the name of the file, without the .key.

Configuring the BLS keys

You need to manually create a folder called .hmy/blskeys:

mkdir -p .hmy/blskeys

Copy all the previously created BLS key(s) to this new folder:

cp *.key .hmy/blskeys

Make sure all your BLS keys belong to the same shard when using multiple BLS keys. You can use the command below to check each one of them.

./hmy --node="https://api.s0.t.hmny.io" utility shard-for-bls [BLS PUBLIC KEY]

For each BLS key file, a corresponding <blskey>.pass file needs to be created inside folder.hmy/blskeyswith the passphrase inside it.

following this format :

echo '[replace_with_your_passphrase]' > .hmy/blskeys/[replace_with_BLS_without_.key].pass

you should finally have in your .hmy/blskeys folder :

ls .hmy/blskeys/
0c8a92c872798742031c612acea7b686a58b16722a02e072442f14ad4f9499e934da97f4db7d1a68307a96335e06bb0c.key
0c8a92c872798742031c612acea7b686a58b16722a02e072442f14ad4f9499e934da97f4db7d1a68307a96335e06bb0c.pass

Setting up a standby node with the same BLS key

It is NOT recommended now to run multiple nodes using the same set of BLS keys. As we are moving towards full decentralization, external validators will become the shard leader and start to propose blocks. If one validator runs multiple nodes using the same set of valid keys, they may all become valid leaders when the key is rotated to this validator, in this case, the blockchain is experiencing a high risk of hard-fork as different valid leaders may propose different blocks. So, do not run redundant validator nodes anymore on the Harmony blockchain. This is also strictly forbidden on all PoS blockchains such as Ethereum 2, Cosmos.

It is recommended to set up standby nodes in case the main node is going to maintenance mode. Check out this bounty on how this feature is supported to run and switch backup nodes.

Syncing DB

Validator Nodes

This document introduces another centralized fast state syncing method using rclone. Please use it with caution. This guide is mainly used for a newly started node to catch up with the blockchain faster. Otherwise, the blockchain syncing may take weeks from genesis block.

Rclone db snapshot is sync’ed with blockchain frequently. However, there maybe a potential race condition when the rclone may fail due to our nodes were updating the db files at the same time. In this case, just re-run the rclone command to re-sync again.

Installing Rclone

For installing Rclone, please follow the instructions at https://rclone.org.

TL;DR, on a Linux system, you may run the following command.

curl https://rclone.org/install.sh | sudo bash

Make sure your rclone version is above v1.53.2 . you can check version by rclone version

There is a daily snapshot of the rclone DB happening at 1:42:30 AM UTC for 3h. It is not advisable to perform a backup during those time.

Configuring Rclone

To check the location of the rclone.conffile run:

rclone config file

The rclone.conf file is usually located at ~/.config/rclone/rclone.conf .

Now run the following command to create the rclone.conf file.

cat<<-EOF > ~/.config/rclone/rclone.conf
[release]
type = s3
provider = Storj
access_key_id = jwuyiin22hl6o5zxzlf76rc6csxa
secret_access_key = jyryzso3fv64zu4i2gmf33ed5hudb7qyatgpst22fnp57dfhviv4k
endpoint = gateway.us1.storjshare.io
EOF

The above rclone config also work for the pangaea testnet network

Running Rclone

Below is the command to sync the shard you want. Replace <ShardID>with the shard number you want to sync. Shard 0 is around 300 Gb as of November 2021.

Mainnet

rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_<ShardID> harmony_db_<ShardID> --multi-thread-streams 4 --transfers=32

Testnet

rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_<ShardID> harmony_db_<ShardID> --multi-thread-streams 4 --transfers=32

If you encounter the following error:

NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults

Add the option --config=/home/harmony/.config/rclone/rclone.conf to the command line.

Cheat Sheet

Since v4.3.12 shard 1/2/3 doesn’t need to download anymore shard 0 DB. Instead, the node will download automatically the blocks required for the node in shard 1/2/3 to work.

Each node will simply need to rclone its own DB.

Shard0 RPC Explorer node (non archival):

Mainnet

rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32

Testnet

rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32

Shard0 Validator node:

SnapDB is a new type of DB that contain a snapshot of the state at a given block (ie as of 4th March 2022, this is block 22816573). It doesn’t have any block history history and hence not suitable for RPC node

rclone -P -L --checksum sync release:pub.harmony.one/mainnet.snap/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32

Shard 1:

Mainnet

rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_1 harmony_db_1 --multi-thread-streams 4 --transfers=8

Testnet

rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_1 harmony_db_1 --multi-thread-streams 4 --transfers=32

Shard 2:

Mainnet

rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_2 harmony_db_2 --multi-thread-streams 4 --transfers=8

Testnet

rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_2 harmony_db_2 --multi-thread-streams 4 --transfers=32

Shard 3:

Mainnet

rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_3 harmony_db_3 --multi-thread-streams 4 --transfers=32

Testnet

rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_3 harmony_db_3 --multi-thread-streams 4 --transfers=8

After the sync, you may use du -h harmony_db_* command to check the size of the downloaded snapshots.

-P will display a download progress & ETA.

Non-Validating/Explorer Nodes

As of 1st December 2022, the size for the shard 0 on mainnet is ~21TiB. Note that the bucket is currently not fully synced.

Please contact us at devops@harmony.one if you need to access our archival db.