Introduction
This document is a long term plan on how to build a successful validator community. Validators are node runners who setups host machines to run Harmony client software to validate the blockchain together. They provide validating service and get rewarded by the blockchain with block rewards. They are also called “miners” in the traditional PoW blockchain, such as bitcoin, ethereum. In the PoS blockchain, validators are the essential foundation. Validators are stakers but they also provide service to other stakers that delegate the stake to them. In line with our mission, one of our goals and key differentiation is to build the most decentralized PoS blockchain with the strong support of the validator community. Thus, we should provide better service to average validators and make sure they can be successful. #validatorsuccess is our success.
Background
As of 09/27/2019, there are 320 nodes run by ~220 external validators. 237 (74%) nodes are online and earning block rewards. The average hourly earning rate is ~180 $ONE. The highest overall earning is 130,780 $ONE. Since we released cross-shard transaction support on 09/11, due to the high CPU load and network bandwidth requirement added to the node program when the beacon chain syncing was enabled, the average hourly rate of block rewards dropped to ~100 $ONE per hour, 45% less than before. Also, we had 5 shard-down accidents between Aug 30 to Sep 17. The community was very upset and asking for stability, better communication, more explanation, and even compensation. This is the pivot point for us to re-examine our plan and strategy on the validator community. We due to layout a longer term plan to build a healthier and stronger validator community for our long term success.
Update: 10/13/2019, we have reached 278/320 (87%) of nodes earning block rewards.
Tenets
- Customer obsession: validators are the extension of the team and the immediate customers of the Harmony blockchain. The plan needs to work backward to fit into customers’ shoes.
- Scalability: decentralization needs participation. All the plan has to be scalable to keep hundreds, if not thousands, of validators in mind.
- Sustainability: long term support from validators is the key for long term success. We need to nurture a healthy culture of the validator community for long term goals.
Plan
The plan to build a stronger and healthier validator community shall cover the following three aspects.
Communication
Communication is the key to build the community. Communication should be transparent, proactive, and inclusive. The strategy includes full coverage of communication, including but not limited to email, twitter, discord, telegram, wechat. All content shall be pushed to all channels with planning and on-time update.
Usability
Usability can help us onboard validator faster and easier. The node software shall be stable, user-friendly, customizable, and hooked with external solutions for logging, metrics, and monitoring. Better tools provided can also help validator onboard and maintain high availability of the nodes, thus providing better security of the network.
Leadership
Community needs leaders. It is our job to identify and promote community leaders. Community leaders are the extended arm of the Harmony and can provide more insight, helping hands, and voices from the community. It is the only way to scale the community.
Actions
Communication
- Email list of all validators in the community
- Dedicated support email from Harmony
- Regular community survey to grasp the voice of the community.
Training and Education
- Validator certification program. Validators have to go through a training program and pass the exam to gain certification in order to run the node on mainnet.
- Training process. Validators should go through testnet before they can be an official validator on the mainnet.
Onboarding
- SLA requirement on the validators, for their uptime requirement and response requirement. Once we have open staking, we still need to be more proactively communicate about the SLA and slashing.
- A portable docker image as the official release.
- Amazon AMI on various regions as the official bootstrap image.
- Prebuilt database snapshots to help new node ramp up faster.
Community Leaders
- Identify 5–10 community leaders and get regular contact with them on a weekly basis.
- Incentivize the community leaders in different ways.
Maintenance
- Compile a common issues we learned from the runners and try to system-wise automate the process and avoid those issues in our software for the first time.
- Build an automation script to create a Harmony DB snapshot that can be used by the validator to quickly recover node.
- Build and document the process to use rclone script to recover validator node faster.
- Validators should respond to the node update within 24 hours.
Monitoring
- Textual UI on validator node to help monitor and triage node issue.
- Central monitoring hub can be provided to validators
- Pagerduty hook-up can be provided to validators.
Resources
- FAQ: https://nodes.harmony.one/foundational-node-playbook/faq
- Knowledge-base: https://nodes.harmony.one/foundational-node-playbook/nodesetupquickref
- 1 Hour Earning Rate: https://harmony.one/1h
- Block Rewards: https://nodes.harmony.one/foundational-node-playbook/mainnet#block-rewards
- FN Survey: https://harmony.one/fn-survey
- TUI Doc: https://nodes.harmony.one/monitoring-and-reporting-tools/text-based-user-interface
— — — — — Harmony Textual UI — — — — —
