docs/advanced/devops.md

Hush DevOps

This documentation is for services that run a Hush full node, such as mining pools, exchanges and lite wallet node operators. It covers advanced things to know which can help reduce downtime.

Minimize Rescan Downtime

Sometimes a Hush node must rescan the history of the blockchain, to look for incoming and outgoing transactions. It normally happens when importing a new private key, but can also happen when some type of crash happens. For instance:

• The physical server or VM runs out of memory
• There is a power outage
• A bug in hushd itself causes a coredump and unclean exit
• kill -9 hushd is used

If the above circumstances happen, hushd will not be able to do it's normal "bookkeeping" when doing a clean exit. Part of this work is updating metadata about shielded transactions, called "witnesses". When hushd starts and notices this data is out of date, it performs a full rescan of all time to update the data, which works, but it's very slow and will cause downtime of potentially hours or longer.

To avoid this downtime, first make sure to be using at least Hush 3.8.0 which has two new features

• -rescanheight is an added option to -rescan that gives a specific height to rescan from instead of the genesis block
• -keepnotewitnesscache which means keep the partial ztx metadata and update it, instead of recreating it from scratch

For example, lets say from the debug.log that a system admin knows that the last block height the node processed was 12345. This can be found by looking for the very last UpdateTip line in the debug.log. That means witness data is correct up to that height, and that is what we can use as a starting point. So the sysadmin would do

hushd -rescan=1 -rescanheight=12345 -keepnotewitnesscache=1

and then hushd will only rescan the history from block 12345 until the current block, which saves huge amount of time.

What can go wrong?

Lets say that the sysadmin uses the wrong -rescanheight on accident, what can go wrong? Is it possible to lose funds? No, it is not possible to lose access to funds with this method, especially if you have backups of the wallet.dat and private keys. If the sysadmin uses a block height further back in history, say 12300, that is perfectly fine. The node will do some extra work and take slightly longer. If the sysadmin uses a block height that is too high, such as 12500, it could cause hushd to be unaware of any transactions between block 12345 and 12500. If incoming transactions occurred, they will not be reflected in wallet balances and if outgoing transactions occured, wallet errors may happen when trying to spend shielded funds which hushd things are unspent but were actually spent between blocks 12345 and 12500.

The fix is simple, the sysadmin can just stop and restart hushd with a -rescanheight further back in the past, such as block 12345 or earlier. If the sysadmin does not know an exact block height to use for -rescanheight than a conservative estimate in the past can be used, such as a block height from one month or even one year ago. This will still help avoid much of blockchain history during the rescan and will still result in drastically less downtime while the node is rescanning.

My RPC interface stops responding to requests!

It is sometimes possible for the RPC interface to stop answering requests, due to a bug inherited from BTC/ZEC and made worse by the privacy additions inside of HUSH. There are two different mitigations Hush has added to hushd to deal with this, while the underlying problem is researched more.

PLZ STOP

If the RPC interface stops answering requests, it becomes impossible to run hush-cli stop and sysadmins are forced to use kill to stop the node, which can lead to problems described above which hushd crashing. So Hush developers made a purely disk-based method to ask hushd to stop. A sysadmin simply creates a file called plz_stop in the same directory as wallet.dat. Every 120 seconds, hushd looks for this file and will stop the node if it is found. The file can be of zero size, it's contents do not matter. So for instance:

cd ~/.hush/HUSH3 # or cd ~/.komodo/HUSH3 for legacy locations
touch plz_stop
# hushd will stop within 2 minutes

With the above "trick", you can avoid using kill and hushd will do it's ztx bookkeeping just before it shuts down, which avoids long rescans.

RPC Work Queue

One way to avoid the "RPC deadlock bug" or make it much rarer is to increase the size of the "RPC work queue" in HUSH3.conf like this:

rpcworkqueue=8192

This makes it take much longer and less likely for all RPC "slots" to be deadlocked at the same time, which prevents issuing RPC commands to hushd. Larger numbers can be used, and in general, the more zaddrs and transactions a wallet has, the more likely it will run into a deadlock and need a higher rpcworkqueue value. This method uses a few more kilobytes of RAM to have more slots, which is a good trade to avoid downtime and node maintenance.