Prune witnesses for notes spent more than X blocks ago #161

This is something we may want, which will improve the speed of syncing and rescanning:

82361aebc3

Upon reviewing this code, it seems reasonable and not too dangerous, except for it adding a new assert() which will crash the node with no error message if that case every happens (maybe due to a corrupt wallet/etc). At the very least an error should be printed to debug.log in that case, and instead of an assert, that iteration of the loop should be skipped.

Additionally, Hush source code no longer has CopyPreviousWitnesses since Sapling consolidation was added, which will make porting this code a bit more challenging. It was remove/refactored into different functions in commit 6c48397cf3 in Feb 2020.

Started working on this on the witness_cache branch

Not only have some functions been renamed/refactored but the entire algorithm for updating/maintaining the witness cache has completely diverged.

Currently our code does not keep track of "spentness" of a note/zutxo, which is kept track of in the spentHeight variable that I added in the witness_cache branch. We need something like UpdateSpentHeightAndMaybePruneWitnesses but it's not clear how to implement it without completely changing our witness cache algorithm.

Once we know a spentHeight for a note/zutxo we can prune witnesses for notes spent more than WITNESS_CACHE_SIZE blocks ago. This means less witnesses to keep track of by a large amount, which means the BuildWitnessCache phase of rescanning will be much faster.

Currently our code keeps track of witnesses for all time while the latest Zcash code only keeps track of the last WITNESS_CACHE_SIZE blocks, which is 110 blocks. So we keep track of an ever-growing list of witnesses in all blocks while this other code only keeps track of a small amount of recent witnesses. This is partially why "Building Witnesses" after a rescan is so damn slow and why it gets slower and slower for each block as the witness cache size increases. This also leads to large wallet.dat sizes since the witness cache is stored inside wallet.dat .

One possible way forward is to write a new function which does nothing but prune witnesses from notes/zutxos spent more than 110 blocks ago. It could be periodically run automatically by the full node and/or have an RPC interface to be run manually.

witness_cache branch now has some code which attempts to port UpdateSpentHeightAndMaybePruneWitnesses . No idea if it works, needs a lot of testing. Currently thinking of the best ways to test it.