This code is a huge reduction in RAM usage for a HUSH full node. It will reduce RAM usage by roughly 1344 bytes * block_height. With HUSH currently at block height 1314861 this means
1314861 * 1344 / (1024^3) ~ 1.65 GB of RAM savings at the very least. With txindex and other indexes enabled, it may be much more.
Equihash block headers are much larger than RandomX, at 1344 vs 32 bytes for the "solution". Even so, DragonX will get a memory savings of 342741 * 32 / (1024^2) ~ 10.46 MB at the very least.
https://github.com/zcash/zcash/pull/6192
This code is a huge reduction in RAM usage for a HUSH full node. It will reduce RAM usage by roughly 1344 bytes * block_height. With HUSH currently at block height 1314861 this means
`1314861 * 1344 / (1024^3) ~ 1.65 GB` of RAM savings at the very least. With txindex and other indexes enabled, it may be much more.
Equihash block headers are much larger than RandomX, at 1344 vs 32 bytes for the "solution". Even so, DragonX will get a memory savings of `342741 * 32 / (1024^2) ~
10.46 MB` at the very least.
I got an assertion in new code immediately after startup, but I was testing with -rescan -rescanheight and the node had recently crashed. Needs more testing in various situations.
I got an assertion in new code immediately after startup, but I was testing with -rescan -rescanheight and the node had recently crashed. Needs more testing in various situations.
I have two identical nodes, the code seems to be effective, RAM usage on the first node is where the code was upgraded to reduce_memory.
I have two identical nodes, the code seems to be effective, RAM usage on the first node is where the code was upgraded to `reduce_memory`.
@onryo thanks for testing, good to know it works for you.
I would expect a lot more RAM savings. How long was the node running when you took those snapshots of RAM usage? I think it takes some time for the new code to "trim" unnecessary RAM usage after startup, I am not sure how long. hushd on my system uses 5.9GB by itself on the dev branch. Also, it's not clear what is being shown, Virtual or Resident memory. Use htop or top and press M to sort by memory, that will give more useful stats.
@onryo thanks for testing, good to know it works for you.
I would expect a lot more RAM savings. How long was the node running when you took those snapshots of RAM usage? I think it takes some time for the new code to "trim" unnecessary RAM usage after startup, I am not sure how long. hushd on my system uses 5.9GB by itself on the dev branch. Also, it's not clear what is being shown, Virtual or Resident memory. Use `htop` or `top` and press `M` to sort by memory, that will give more useful stats.
@onryo the code seems to work for you but coredumps for @fekt and I, so it would be useful to know what you are doing differently than us. Did you do a fresh sync or were you using the master branch (or last release) before switching to this branch?
@onryo the code seems to work for you but coredumps for @fekt and I, so it would be useful to know what you are doing differently than us. Did you do a fresh sync or were you using the master branch (or last release) before switching to this branch?
I was running the latest master branch, then I stopped the node, pulled the new commits, built and run with -reindex. RAM usage stays the same as in the above snapshot and to be fair 900MB reduction is a huge deal as well.
I was running the latest master branch, then I stopped the node, pulled the new commits, built and run with `-reindex`. RAM usage stays the same as in the above snapshot and to be fair 900MB reduction is a huge deal as well.
@onryo ok, good to know. I recently pushed another commit which should speed up startup by 3X and reduce disk io of startups by a lot, please test that as well.
My goal is to improve the code so a -reindex is not needed. If phashBlock is empty, the code should be smart enough to populate it with the CBlockHeader.GetHash() method
@onryo ok, good to know. I recently pushed another commit which should speed up startup by 3X and reduce disk io of startups by a lot, please test that as well.
My goal is to improve the code so a `-reindex` is not needed. If `phashBlock` is empty, the code should be smart enough to populate it with the `CBlockHeader.GetHash()` method
@onryo it also would be good to know: Do you get an assertion/crash if you do not use -reindex ? Note that the assertion might only be triggered when switching branches, so we need to know if you switched branches since the last sync, and if -reindex was used. This is important to test, because for the average user, they will be syncing from the previous release, then they will be doing the equivalent of "switching branches" if they upgrade to a new release which has the reduce_memory code.
@onryo it also would be good to know: Do you get an assertion/crash if you do not use `-reindex` ? Note that the assertion might only be triggered when switching branches, so we need to know if you switched branches since the last sync, and if `-reindex` was used. This is important to test, because for the average user, they will be syncing from the previous release, then they will be doing the equivalent of "switching branches" if they upgrade to a new release which has the `reduce_memory` code.
this branch has been merged to dev, please test thoroughly! I am seeing roughly 2GB memory savings. Open up a new issue for any problems that may arise
this branch has been merged to dev, please test thoroughly! I am seeing roughly 2GB memory savings. Open up a new issue for any problems that may arise
duke
https://github.com/zcash/zcash/pull/6192
This code is a huge reduction in RAM usage for a HUSH full node. It will reduce RAM usage by roughly 1344 bytes * block_height. With HUSH currently at block height 1314861 this means
1314861 * 1344 / (1024^3) ~ 1.65 GBof RAM savings at the very least. With txindex and other indexes enabled, it may be much more.Equihash block headers are much larger than RandomX, at 1344 vs 32 bytes for the "solution". Even so, DragonX will get a memory savings of
342741 * 32 / (1024^2) ~ 10.46 MBat the very least.Code has been pushed on the
reduce_memorybranch, needs testingI got an assertion in new code immediately after startup, but I was testing with -rescan -rescanheight and the node had recently crashed. Needs more testing in various situations.
I have two identical nodes, the code seems to be effective, RAM usage on the first node is where the code was upgraded to
reduce_memory.@onryo thanks for testing, good to know it works for you.
I would expect a lot more RAM savings. How long was the node running when you took those snapshots of RAM usage? I think it takes some time for the new code to "trim" unnecessary RAM usage after startup, I am not sure how long. hushd on my system uses 5.9GB by itself on the dev branch. Also, it's not clear what is being shown, Virtual or Resident memory. Use
htoportopand pressMto sort by memory, that will give more useful stats.@onryo the code seems to work for you but coredumps for @fekt and I, so it would be useful to know what you are doing differently than us. Did you do a fresh sync or were you using the master branch (or last release) before switching to this branch?
I was running the latest master branch, then I stopped the node, pulled the new commits, built and run with
-reindex. RAM usage stays the same as in the above snapshot and to be fair 900MB reduction is a huge deal as well.@onryo ok, good to know. I recently pushed another commit which should speed up startup by 3X and reduce disk io of startups by a lot, please test that as well.
My goal is to improve the code so a
-reindexis not needed. IfphashBlockis empty, the code should be smart enough to populate it with theCBlockHeader.GetHash()method@onryo it also would be good to know: Do you get an assertion/crash if you do not use
-reindex? Note that the assertion might only be triggered when switching branches, so we need to know if you switched branches since the last sync, and if-reindexwas used. This is important to test, because for the average user, they will be syncing from the previous release, then they will be doing the equivalent of "switching branches" if they upgrade to a new release which has thereduce_memorycode.Merging this branch is blocked on #292. The latest code uses
CBlockHeader.GetHash()ifphashBlock == NULLbut more is needed.this branch has been merged to dev, please test thoroughly! I am seeing roughly 2GB memory savings. Open up a new issue for any problems that may arise