reply on: How to set up a Raspberry Pi node \ stacker news

pull down to refresh

326 sats \ 6 replies \ @Murch 28 Feb \ parent \ on: How to set up a Raspberry Pi node lightning bitcoin_beginners

Let me caveat all the following with saying that I’m not much of an expert for Raspberry Pis, disk performance, or the nitty-gritty details of IBD, but this is my understanding:

Under no circumstances should one use an SD card or thumb drive as read-write disk, it’ll run out of cycles very quickly (learned that one myself over ten years ago).
One thing that I have read several times is that the Raspberry Pi itself cannot deliver enough power for external HDDs to perform properly. My understanding is that using a powered USB hub will make a big difference and can drastically improve disk speed.
I’ve seen some people recommend that the dbcache should be set to a value larger than the UTXO set. This would lead to swapping (i.e., writing memory contents to disk) on anything but the most powerful machines. Instead, the dbcache value should not be set higher than 50–75% of the RAM the device has, less if the device generally has low RAM or if bitcoind is not the only thing you’re running.
If you are running only few services on your device, and have enough RAM, you may want to turn off swapping altogether. I’m not sure whether 8 GB would be enough for that.
For the Raspberry Pi 4 there was a known issue with it not playing nice with some types of disks, something about a driver not being implemented properly on those disks. Given that it was reported several years ago, this is hopefully resolved now: https://forums.raspberrypi.com/viewtopic.php?f=28&t=245931
Generally, writing the blocks is not an issue, but loading UTXOs back in the cache from disk can be slow, especially on HDDs. If you have an SSD, but it is not big enough for the whole blockchain, you can store the UTXO set on the small SSD drive (even 100 GiB would be enough) and store the blockchain on an HDD. This can be achieved by setting datadir to the SSD and blocksdir to the HDD in the bitcoin.conf.
SSDs that are close to full slow down. If your disk is more than 85% or even 90% full, it lacks the space for efficient data management. You should consider either getting a bigger disk, adding an HDD to move the blocks there, or pruning.
If you are syncing an older node version, the default assumevalid block will be set further in the past from the chaintip. Full script validation may make CPU load the bottleneck of a weaker device. You may be able to shave off days from IBD by setting the assumevalid block to something close to the current block height (e.g. 1000 blocks below the chaintip).

I could imagine that there are also issues with some cheaper adapters for external drives, but I know nothing concrete there. There were also several IBD performance improvements in v30, and more are coming in v31. Unfortunately, the one I’m most excited will miss v31, but will hopefully be in v32. It would be great if we also get Swiftsync in v32, but we’ll have to see about that.

Maybe @optimism can comment on what I wrote, judging from other comments in this topic, @optimism may know better about a lot of these topics than me.

168 sats \ 4 replies \ @optimism 28 Feb

Hey Murch,

All this is taken care of in my repro, except your point #2. I had tested with the "raspibolt" recommendation of 2GB. The problem on rpi4 is txin verification:

- Verify 9995 txins: 34716.35ms (3.473ms/txin) [51906.08s (65.15ms/blk)]

but it's unlikely to be the disk or the cache! I cannot reproduce on an Odroid M2 with exactly the same disk (as in I plugged it out of the rpi4 and into the odroid) and cache settings.

I've seen some traces from others where the main issue are secp256k1 functions and I remember some changes have been made to replace hand coded ASM with "clang is better". I'm trying to find a few days to get into the weeds on this one and just create a bunch bench cases to see if I can pinpoint it without needing to pprof a live mainnet process. Worst case I'll build a regtest script and do it that way. Anything to not have to profile a mainnet bitcoind haha.

103 sats \ 3 replies \ @Murch 28 Feb

txin verification would indeed make me think that it’s the retrieval of UTXO cache misses. I’ve been hearing that the disk interface of Raspberry Pis is pretty slow, so the disk itself might not be the issue, but rather the speed at which the Pi talks to the disk?

the main issue are secp256k1 functions

Is this during the full validation part of IBD after passing the assumevalid block? Otherwise, I wouldn’t expect many calls to secp.

45 sats \ 2 replies \ @optimism 28 Feb

It starts slowing down before that (around block 790k), but it gets real nasty after it (840k+ on 29.x). I'm not 100% convinced that it is only secp - am natively compiling bench right now for 26, 27, 28, 29 and 30 - see if I can spot a regression

103 sats \ 1 reply \ @Murch 28 Feb

Mh, the default assumevalid block was 886'157 in v29.0.

Oh, you know what? I have a theory. There was a period of time when people started to churn UTXOs to sift for rare sats. Perhaps the slowdown coincides with a big increase of old UTXOs being spent?

Looking at a UTXO Age band chart, the number of coins held in UTXOs older than one year peaked in November 2023. I haven’t found data on UTXO counts itself yet.

45 sats \ 0 replies \ @optimism 28 Feb

There was a period of time when people started to churn UTXOs to sift for rare sats.

This is plausible, as well as massive consolidation of near-dust. The first ones I found that were slow going were blocks that were heavy on consolidation txs.

45 sats \ 0 replies \ @SimpleStacker 28 Feb

Hmm, very interesting.

I think I can rule out #1 (power) because I tried it with a separately powered external SSD, and I still had the same IBD slowness. \

I can also rule out #5 because the SSD was not close to full, either.

All the others, I'd have to check. I don't remember what I set dbcache at, but I do remember playing around with it, setting both higher and lower, and not noticing any meaningful speed up in IBD.