pull down to refresh
I would not recommend setting up a node on Raspberry Pi anymore. There is something going on which makes initial block download extremely slow on the RPi
The way a guy from Start9 explained it to me was that memory usage increases dramatically at some point as it processes full blocks. And as the Pi starts paging more and more to offload to swap, it gets into paging hell where the throughput of the USB port can't effectively keep up with the traffic being shoveled between swap and memory.
I think that's what happened to my Pi4. So what's the bottle neck? The usb cable's transfer speed?
The thing I didn't understand was why it was using swap so much if the memory didn't seem like it was being pressured (had 8gb memory)
Let me caveat all the following with saying that I’m not much of an expert for Raspberry Pis, disk performance, or the nitty-gritty details of IBD, but this is my understanding:
- Under no circumstances should one use an SD card or thumb drive as read-write disk, it’ll run out of cycles very quickly (learned that one myself over ten years ago).
- One thing that I have read several times is that the Raspberry Pi itself cannot deliver enough power for external HDDs to perform properly. My understanding is that using a powered USB hub will make a big difference and can drastically improve disk speed.
- I’ve seen some people recommend that the
dbcacheshould be set to a value larger than the UTXO set. This would lead to swapping (i.e., writing memory contents to disk) on anything but the most powerful machines. Instead, thedbcachevalue should not be set higher than 50–75% of the RAM the device has, less if the device generally has low RAM or ifbitcoindis not the only thing you’re running.
If you are running only few services on your device, and have enough RAM, you may want to turn off swapping altogether. I’m not sure whether 8 GB would be enough for that. - For the Raspberry Pi 4 there was a known issue with it not playing nice with some types of disks, something about a driver not being implemented properly on those disks. Given that it was reported several years ago, this is hopefully resolved now: https://forums.raspberrypi.com/viewtopic.php?f=28&t=245931
- Generally, writing the blocks is not an issue, but loading UTXOs back in the cache from disk can be slow, especially on HDDs. If you have an SSD, but it is not big enough for the whole blockchain, you can store the UTXO set on the small SSD drive (even 100 GiB would be enough) and store the blockchain on an HDD. This can be achieved by setting
datadirto the SSD andblocksdirto the HDD in thebitcoin.conf. - SSDs that are close to full slow down. If your disk is more than 85% or even 90% full, it lacks the space for efficient data management. You should consider either getting a bigger disk, adding an HDD to move the blocks there, or pruning.
- If you are syncing an older node version, the default
assumevalidblock will be set further in the past from the chaintip. Full script validation may make CPU load the bottleneck of a weaker device. You may be able to shave off days from IBD by setting theassumevalidblock to something close to the current block height (e.g. 1000 blocks below the chaintip).
I could imagine that there are also issues with some cheaper adapters for external drives, but I know nothing concrete there. There were also several IBD performance improvements in v30, and more are coming in v31. Unfortunately, the one I’m most excited will miss v31, but will hopefully be in v32. It would be great if we also get Swiftsync in v32, but we’ll have to see about that.
Maybe @optimism can comment on what I wrote, judging from other comments in this topic, @optimism may know better about a lot of these topics than me.
Hey Murch,
All this is taken care of in my repro, except your point #2. I had tested with the "raspibolt" recommendation of 2GB. The problem on rpi4 is txin verification:
- Verify 9995 txins: 34716.35ms (3.473ms/txin) [51906.08s (65.15ms/blk)]but it's unlikely to be the disk or the cache! I cannot reproduce on an Odroid M2 with exactly the same disk (as in I plugged it out of the rpi4 and into the odroid) and cache settings.
I've seen some traces from others where the main issue are secp256k1 functions and I remember some changes have been made to replace hand coded ASM with "clang is better". I'm trying to find a few days to get into the weeds on this one and just create a bunch bench cases to see if I can pinpoint it without needing to pprof a live mainnet process. Worst case I'll build a regtest script and do it that way. Anything to not have to profile a mainnet bitcoind haha.
txin verification would indeed make me think that it’s the retrieval of UTXO cache misses. I’ve been hearing that the disk interface of Raspberry Pis is pretty slow, so the disk itself might not be the issue, but rather the speed at which the Pi talks to the disk?
the main issue are secp256k1 functions
Is this during the full validation part of IBD after passing the assumevalid block? Otherwise, I wouldn’t expect many calls to secp.
Mh, the default assumevalid block was 886'157 in v29.0.
Oh, you know what? I have a theory. There was a period of time when people started to churn UTXOs to sift for rare sats. Perhaps the slowdown coincides with a big increase of old UTXOs being spent?
Looking at a UTXO Age band chart, the number of coins held in UTXOs older than one year peaked in November 2023. I haven’t found data on UTXO counts itself yet.
Hmm, very interesting.
I think I can rule out #1 (power) because I tried it with a separately powered external SSD, and I still had the same IBD slowness. \
I can also rule out #5 because the SSD was not close to full, either.
All the others, I'd have to check. I don't remember what I set dbcache at, but I do remember playing around with it, setting both higher and lower, and not noticing any meaningful speed up in IBD.
I only have a 1TB HDD, and since the Bitcoin blockchain is already close to that size, I’m hesitant to install a full node on it. I also don’t want to buy a new 2TB drive right now, is too expensive.
I came across this post #1265438 about lightning.pub, which uses Neutrino. Has anyone else here tried it?
I concur for a Pi4. Pi5 works better. Thanks for reminding me to spend time on it.
I right now have an Odroid M2 with an 8TB nvme for a total cost of ~500k sats that works really well, but it's hard to secure the OS for it due to the slow maintenance of the custom firmware vs the distro obsolescence cycles, so I'm still not going to recommend it.
Pi4 4MB RAM is what I got, isn't the best but is something to play with. Which OS to install is what currently evaluating.
If you intend to do playing on mainnet, I recommend to not attempt to run bitcoind on it for now, until either we have been able to locate the regression in tx processing that causes it to grind to a halt, or have verified that running another main chain client . Instead, you can use lnd with neutrino for starters.
However, since you're hinting that you want this to be a routing node, I'd suggest not playing on mainnet straight, but instead play on mutinynet or testnet4 for starters, and learn about counterparty force closure, what happens when your power dies, state corruption because pi4 is hell when it comes to write guarantees... and so on.
Maybe one of our resident experts on this, like @justin_shocknet or @Filiprogrammer can chime in though - I'm not an expert on LN deployment at all.
another main chain client .
@sox I have seen a couple of times now that I select a word on my not-goog android and then it disappears, which seems to have happened here with "functions", lol.
I'll spend some time trying to repro (it never happens when I try to repro straight) if you expect that that's an issue with something other than just my phone
The harsh reality is that Lexical, the library we used to develop our new editor, was never seriously tested on Android devices. All tests were conducted on Gboard ... and it shows.
@rblb did an awesome job with his patches, but I truly wonder when we'll have a stable experience on non-Google Androids lol.
Ok rant aside, I'll reproduce and find a fix. Never stop reporting these bugs, can't thank you guys enough!
I wouldn't recommend neutrino for a a heavy routing node, it has to make some assumptions about channels since it doesn't have the full chain picture. Neutrino is bet for personal/SMB with a few channels.
A Pi4 likewise is fine for a personal/SMB node, but I wouldn't host a large amount of coin across many channels for routing.
For routing, you really want something with nvme storage at a minimum, ideally redundant storage, and a backup battery to reduce the risk of a corrupted channel state from a failed write.
Performance issues on Pi4's are usually due to inadequate RAM, 4GB is the absolute bare minimum for bitcoind+lnd.
btcd in my experience is a little less memory hungry and is actually an LND library, a better back-end if you're only planning to use it for a Lightning node.
You can also use assumevalid with either bitcoind or btcd up to a recent consensus block to bypass most of the heavy compute at initial sync. Forward blocks will still fully validate/enforce.
The consensus OS for PI's is Armbian afaik, that's just Debian compiled for that hardware and receives all of the same updates. I recommend to enterprise clients not doing auto-updates and instead putting nodes on a militarized network with manual release updates.
Pi's aren't a particularly good value imo since you can get a used laptop with nvme storage, beefier chip, and included battery for less than a few hundred bucks. Never have to fuck with hooking up a KVM is gravy on top.
Raspibolt is obsolete, use ramix instead: https://ramix.minibolt.info/
But yes for other arm boards rpi4 is also obsolete for a node.
No, I've been using it without any problems. Of ocurse it's for someone that has a brian and use with experience on linux. I have seen a bug here or there but it was like an error of downloading LND x86_64 instead of arm64. And it's a guide not something you have to follow 100% you can always customize and do whatver you want with it.
cmon man, let SS to answer all questions here. He's the expert now.
I would not recommend setting up a node on Raspberry Pi anymore. There is something going on which makes initial block download extremely slow on the RPi: #1262279
I am not sure the technical reason for it, but multiple people have confirmed this issue: https://github.com/raspibolt/raspibolt/issues/1482