This is incredibly true. It's a common mistake because it's hard to determine what matters. The best way I've found to determine what matters is to build something fast to the point where its quality becomes so poor that it's nearly unapproachable. If it matters, speed elsewhere will suffer from its poor quality.

This was the case with our wallets and payments code. Until we built the fast versions, it wasn't clear how much implementing them well mattered until improving other things in the product became incredibly hard. This code mattered so much that even our good but fast-as-we-could versions weren't good enough. The good version of this component was impassible and everything we wanted to do required us passing over it constantly.

Still, and especially if you don't have much of a legacy to maintain/overhaul, you can continue to move fast by deploying pieces of what matters in a progressive fashion.