4. I don't think memory bandwidth is a limitation in the vast majority of applications. L1 cache hit rates are >90% (>95% even i think) in modern CPUs. L2 will catch much of the rest. Only in certain massively streaming parallel processing applications will you be memory bandwidth limited, but then those tend to be processing heavy as well, so you are probably processing power limited before that.

Prefetching tries to hide latency, it can't hide a lack of memory bandwidth. It increases memory transactions because you don't always prefetch the right data.

Programming wise, it's a matter of where you spend your time optimizing of course. Good programming practices coupled with a good compiler can do very well. Of course for your super critical stuff you will want to go down to ASM, but otherwise, you can probably get better improvements by simply spending your time improving your algorithm rather than hacking at ASM code. Same thing with hardware design. Biggest gains come from improvement in algorithms, the gain typically gets smaller as you move down to a lower level.

It is in the software with millions of lines of code (such as an OS) that you typically see a lot of hand tuned ASM code. You don't have to do it all in ASM from scratch, you can compile it and then hand optimize the ASM code for certain critical stuff.  Obviously it makes sense to do for a function you call a billion times, or for an important inner loop.