In short, use

echo deadline > /sys/block/sda/queue/scheduler

For details, check
http://blog.vacs.fr/index.php?post/2010/08/28/Solving-Linux-system-lockup-when-intensive-disk-I/O-are-performed

And happily, the problem IS much better in newer kernels. In 2.6.37 it was pretty bad, but in 2.6.38, it’s down to a few seconds of stalling before it clears up. Perhaps in 2.6.39 it’ll be milliseconds…?

And it’s solved in recent kernels, so the solution is to upgrade. If you don’t mind the new bugs…

It (IMHO) has always been the case under Linux that when the kernel thinks there is an urgent need to write back dirty pages, it won’t let anything else happen until the last dirty page has been synced to disc.

And yes, that’s lame. It’s a reason why linux is not suited for mission-critical stuff. There is no useful disk QoS. The developers all concentrate on the CPU scheduler — which seems to be a pretty simple task because you just don’t have to implement caching/buffering there because it is done in hardware already.

Maybe your situation could be improved by using a battery backed write cache controller? Another solution could be to limit those processes’ max memory usage, who do a lot of I/Os. Recent kernels support that via the cgroupfs. Interestingly, that also limits cache usage and thereby prevents cache draining by processes who read a lot of data that is never read twice.

Copying files – The system grinds to a halt. It’ll stall for hours.

Creating an endless list in Python – The system grinds to a halt again. No root privileges required.

The disk in question is encrypted with LUKS, and I remember there being a previous problem with LUKS-encrypted swap causing the same kind of stalling and the intense flashing of the disk LED. Perhaps using LUKS and/or LVM and/or other virtual block devices can cause some sort of race condition which stalls the system.

At any rate, life’s too short for this kind of stuff. Time to look at PC-BSD!