I didn't understand before, but now I do, why "realtime" stuff is incompatible with "virtual memory" stuff.

I have two words and a gutteral sound for you: priority inversion.  Gaaaaaaaaaaaaaaaarghgggh.

kswapd: a non-realtime process that needs to swap pages from your realtime process in and out. That doesn't work, of course, leading to deadlocks.  You might think you could solve it by using mlockall(MCL_FUTURE), which actually does mostly work, right up until you try to open a file. Then because of the prior mlockall(), it decides to swap in all the pages right away. But that's kswapd's job.  Maybe with some help from nfsiod or mtdblock0 or one of the many other non-realtime-priority kernel threads.

Even if your realtime thread doesn't open any files - and it doesn't because we're not crazy - it doesn't matter.  A non-realtime-priority thread in the same program can open a file, and you end up with pages mapped into your address space, and those pages need to be paged in right away because of mlockall, and the kernel helpfully chooses your realtime thread to do it, because after all, we don't want none of that priority inversion where your realtime thread can't continue because a low-priority thread is blocked on reading your pages.

Except it blocks anyway, because the kernel threads responsible for actually doing such things are not realtime threads.

My solution: if you have a page fault in your realtime process, nanosleep() for a bit to let other people run.  Sure, it's horrible, but it was worse before.  And actually trying to fix the priority inversion directly (eg. by boosting the priority of dependent processes) is a losing battle, because you don't know which processes you're dependent on, in the general case.

Sigh.