Mmap is a way to provide file IO in a nice simple manner. Create a buffer area, and as you read/write into that buffer, this is reflected physically into the file.
Oversimplification, but this is basically what it is.
In most operating system, mmap makes direct use of the paging paths in the kernel.
Why am I writing about this?
Because the paging paths are some of the slowest paths in modern kernels, typically doing IO a page at a time. In the case of Linux, this is 4k bytes at a time for most systems.
Unfortunately, it gets worse.
Tuning this path would be possible if we could read/write larger pages, or groups of pages. Linus and others have pushed hard against the concept of using larger than 4k byte pages. Yeah, there is some question as to whether or not he wrote that.
But with small pages comes small IOs in the paging path. Which. Are. Horrible. On. Real. World. Disks.
So mmap performance, like it or not, is gonna be sub par for the forseeable future.
For most people, this isn’t an issue. For the apps that depend upon it, because they designed their code around it, their performance is also going to be sub par.
Our units do great with direct IO on uncached/uncachable loads (this means our disks are really fast), and cached loads. Its hard to tune the IO system for cases where the kernel is the rate limiting factor. Theres not much we can do.
Well, in the face of the unlikely to ever change mmap situation, we can educate code writers and ISVs on how to get good performance. I’ve heard Linus dislikes direct IO as well. Thats a shame. Direct IO is, in many performance and use cases we run into, far superior to cached IO. All you need are fast disks, and then cache and cache management become the bottleneck.
A bottleneck at a gigabyte per second is much better than a bottleneck at under 100 MB/s. But its still a bottleneck.
Gonna poke around in ./arch/x86/mm/mmap.c and see if there is any hope for tuning this old technology.
As a lesson to ISV coders, this is what happens when vendors slip you info on how to get the best performance … on their systems. Don’t build business cases around their tech. Abstract it, so you can change later.