Memory mapped IO and files
The lower-level parts of computers
General: Computer power consumption · Computer noises Memory: Some understanding of memory hardware · CPU cache · Flash memory · Virtual memory · Memory mapped IO and files · RAM disk · Memory limits on 32-bit and 64-bit machines Related: Network wiring notes - Power over Ethernet · 19" rack sizes Unsorted: GPU, GPGPU, OpenCL, CUDA notes · Computer booting
|
Note that
- memory mapped IO is a hardware-level construction, while
- memory mapped files are a software construction -- because files are.
Memory mapped files
Memory mapping of files is a technique (OS feature, system call) that pretends a file is accessible at some address in memory.
When the process accesses those memory locations, the OS will scramble for the actual contents from disk.
...it can also be cached -- whether the OS does that at all depends a little on the OS and details(verify), whether it is in RAM depends on whether it was recently accessed by something.
For caching
In e.g. linux you get that interaction with the page cache, and the data stays cached as long as the RAM isn't used by other things.
This can also save memory - in that without memory mapping,
compared to the easy choice of manually cacheing the entire thing in your process.
With mmap you may cache only the parts you use, and if multiple processes want this file, you may avoid a little duplication.
The fact that the OS can flush most or all of this data can be seen as a limitation or a feature - it's not always predictable, but it does mean you can deal with large data sets without having to think about very large allocations, and how those aren't nice to other apps.
Most kernel implementations allow multiple processes to mmap the same file -- which effectively shares memory, and probably one of the simplest in a protected mode system. (Some methods of Inter-Process communication work via mmapping)
Not clobbering each other's memory is still something you need to do yourself.
The implementation, limitations, and method of use varies per OS / kernel.
Often relies on demand paging to work.
Memory mapped IO
Map devices into memory space (statically or dynamically), meaning that memory accesses to those areas will actually be served via IO accesses (...that you can typically also do directly).
This mapping is made and resolved at hardware-level, and only works for DMA-capable devices (which is many).
It seems to often be done to have a simple generic interface (verify) - it means drivers and software can avoid many hardware-specific details.
See also:
DMA
Direct Memory Access means that hardware can be asked to copy bytes from one memory range to another.
DMA is independent enough at hardware level that its transfers can work at high clock rates and so fairly high throughput - and often just as importantly, the CPU isn't occupied by sending all this data back and forth and is free to do other things, meaning that from a CPU-time perspective, DMA transfers are relatively free.
First party DMA, a.k.a. bus mastering, means any device on the bus could potentially initiate a DMA transaction.
Third party DMA means there is a system DMA controller actually does the transfer.
PATA, SATA, and USB are themselves third party DMA (you can ask something else to do bus transfers), their host controllers's connection to the system is first party DMA.
Aside from memory-to-memory use, it also allows memory-to-peripheral copies (if a specific supporting device is memory mapped(verify)).
While first party DMA could theoretically have any device directly talk to any other, it is much more generic (and thereby more compatible) to have the OS drivers deal with just one device, meaning that in practice the'll all DMA to RAM (often also because what the driver wants to do with it next).
DMA is often triggered by DRQ (similar in concept to IRQs, but triggering only a copy, rather than arbitrary code), and typically coordinates only smaller-chunk copies (each copy operation does not take too much time or memory).
The details look intimidating at first, but mostly because they are low-level. The idea is actually relatively simple.
And yes, you do often have to worry about avoiding race conditions,
though there are some standard-ish tricks.