December 95 - The New Device Drivers: Memory Matters
The New Device Drivers: Memory Matters
Martin Minow
If you're writing a device driver for the new PCI-based Macintosh computers, you
need to understand the relationship of the memory an application sees to the memory
the hardware device sees. The support for these drivers (which will also run under
Copland, the next generation of the Mac OS) includes the PrepareMemoryForIO
function, as discussed in my article in Issue 22. This single coherent facility connects
the application's logical view of memory to the hardware device's physical view.
PrepareMemoryForIO has proven difficult to understand; this article should help
clarify its use.
If you managed to struggle through my article "Creating PCI Device Drivers" indevelop
Issue 22, you probably noticed that it got rather vague toward the end when I tried to
describe how the PrepareMemoryForIO function works. There are a few reasons for
this: the article was getting pretty long and significantly overdue (the excuse), and I
really didn't understand the function that well myself (the reason). Things are a bit
better now, thanks to the enforced boredom of a very long trip, the need to teach this
algorithm to a group of developers, and some related work I'm doing on the SCSI
interface for Copland.
My previous article showed the simple process of preparing a permanent data area
that might be used by a device driver to share microcode or other permanent
information with a device. This article attacks a number of more complex problems
that appear when a device performs direct memory access (DMA) transfers to or from
a user data area. It also explores issues that arise if data transfers are needed in
situations where the device's hardware cannot use DMA.
A later version of the sample device driver that accompanied the Issue 22 article is
included in its entirety on this issue's CD. Of course, you'll need a hardware device to
use the driver and updated headers and libraries to recompile it. Included is the source
code for the DMA support library (files DMATransfer.c and DMATransfer.h), which
consists of several functions I've written that interact with PrepareMemoryForIO; the
revised sample device driver shows how this library can be incorporated into a
complete device driver for PCI-based Power Macintosh computers.
I'll assume that you've read my earlier article (which you can find on the CD if you
don't have it in print). That article gives an overview of the new device driver
architecture and touches on the PrepareMemoryForIO function, but for a
comprehensive description of the architecture and details about the function, see
Designing PCI Cards and Drivers for Power Macintosh Computers(available from
Apple Developer Catalog). I'll also assume that you're reasonably familiar with the
basic concepts of a virtual memory operating system, including memory pages and
logical and physical addresses; for a brief review, see "Virtual Memory on the
Macintosh.
______________________________
VIRTUAL MEMORY ON THE MACINTOSH
BY DAVE SMITH
Virtual memory on the Macintosh has two major functions: it increases the
apparent size of RAM transparently by moving data back and forth from a disk
file, and it remaps addresses. Of the two, remapping addresses is more
relevant to device driver developers (and, incidentally, much more of a
headache).
When Macintosh virtual memory is turned on, the processor and the code
running on the processor always access logical addresses. A logical address is
used the same way as a physical address; however, the Memory Management
Unit (MMU) integrated into the processor remaps the logical address on the
fly to a physical address if the data is resident in memory. If the data isn't
resident in memory, a page fault occurs; this requires reading the desired data
into memory from the disk and possibly writing other, unneeded data from
memory to the disk to free up space in memory. (This explanation is slightly
simplified, of course.)
Since it would be impractical to have a mapping for each byte address,
memory is subdivided into blocks called pages. A page is the smallest unit that
can be remapped. Memory is broken into pages on page boundaries, which are
page-size intervals starting at 0. The remapping allows physical pages that
are not actually contiguous in physical memory to appear contiguous in the
logical address space.
The Macintosh currently uses a page size of 4096 bytes; however, future
hardware may use a different page size. You should call the GetLogicalPageSize
function in the Driver Services Library to determine the page size if you need
it.
DMA is performed on physical addresses since the MMU of the processor is not
on the address bus that devices use. One of the functions of
PrepareMemoryForIO is to translate logical addresses into physical addresses
so that devices can copy data directly to and from the appropriate buffers.
Many virtual memory systems provide multiple logical address spaces to
prevent applications from interfering with each other. It appears to each
application that it has its own memory system, not shared with any other
application. The Macintosh currently has only one logical address space, but
future releases of the Mac OS will support multiple logical address spaces.
PREPARING MEMORY FOR A USER DATA TRANSFER
At the beginning of a user data transfer (a data transfer on behalf of a program that's
calling into your driver), the device driver calls PrepareMemoryForIO to determine
the physical addresses of the data and to ensure the coherency of memory caches. At the
end of the transfer, the driver calls the CheckpointIO function to release system
resources and adjust caches, if necessary. PrepareMemoryForIO performs three
functions that are necessary for DMA transfers: it locates data in physical memory; it
ensures that the data locations contain the actual data needed or provided by the device;
and, with the help of CheckpointIO, it maintains cache coherence.
Your device driver can call PrepareMemoryForIO from task level, from a software
interrupt, or from the mainline driver function (that is, DoDriverIO). CheckpointIO
can be called from task level, from a software interrupt, or from a secondary
interrupt handler. (For more on the available levels of execution, see "Execution
Levels for Code on the PCI-Based Macintosh.") In a short while, we'll see how the fact
that these functions must be called from particular points affects the transfer process.
______________________________
EXECUTION LEVELS FOR CODE ON THE PCI-BASED MACINTOSH
BY TOM SAULPAUGH
Native code on PCI-based Macintosh computers may run in any of four
execution contexts: software interrupt, secondary interrupt, primary
interrupt, or task. All driver code contexts have access to a driver's global
data. No special work (such as calling the SetA5 function on any of the 680x0
processors) is needed to access globals from any of these contexts.
SOFTWARE INTERRUPT
A software interrupt routine runs within the execution environment of a
particular task. Running a software interrupt routine in a task is like forcing
the task to call a specific subroutine asynchronously. When the software
interrupt routine exits, the task resumes its activities. A software interrupt
routine affects only the task in which it's run; the task can still be preempted
so that other tasks can run. Those tasks, in turn, can run their own software
interrupt routines, and a task running a software interrupt routine can be
interrupted by a primary or secondary interrupt handler.
All software interrupt routines for a particular task are serialized; they don't
interrupt each other, so there's no equivalent to the 680x0 model of nested
primary interrupt handlers.
Page faults are allowed from software interrupt routines. A software
interrupt routine is analogous to a Posix signal or a Windows NT asynchronous
procedure call. A software interrupt routine running in the context of an
application, INIT, or cdev doesn't have access to a driver's global data.
SECONDARY INTERRUPT
The secondary interrupt level is the execution context provided to a device
driver's secondary interrupt handler. In this context, hardware interrupts
are enabled and additional interrupts may occur. A secondary interrupt
handler is a routine that runs in privileged mode with primary interrupts
enabled but task switching disabled.
All secondary interrupt handlers are serialized, and they never interrupt
primary interrupt handlers; in other words, they resemble primary
interrupt handlers but have a lower priority. Thus, a secondary interrupt
handler queued from a primary interrupt handler doesn't execute until the
primary interrupt handler exits, while a secondary interrupt handler queued
from a task executes immediately.
Page faults are not allowed at primary or secondary interrupt level. A
secondary interrupt handler is analogous to a deferred task in Mac OS System
7 or a Windows NT deferred procedure call. Secondary interrupt handlers,
like primary interrupt handlers, should be used only by device drivers. Never
attempt to run application, INIT, or cdev code in this context or at primary
interrupt level.
PRIMARY INTERRUPT
The primary interrupt level (also called hardware interrupt level) is the
execution context in which a device's primary interrupt handler runs. Here,
primary interrupts of the same or lower priority are disabled, the immediate
needs of the device that caused the interrupt are serviced, and any actions that
must be synchronized with the interrupt are performed. The primary
interrupt handler is the routine that responds directly to a hardware
interrupt. It usually satisfies the source of the interrupt and queues a
secondary interrupt handler to perform the bulk of the servicing.
TASK (NON-INTERRUPT)
The task level (also called non-interrupt level) is the execution environment
for applications and other programs that don't service interrupts. Page faults
are allowed in this context.
If the data is currently in physical memory, PrepareMemoryForIO locks the memory
page containing the data so that it cannot be relocated. If the data isn't in physical
memory, PrepareMemoryForIO calls the virtual memory subsystem and a page fault
occurs, reorganizing physical memory to make space in it for the data. After the
transfer finishes, CheckpointIO releases the memory page locks.
PrepareMemoryForIO and CheckpointIO perform an important function related to the
use of caches. A cache is a private, very fast memory area that the CPU can access at
full speed. The processor runs much faster than its memory runs; to keep the
processor running at its best speed, the CPU copies data from main memory to a cache.
Both the PowerPC and the Motorola 68040 processors support caching, although their
implementation details differ. The important point is that a value of a data item in
memory can differ from the value for the same data item in the cache (called cache
incoherence). Furthermore, you have to explicitly tell the PowerPC or 680x0
processor to synchronize the cache with memory.
Normally, the processor hardware prevents cache incoherence from causing data value
problems. However, for some processor architectures, DMA transfers access main
memory independently of the processor cache. PrepareMemoryForIO (for write
operations) and CheckpointIO (for read operations) synchronize the processor cache
with main memory. This means that DMA write operations write the valid contents of