Self Modifying Code
Volume Number: 8
Issue Number: 4
Column Tag: Article Rebuttal
Self Modifying code is a No-No! 
A better way to do an event patch without self-modifying code or
By Scott T. Boyd, Apple Computer, Inc. and Mike Scanlin, MacTutor
Regular Contributing Author
Note: Source code files accompanying article are located on MacTech CD-ROM orsource code disks.
Mike Scanlin’s article “Rotten Apple INIT for April Fool’s” brings up a minor
but essential point. The GetNextEvent patch looks like:
;1
@first
Lea @exitAddress, A0
Move.L (SP)+,(A0)
Lea @ eventRecPtr,A0
Move.L (SP),(A0)
Pea @tailPatch
DC JmpInstruction
@origTrap
NOP
NOP
The NOPs get replaced with the original GetNextEvent trap address by the
installation code:
;2
Lea @origTrap,A1
Move.L A0,(A1)
Now, consider the processor instruction cache. It’s a piece of the processor
which remembers what’s at a set of memory locations. It does this so the CPU won’t
have to do a memory access for recently referenced instructions. This is designed to
save time. It’s a neat hardware feature.
However, the Macintosh system software doesn’t make a distinction between code
and data. That’s different from OSs like Unix, which keep code and data in separate
address spaces. When Mike’s code installs the original trap address with the Move.L
A0,(A1), it’s putting an address into the middle of a piece of code. Unfortunately, the
cache which records the new value is the data cache.
The instruction cache has no clue that instructions just changed. This is one way
of doing what’s commonly called “self-modifying code”.
Self-modifying code is, in general, a bad thing to write. Apple has long
discouraged, and continues to discourage self-modifying code.
This is bad in this case because the processor, if it were to execute this code
right away, might believe (for some unspecified reason) that those memory locations
were cached in the instruction cache. If it did, it would pull whatever had been in that
location the last time code was executed from that spot, then try to execute whatever
was there. The odds that the instruction cache held the value you just put into the data
cache are not in your favor.
Contrast that approach with this approach:
;3
@first
...
Move.L @origTrap,A0
JMP.L (A0)
...
/* branch around this, or put it somewhere else, but don't let the PC run
through here */
@origTrap
NOP
NOP
While this approach still stores the old address into a piece of code, it’s never
referenced as code by the processor. It’s treated specifically as data. The instruction
cache never comes into play since the original address is moved as data.
Yet another approach, which also saves a register:
;4
@first
...
Move.L @origTrap,-(SP)
RTS
...
TN #261: “Cache As Cache Can” discusses this topic in more detail, e specially
with regard to moving whole chunks of code around.
As it happens, the caches are almost certainly flushed before this particular
eight bytes ever get loaded for execution, but that’s a happy coincidence, and not
something you should rely on. What’s happening is that we have made several traps
flush the caches (guaranteeing that there won’t be any misunderstanding about
something being in the instruction cache when it’s not), but we may change our minds
about which traps should flush, and when. You shouldn’t count on any given trap’s
current cache-flushing behavior.
One final consideration. Putting data into code does not work if code is ever
write-protected, and that may happen one day. So where can you put something when
you can’t allocate any global storage (e.g., PC-relative data or low-memory globals
with a fixed address)? You can use NewGestalt to register a new selector. When you
call Gestalt, it can return a value which is actually a pointer (or handle) to your global
data. This technique won’t work well if you can’t afford to make the trap call (like
from some time-sensitive routine you’ve patched), but it works nicely if you have the
time and you want to avoid putting data into your patch code.
Scott T Boyd, Apple Computer, Inc.
Mike Scanlin Says
Scott's point about stale code in the instruction cache is well taken and I deserve
a thumping for having written it. I made the poor judgement call that it wouldn't matter
in this case because I expected the instruction cache to be flushed between the time the
patch installation code finished and the first time the patch code was executed. I hang my
head in shame.
As partial retribution (and to satisfy a few requests for a non-assembly
version) I have written a trap patching shell in C that doesn't use any self-modifying
code (see listing below). It obeys all of the rules except for the one about storing data
into a code segment (Scott's solutions have this problem, too, as he mentions). Until we
have write-protected code segments, this will not be a problem.
Mike Scanlin
/*********************************************************
* PatchGNE.c:
* This INIT installs a patch on GetNextEvent and
* SystemEvent that intercepts keyDown and autoKey events.
* For this example, the intercepted key events are
* converted to lower case if both the capsLock key and the
* shiftKey are down (thus making the Mac keyboard behave
* like an IBM keyboard). However, you can use this shell to
* do generalized event intercepting as well as generalized
* trap patching (with no asm and no self-modifying code).
* If your patches need globals, put them in the
* PatchGlobals struct and initialize them in main.
* In Think C, set the Project Type to Code Resource, the
* File Type to INIT, the Creator to anything, the Type to
* INIT, the ID to something like 55 (55 will work but it
* doesn't have to be 55), turn Custom Header ON and Attrs
* to 20 (purgeable) and Multi Segment OFF.
*
* Mike Scanlin. 16 May 1992.
*********************************************************/
#include "Traps.h
/**********************************************************
* typedefs
*********************************************************/
typedef pascal short (*GNEProcPtr)(short eventMask,
EventRecord *theEvent);
typedef pascal short (*SEProcPtr)(EventRecord *theEvent);
typedef struct PatchGlobals {
GNEProcPtr pgOldGNE;
SEProcPtr pgOldSE;
} PatchGlobals, *PatchGlobalsPtr;
/**********************************************************
* prototypes
*********************************************************/
void main(void);
void StartPatchCode(void);
pascal short MyGetNextEvent(short eventMask, EventRecord
*theEvent);
pascal short MySystemEvent(EventRecord *theEvent);
void CheckKeyCase(EventRecord *theEvent);
void EndPatchCode(void);
/**********************************************************
* main:
* Gets some memory in the system heap and installs the GNE
* and SE patches (as well as allocating and initializing
* the patc 8.4 Self Modifying Codeutine that gets
* executed at startup time (by the INIT mechanism).
*
* The block of memory that main allocates will look like
* this when main has finished:
*
* +--------------------+
* | PatchGlobals |
* +--------------------+
* | StartPatchCode() |
* GNE trap addr -> +--------------------+
* | MyGetNextEvent() |
* SE trap addr -> +--------------------+
* | MySystemEvent() |
* +--------------------+
* | CheckKeyCase() |
* +--------------------+
* | EndPatchCode() |
* +--------------------+
*
*********************************************************/
void main()
{
Ptr patchPtr;
PatchGlobalsPtr pgPtr;
long codeSize, offset;
/* try and get some memory in the system heap for code
and globals */
codeSize = (long) EndPatchCode - (long) StartPatchCode;
patchPtr = NewPtrSys(codeSize + sizeof(PatchGlobals));
if (!patchPtr)
return; /* out of memory -- abort patching */
/* initialize the patch globals at the beginning
of the block */
pgPtr = (PatchGlobalsPtr) patchPtr;
pgPtr->pgOldGNE = (GNEProcPtr)
GetTrapAddress(_GetNextEvent);
pgPtr->pgOldSE = (SEProcPtr)
GetTrapAddress(_SystemEvent);
/* move the code into place after the globals */
BlockMove(StartPatchCode, patchPtr +
sizeof(PatchGlobals), codeSize);
/* set the patches */
patchPtr += sizeof(PatchGlobals);
offset = (long) MyGetNextEvent - (long) StartPatchCode;
SetTrapAddress((long) patchPtr + offset, _GetNextEvent);
offset = (long) MySystemEvent - (long) StartPatchCode;
SetTrapAddress((long) patchPtr + offset, _SystemEvent);
}
/**********************************************************
* StartPatchCode:
* Dummy proc to mark the beginning of the code for the
* patches. Make sure all of your patch code is between
* here and EndPatchCode.
*********************************************************/
void StartPatchCode()
{
}
/*********************************************************
* MyGetNextEvent:
* Tail patch on GetNextEvent.
*
* The reason this returns a short instead of a Boolean is
* because we need to make sure the low byte of the top word
* on the stack is zero because some programs do a Tst.W
* (SP)+ when this returns instead of Tst.B (SP)+ like they
* should (which is technically their bug but, we might as
* well work around it since it's not hard).
*
* If you want to eat the event and not pass it on to the
* caller then set returnValue to zero.
*********************************************************/
pascal short MyGetNextEvent(short eventMask,
EventRecord *theEvent)
{
PatchGlobalsPtr pgPtr;
short returnValue;
/* find our globals */
pgPtr = (PatchGlobalsPtr) ((long) StartPatchCode -
sizeof(PatchGlobals));
/* call original GNE first */
returnValue = (*pgPtr->pgOldGNE)( eventMask, theEvent);
/* do some post-processing */
CheckKeyCase(theEvent);
/* return to original caller */
return (returnValue);
}
/**********************************************************
* MySystemEvent:
* Tail patch on SystemEvent.
*
* The reason this returns a short instead of a Boolean is
* because we need to make sure the low byte of the top word
* on the stack is zero because some programs do a Tst.W
* (SP)+ when this returns instead of Tst.B (SP)+ like they
* should (which is technically their bug but, we might as
* well work around it since it's not hard).
*
* We need this patch as well as the one on GetNextEvent
* because of desk accessories. If you don't patch
* SystemEvent then the patch will not apply to events that
* are sent to DAs.
*
* If you want to eat the event and not pass it on to the
* caller then set returnValue to zero.
*********************************************************/
pascal short MySystemEvent(EventRecord *theEvent)
{
PatchGlobalsPtr pgPtr;
short returnValue;
/* find our globals */
pgPtr = (PatchGlobalsPtr) ((long) StartPatchCode -
sizeof(PatchGlobals));
/* call original GNE first */
returnValue = (*pgPtr->pgOldSE)(theEvent);
/* do some post-processing */
CheckKeyCase(theEvent);
/* return to original caller */
return (returnValue);
}
/*********************************************************
* CheckKeyCase:
* If theEvent was a keyDown or autoKey event, this checks
* if both the shiftKey and the capsLock key were down. If
* so, it changes theEvent to be a lowercase letter. If not,
* nothing is changed. Also, if either the optionKey or
* cmdKey is down then nothing is changed.
********************************************************/
void CheckKeyCase(EventRecord *theEvent)
{
register long theMods, theMessage;
register char theChar;
if (theEvent->what == keyDown ||
theEvent->what == autoKey) {
theMods = theEvent->modifiers;
theMods &= shiftKey | alphaLock |
optionKey | cmdKey;
theMods ^= shiftKey | alphaLock;
if (!theMods) {
theMessage = theEvent->message;
theChar = theMessage & charCodeMask;
if (theChar >= 'A' && theChar <= 'Z') {
theMessage &= ~charCodeMask;
theMessage |= theChar + 'a' - 'A';
theEvent->message = theMessage;
}
}
}
}
/*********************************************************
* EndPatchCode:
* Dummy proc to mark the end of the code for the patches.
* Make sure all of your patch code is between here and
* StartPatchCode.
*********************************************************/
void EndPatchCode()
{
}