Hide full article
This is the tale of a U.S. government statute, a certain Cupertino-based computer company, and a small revolution taking place quietly on your computer, if you use Mac OS X 10.2 Jaguar.
To understand this revolution, you need to know what a macro program is. I've been writing in TidBITS about such programs for years. Simply put, there are certain frequent or repetitive computer tasks that one would like to be able to automate. This is often straightforward if the program that performs these tasks is scriptable, meaning that it responds to a repertory of AppleScript commands: you just write a script expressing what you want done. But what if the program is not scriptable?
Of Macs and Macros, Take Two -- One solution that worked pretty well for me in previous incarnations of the Mac OS was to use a macro program. Such a program must hack into the Mac OS itself so as to be able to simulate the presence of a "ghost" user who can press keys and move and click the mouse just as a real user would. By stringing together such actions, the computer could sometimes be forced to make up for the lack of scriptability in certain applications. Macro programs that I formerly used include PreFab Player, QuicKeys, OneClick, and KeyQuencer.
<http://db.tidbits.com/article/00801>
<http://db.tidbits.com/series/1044>
Now wind the clock forward to Mac OS X. The trouble here is the word "hack" in the previous paragraph. One of the main points of Mac OS X is that you can't hack the operating system; to preserve stability, Apple has abolished the kind of extensions that used to make such hacks possible. Unfortunately, this means that macro programs don't work all that well on Mac OS X. When QuicKeys was migrated to Mac OS X, I was appalled how poorly it worked. It had trouble performing certain kinds of actions, such as choosing from menus and pushing buttons; and, more important, it couldn't "see" what menu items and buttons were present, so it couldn't make intelligent decisions or wait for the right window to appear.
<http://db.tidbits.com/article/06603>
These limitations, of course, were not the fault of QuicKeys. Since QuicKeys couldn't hack into Mac OS X, it could see and act only in ways permitted to it by the various legitimate application programming interfaces (APIs) provided by Apple. And Apple, concerned with stability, wasn't about to let just any old program reach in and start meddling with some other program's windows and menus. Knowing this, however, did not solve the problem; and users were left staring at QuicKeys, waiting for it to improve. It did; but not much.
<http://db.tidbits.com/article/07022>
<http://db.tidbits.com/article/06786>
Now for the revolution. While we were all staring the wrong way - at QuicKeys - the problem was being solved by Apple itself. And the solution (in general, and for future versions of QuicKeys as well) is in place, right now. You can script unscriptable applications, manipulating their menus and windows and buttons, and typing text, right now. It's just that Apple hasn't bothered to say much about it, so few people know.
Enter Uncle Sam -- How could this happen? And why did it happen? It all goes back to a U.S. statute called the Workforce Investment Act of 1998, commonly known as Section 508. This statute says that when federal agencies procure electronic information technology, that technology must give disabled people access to information that's comparable to the access it gives non-disabled people. In plain English, if you manufacture computers, and you want any government agency to be able to buy any of them, they must be fully operable by people with disabilities.
<http://www.usdoj.gov/crt/508/508law.html>
To see what this statute might mean for Apple Computer, Inc., let's put ourselves inside the mind of someone at Apple Computer, Inc. This person is imagining a Mac being operated by someone with a disability. And he or she is thinking: "Suppose this disability is such that the user can't work with the keyboard or the mouse. Holy cow! Unless we want our government sales to be zero, we'd better provide a way, in Mac OS X, for some other device, such as one of those sip-and-puff joysticks, to be able to operate all parts of the user interface." Clearly such operation implies something far beyond merely a few keyboard shortcuts that substitute for the mouse some of the time, as in the Universal Access and Keyboard preferences panes. It means that the system itself must be able to "see" and access all the individual interface elements of any application, so as to provide a pathway whereby a joystick, or any other assistive device, can press every button and click in every text box and somehow reach every interface element that appears on the screen.
<http://www.apple.com/macosx/jaguar/ universalaccess.html>
<http://www.abilityhub.com/mouse/joystick.htm>
The idea, then, is this. Given any application on Mac OS X, it needs to be possible for some other application to discover what interface elements it is displaying. The other application needs to be able to "read" these elements ("There's a button that says OK") and it needs to be able to access them (click the OK button). Apple went to work on the problem, and the result, which first emerged to public view in Jaguar, is the Accessibility API. Actually, there are two things going on. Under the surface, Apple has provided some deep magic, in the operating system itself, that does the reading and the accessing of all the interface elements present on the screen. Closer to the surface, Apple has provided the API, which is a set of commands a programmer can use to take advantage of this magic.
<http://developer.apple.com/techpubs/macosx/ Cocoa/TasksAndConcepts/ProgrammingTopics /Accessibility/>
<http://developer.apple.com/techpubs/macosx/ Carbon/HumanInterfaceToolbox/ Accessibility/MakingAppsAccessible/>
Now stop and think about what I just said. This is all present in Jaguar, right now. So anyone can write a program that can see and access the interface elements of any other program, right now. But such a program would, in effect, be a macro program! In short, the various things QuicKeys couldn't do when it first appeared on Mac OS X, because it couldn't hack into the Mac OS, any program can now do, thanks to the Accessibility API.
But it gets better. Read on.
Blessed (System) Events -- At some point, someone at Apple put two and two together, as follows. "On the one hand, we've got the Accessibility API, which lets any program access the interface elements of any other program. On the other hand, we've got AppleScript, which lets any user write a script to give commands to any scriptable program. So what would happen if we put them together? Any user would be able to write a script to give commands to a scriptable program which would use the Accessibility API to access the interface elements of any program." To see the significance of this, just take out the middle terms, and what have you got? Any user can write a script that can access the interface elements of any program. Any user can script an unscriptable program, using AppleScript! You wouldn't need any special macro program, because AppleScript itself, which is already on your computer, would become a macro language.
Of course, for this to work, there has to be that "middle term" - the go-between, the application that receives AppleScript commands and talks to the Accessibility API. That application is called System Events. It's on your computer right now, in /System/Library/Core Services. But the version of System Events that knows about the Accessibility API is probably not on your computer, because it's in beta and is not part of the standard Mac OS X release. You must download and install it separately, from Apple's GUI Scripting Web page.
<http://www.apple.com/applescript/GUI/>
Let's try it out! There are two preliminary steps.
Download and install the beta version of System Events.
In the Universal Access preferences pane, check the box at the bottom that says "Enable access for assistive devices." This is crucial because it throws the virtual switch that brings the Accessibility API into play.
We'll also need an unscriptable application to operate on. For this example, we'll use my own freeware MemoryStick, so download it (and shame on you) if you haven't got it.
<http://pages.sbcglobal.net/mattneub/downloads/ MemoryStick%20Installer.dmg.gz>
MemoryStick isn't scriptable, and was never intended to be. Nevertheless, as if by magic, we're going to open its Preferences window, find the Poll tab item, read how often MemoryStick is set to poll the system, and click the spinner up or down the right number of times so that the setting ends up at "5 seconds" - and then close the Preferences window. Ready?
Start up MemoryStick if it isn't running. Start up Script Editor (it's in /Applications/AppleScript). Paste in the following script:
tell application "MemoryStick" to activate
tell application "System Events"
tell application process "MemoryStick"
click menu item "Preferences..." of menu "MemoryStick" of menu bar 1
tell tab group 1 of window "MemoryStick Preferences"
click radio button "Poll"
get value of static text 3
copy the result as number to theVal
set theDec to theVal - 5
if theDec > 0 then
repeat theDec times
decrement incrementor 1
end repeat
else if theDec < 0 then
set theDec to 0 - theDec
repeat theDec times
increment incrementor 1
end repeat
end if
end tell
click button "Done" of window "MemoryStick Preferences"
end tell
end tell
Now run the script, and watch the fun. Don't blink, or you'll miss it!
Future Directions -- By now, you're probably saying, "Wow! That was great! How can I learn to give these sorts of commands to any application, so that I can script unscriptable applications and turn AppleScript into a macro programming language?"
Luckily, there's a splendid way to do this. It's called PreFab UI Browser, the work of Scott Lawton of PreFab Software, who brought us PreFab Player and TextMachine, and Bill Cheeseman, the well-known AppleScript guru and author of Cocoa Recipes for Mac OS X. This brilliant little program uses the Accessibility API to "look" at all the interface elements of any running application, and generates the AppleScript commands you'd use to click them, read them, type into them, or whatever. It's $25 (until mid-April, when the price goes up), and a 30-day demo is available.
<http://www.prefab.com/uibrowser/>
<http://db.tidbits.com/article/04198>
<http://www.applescriptsourcebook.com/>
<http://www.amazon.com/exec/obidos/ASIN/ 0201878011/tidbitselectro00>
It will be interesting to see what further applications will emerge that take advantage of the Accessibility API. For example, one of my favorite utilities in earlier days was an extension that listed all the windows in all running applications, so that you could switch directly to the right window, no matter what application you were in now. Before the Accessibility API, such a utility was impossible in Mac OS X, because no application could "see" another application's windows, let alone switch amongst them - the Dock could do it, but only because it belonged to Apple and was privy to System-level secrets no one else had. But now, writing such a utility should be easy.
The story is not over, not least because both the Accessibility API and the Scripting Events beta are still young, still in development, and still buggy. Nevertheless, they do work, as the example proves; and eventually, perhaps as part of the next major update to Mac OS X, I would expect the special version of System Events to come out of beta and be made part of the standard release. Meanwhile, now that you know the secret, you can start playing with AppleScript in its wonderful new role as a macro language on Mac OS X.
As you do, keep in mind that even the Accessibility API can't perform miracles; it can't detect interface items that are not constructed by way of Mac OS X's built-in toolbox. A program that uses non-standard interface items can be written deliberately to provide Accessibility API access to them; but if it isn't, the Accessibility API is blind. For example, the Accessibility API can't see any of Microsoft Word's menus, toolbars, or even the text of a document.
Not that this matters, of course, because Microsoft Word is completely scriptable already. Which raises another point: There's no substitute for true scriptability. To write an application of any serious power without giving it the ability to be driven with AppleScript is simply poor programming practice. So, if you're able to automate some previously unscriptable area of your workflow through Accessibility API scripting, that's splendid; but if the target program is still being actively developed, please do also write to the developers and ask that they make that same functionality genuinely scriptable.
PayBITS: If Matt's article makes a difference for what you
can script, consider sending him a few bucks via PayPal!
<https://www.paypal.com/xclick/ business=matt%40tidbits.com>
Read more about PayBITS: <http://www.tidbits.com/paybits/>
