TidBITS: ADB Restart Devices

Article 1 of 2 in series

20 Jul 1998 | Print

The Battle of the Bouncers, Part 1

I run a number of Macintosh-based Internet servers, and for the most part, these servers are stable. Crashes aren't frequent, but they do happen often enough to be a nuisance, particularly on Web servers that need to run all the time. Worse, several of my servers are a 45-minute drive away, in an office building I can't access after business hours. The laws of the universe state that crashes tend to happen on Friday evenings so that the server can be unavailable for as long as possible.

In part, I work around the laws of the universe by using crash-detection devices that can watch for crashes and restart a hung Mac without human interference. These devices connect to the ADB port (and sometimes the power outlet) and communicate with special software to determine if the Mac is operational. When they determine a crash has occurred, they restart the Mac, either by turning it off and on again, or by sending a keyboard restart over ADB - just as though you had pressed Command-Control-Power.

The main device I've used over the years to watch for crashes and restart Macs automatically is the PowerKey Pro from Sophisticated Circuits. A pair of PowerKey Pros watch our main servers in Seattle, and a third kept an eye on our internal file server until recently, when the time came to test several newcomers. On our internal file server, I installed the new crash-detection device from Sophisticated Circuits called Rebound. On our SE/30-based mailing list host I installed a crash-detection peripheral from Kernel Productions, called Lazarus. I never received another competitor, MacCoach from the Belgian company Neuron Data Systems, but I'll try to include information about it when possible.

<http://www.sophisticated.com/products/ powerkey.html>
<http://www.sophisticated.com/products/ rebound.html>
<http://www.kernel.com/kernel/lazarus.html>
<http://www.neuronsys.com/Products/>

Although I've found ways all that these devices can be fooled into restarting a Mac that hasn't crashed, the most important fact is that none have so far failed to restart a crashed Mac. That said, how do these products compare?

Hardware -- The PowerKey Pro comes in two different models, the 200 and the 600. The two units are physically similar, each offering six power outlets, a telephone jack, and an ADB plug. The 200 lacks the phone tone control and manual switches of the 600, but the 200 has 160 Joules of surge protection, whereas the 600 can take an add-on surge protector. (Omitting surge protection from the PowerKey Pro 600 is intentional, since people who buy the PowerKey Pro 600 also often have a UPS - uninterruptible power supply - and many UPS manufacturers don't recommend plugging surge protectors into a UPS.) The important feature for monitoring servers comes with the software, an optional $39.95 component called the Server Restart Option (SRO) that's currently bundled with the 600. Because the PowerKey Pro offers six power outlets and has space for the telephone jack, the ADB port, and its own power cord, it's quite a bit larger than the other devices I evaluated.

The PowerKey Pro's telephone jack bears mention. Before the SRO became available, we plugged the PowerKey Pro into a telephone line and created events that would restart the server if we called and let it ring 12 times. The PowerKey Pro can also detect distinctive ringing, so you could set up an alternate number for your main phone line and let the PowerKey Pro only detect calls specifically to the alternate number.

The minuscule Rebound is a stark contrast to the beefy PowerKey Pro. It's a sleek three-inch long black ADB cable tipped with a bright yellow ball. It fits anywhere in your ADB chain, so you can move it around. TidBITS Technical Editor Geoff Duncan ran into some "problems" with the Rebound's physical design - you'll enjoy reading his report and Sophisticated Circuits' response.

<http://www.quibble.com/geoff/rebound/>

Lazarus falls in between the previous two in size, since it's a black plastic box with a pair of power plugs (one for the cord to the wall, the other for a cord to the Mac), and a pair of ADB ports so it can sit in your ADB chain. The Lazarus device I received was an early unit and had a distinct hand-made feel.

MacCoach closely resembles Rebound, with a boxy connector replacing the yellow ball on one end of the ADB cable.

Restart Method -- Function follows form with these devices. The PowerKey Pro, with its six power outlets, restarts a Mac by cutting power to the Mac's power outlet. If the Mac isn't plugged into one of the switchable outlets in the PowerKey Pro 200, then the PowerKey software instead tries a keyboard restart, which is the same as pressing Command-Control-Power on most (but not all) Macs. Ideally, the PowerKey Pro should first try Command-Control-Power and if that fails, toggle power to the Mac, since toggling power can be stressful for the Mac's power supply and other components. This is especially true if an application crashes on startup, causing the Mac to restart constantly until someone intervenes.

The tiny Rebound can perform only keyboard restarts, limiting its range and effectiveness slightly. Some older Macintosh models don't support keyboard restarts, so the Rebound doesn't work for them. In addition, a crash could totally wipe out ADB, at which point a keyboard restart would fail. Luckily, such severe crashes are extremely unusual.

Like the PowerKey Pro, Lazarus toggles power to restart the Mac, so it works on all desktop Macs. One small advantage Lazarus has over the PowerKey Pro is that international users can use Lazarus with no trouble (as I understand it, the female end of power plugs is standardized, whereas the male end varies by country). The PowerKey Pro's power cable is permanently attached, so it would need an adapter in other countries. Neither Rebound nor MacCoach worry about power plugs, being ADB devices.

None of these devices are likely to work well on PowerBooks. PowerBook servers probably still have batteries installed, so they can withstand short power outages. I think of an internal PowerBook battery as a sort of built-in UPS. Thus, neither the PowerKey Pro nor Lazarus could restart a PowerBook by toggling power. In addition, PowerBooks reportedly have different ADB controller chips than desktop Macs, which eliminates MacCoach and Rebound (Sophisticated Circuits thought Rebound might work on the PowerBook 520/540 series and older, but not on newer PowerBooks). The MacCoach information explicitly says it won't work on PowerBooks or some 68K Macs.

Crash Detection -- All of these devices use essentially the same method of detecting crashes. They rely on software that regularly "pings" the hardware to check that the Mac is still alive, in essence continually resetting a timer. If that timer runs out because the hardware fails to receive pings for a user-specified amount of time, the device restarts the Mac.

The PowerKey Pro and Rebound use a faceless background application loaded from the Extensions folder to perform the pinging, whereas Lazarus relies on a normal application launched from the Startup Items folder. It's possible to quit the Lazarus application and lose protection, but it's also easier to turn off Lazarus's protection for maintenance or troubleshooting. MacCoach reportedly relies on a driver that loads very early, but may not be as sensitive as an application.

The PowerKey Pro, Rebound, and Lazarus can make sure specific applications remain active. Applications must explicitly support the PowerKey Pro and Rebound, whereas Lazarus can watch any application to make sure it's still active (as can the 2.0 version of the MacCoach software, reportedly). The list of server applications supporting the PowerKey Pro and Rebound currently includes WebSTAR, LetterRip Pro, Newsstand, Quid Pro Quo, WebServer 4D, WebSiphon, CommuniGate, TeleFinder, and PageSentry. Applications support the PowerKey Pro and Rebound by continually resetting a hardware timer. An advantage of the PowerKey Pro/Rebound method is that applications can restart the Mac if they detect low memory situations or other potential problems before they cause a crash.

For multiple server setups, the PowerKey Pro can work with Maxum's PageSentry Pro monitoring software to restart a number of servers. You plug the PowerKey Pro's ADB cable into the Mac running PageSentry, then plug the servers into the PowerKey Pro's switchable power outlets. When PageSentry detects that a server isn't responding, it toggles power to that server. My servers aren't even in the same area code, so this feature doesn't help me, but it's useful for many people running multiple servers.

<http://www.maxum.com/pagesentry/>

Be careful when deciding which applications Lazarus should watch. There's a button to add all the active applications to Lazarus's list, but when I accidentally added Email Admin (a LetterRip Pro processor), I ran into troubl, since Email Admin launches when LetterRip Pro receives its first message, not at startup. After startup, when all the other applications launched, Lazarus noticed that Email Admin wasn't running, and it restarted the Mac repeatedly until I noticed.

I prefer the method used by Karl Pottie's Keep It Up, a $22 shareware application that watches applications and when they quit or crash, tries to relaunch them, restarting the Mac only if relaunching fails.

<http://www.vl-brabant.be/mac/kiu.html>

Restart Next Week -- The second part of this article will discuss the documentation, logging features, interface, and pricing of these restart devices.

Article 2 of 2 in series

27 Jul 1998 | Print

The Battle of the Bouncers, Part 2

by Adam C. Engst

In the first part of this article in TidBITS-439, I looked at how three crash detection devices - the PowerKey Pro, Rebound, and Lazarus - compare in terms of hardware, restart method, and crash detection capabilitiesShow full article

Hide full article

<http://db.tidbits.com/article/05002>
<http://www.sophisticated.com/products/ powerkey.html>
<http://www.sophisticated.com/products/ rebound.html>
<http://www.kernel.com/kernel/lazarus.html>

Since last week's article appeared, I've received a MacCoach unit from Neuron Data Systems (who are located in the Netherlands, not Belgium). I'll include more about MacCoach in this part, but since I haven't had time to test the device, I'll leave full evaluation for the future.

<http://www.neuronsys.com/Products/>

Interface -- The PowerKey Pro's interface, available via the PowerKey Editor application, is by far the most convoluted of the devices, mostly because restarting crashed servers is only one of its many capabilities. Other actions include toggling different power outlets; starting up, shutting down, and restarting Macs; executing AppleScripts; typing keystrokes; opening files; quitting applications; mounting SCSI devices; and logging actions. Each event can have multiple actions, plus a trigger. Triggers can activate once, on a repeating basis, on days of week or month, when the Power key is pressed, when a hot key is pressed, when the phone rings, when the system is idle, when power returns, and at shut down. The Server Restart Option adds triggers that kick in when the system crashes and when a timer expires. Triggers have qualifiers, so you can limit them to particular times and dates, or have them activate based on how the system was started, whether the system was idle, and so on. The possibilities are almost endless. To restart a server, you just need an event using the When System Crashes trigger and a Restart action (I always throw in an Add to Log action as well). It's also a good idea to create an event that restarts the computer when power returns after a power failure.

I don't want to imply that the PowerKey Editor application is hard to use. It's not, but you must think about how you want to use it. Ideally, Sophisticated Circuits would add a menu listing common events like restarting after a crash or restarting after a power failure, so you could start with those options and customize them later.

Rebound's simple control panel enables you to toggle system and application crash detection, and set three variables: length of time before restarting, time allowed for the system to restart, and number of restart attempts. Keep the restart events set to one in most cases, but resist the temptation to set the times too low. I was once running Norton Disk Doctor, and the file check took more than five minutes, causing Rebound to restart the system during the check. Worse, I pressed Shift to boot without extensions, and that combined with fact that the Mac had restarted ungracefully meant Rebound's timer hadn't been reset. Five minutes later, it restarted the Mac again. I've eliminated this problem by increasing time before restarting to ten minutes and setting the number of restart attempts to one, so Rebound won't repeatedly restart the Mac again in this situation.

Lazarus provides a small status window that tells you it's enabled and lists the time and date of the next scheduled restart, if any. I like the status window, but it should also provide the date, time, and details of the last crash.

Lazarus also offers several settings. You can set it to restart the Mac automatically every few days at a specific time (but there's no way of isolating particular days). You can have Lazarus monitor applications, and a Choose button brings up a dialog where you can add all open applications, add a specific application, or delete an application from the list. Overall, Lazarus's software is functional and easy to use, but lacks the polish and professional feel of the Sophisticated Circuits interfaces.

Logging -- It's important for these devices to log their actions because logs are often the only indication of trouble. Otherwise, you may never realize the server is being restarted every few hours.

The PowerKey Pro offers basic logging capabilities via an Add to Log action that writes a user-specified message to a text file (called PowerKey Log in the PowerKey Folder in the Preferences folder), along with the date and time. You must view the log in another program, such as SimpleText (the PowerKey Pro keeps the log under 32K), which requires setting up aliases for quick access. You can create an action that automatically opens the log file after a crash, but that's best done with a low-volume local server, not a high-volume remote server.

Rebound provides minimal logging. In its control panel's About box, it counts the number of system and application crashes, and gives you the time and date of the last restart. I'd like to see significantly more logging information in Rebound, since I'd want to know if the Mac has crashed ten times in the last three hours.

Lazarus's logging is the best of the lot, but it's still mediocre. You can display Lazarus's log from within the application, but it's a small window that can't be resized and lacks a scroll bar. You can also save its log as an HTML file. Within the log, Lazarus lists the date and time of each event (a crash or scheduled restart), and identifies crashes either as global or as caused by a specific application.

According to the Neuron Data Systems Web site, MacCoach 2.0 now includes comprehensive logging, including a report of when the Mac starts up and shuts down, when applications launch and quit, and when the system or applications crash. The MacCoach control panel displays the log, plus it can save as an HTML file automatically.

Documentation -- The PowerKey Pro comes with an extensive manual explaining its triggers, qualifiers, and actions, and provides examples of how one might use these features. If anything, I'd like even more examples, since the hardest part about using the PowerKey Pro is figuring out what you can do with it.

For example, one useful setup would be to plug the server's monitor into a controlled outlet. When the system is idle, power down the monitor if it's not capable of entering a sleep mode (my favorite 12-inch monochrome monitors can't do this). However, if you restart a server with the monitor off, Timbuktu Pro can refuse to connect, since it doesn't know how large a screen to simulate. The PowerKey Pro could turn the monitor on after a crash so the Mac sees a monitor at startup, then turn the monitor off again at idle time to save electricity.

Rebound provides an 11-page manual in PDF format that covers the bases well, providing installation and setup help, along with details of Rebound's AppleScript support. Rebound also includes Apple Guide-based help that answers "how to" questions but ignores most "why" questions.

Lazarus comes with only a sheet of paper that details installation (which is easy) and briefly explains basic operation. At the very least, I'd like to see Kernel Productions create detailed online documents that explain how Lazarus works, plus provide real-world troubleshooting.

MacCoach's paper documentation is a sheet of paper containing essential information, though with little expansion of complex topics. More information is on Neuron Data System's Web site.

Price -- The PowerKey Pro 200 costs $140 with the necessary Server Restart Option; the PowerKey 600, which currently includes the SRO, is $200. The PowerKey Pro is available directly from Sophisticated Circuits and a variety of vendors, including TidBITS sponsor Cyberian Outpost.

Rebound costs $99 directly from Sophisticated Circuits and should be widely available soon. It debuted for $50 for those who purchased WebSTAR 3.0 directly from StarNine Technologies, and that deal remains in place as long as supplies last.

<http://store2.starnine.com/store/starnine.tmpl>

Lazarus costs $100 directly from Kernel Productions.

MacCoach costs $100 and is available from TidBITS sponsor Cyberian Outpost. For a limited time, it's also available for $55 from Tenon Intersystems for those who own or buy Tenon's WebTen Web server.

<http://www.tenon.com/products/webten/ Funnel.MacCoach.shtml#MacCoach>

Picking & Choosing -- All the devices I tested were similar in terms of efficacy, restarting Macs when necessary (which wasn't all that often). The two PowerKey Pro models cost the most, but are more flexible and support all desktop Macs. If you started to cogitate about possible things to automate as I reeled off the PowerKey Pro's list of features, you definitely want one. Rebound and Lazarus have similar price points, but differ in technique. Lazarus works with all desktop Macs, has a display and better logging, can watch all applications, and can restart the Mac on a schedule. However, Rebound's software is more polished and less prone to user error, and Rebound is physically better assembled and will be easier on hardware thanks to its keyboard restart method. MacCoach is probably comparable to Rebound, but not having tested it yet, I can't recommend it either way.

It's hard to make product recommendations in this category. My feeling is that Rebound is the best option if you want something simple to install and forget. Sophisticated Circuits is very experienced in this field, so I trust their hardware and software. As a newcomer, Lazarus is less tested and less polished, but it offers a different approach plus a few more features than Rebound. And finally, the PowerKey Pro offers an unparalleled level of control and flexibility, especially if you need to automate physical devices beyond restarting crashed servers.

TidBITS Watchlist

Set Time Zone Automatically in Snow Leopard

Recent TidBITS Talk Discussions

Series: ADB Restart Devices

The Battle of the Bouncers, Part 1

The Battle of the Bouncers, Part 2