Hide full article
The press loves to quote numbers, especially when predicting the immediate demise of Apple Computer. However, the numbers the press uses are often less precise than they would have you believe. For instance, the Software Publishing Association (SPA) tracks software sales on a regular basis and reported last year that Macintosh software sales in 1995 were down roughly 14 percent. However, the SPA releases sales figures for a given year twice - a preliminary set about three months into the next year, and a final set a year later. That 14 percent drop came from preliminary figures for 1995; now that 1996 is over, the SPA has released the final numbers for 1995, which show that Macintosh software sales were in fact up 24 percent. Oops. Let's take a look at how the SPA gathers this information and what it all means.
What is the SPA? The recently released numbers come from the SPA's ongoing sales survey research effort. The SPA is a non-profit membership organization that works to advance recognition of key software industry issues with the government and business communities. Past SPA initiatives have included extensive anti-piracy efforts, lobbying for pro-software legislation in Washington D.C. (including efforts against tariffs and export controls and in favor of cryptography), and educational efforts like the "Cybersurfari" contests that help students learn about the Internet.
As a trade group, the SPA's members are naturally interested in using the organization's resources to advance their business goals. To that end, the software surveys of SPA members exist (more or less) to convince the world that the software business is booming and is therefore a fertile ground for investment and career choices. Even with such a purpose, though, the SPA is not afraid to publish results that indicate bad times - for example, the last release in December showed an overall decline in software sales for the third calendar quarter of 1996, but the organization did put the best possible spin on their results, pointing out how sales the previous year were artificially high due to the introduction of Windows 95 and associated software.
Companies participating in SPA surveys are asked to submit sales totals in seventeen categories, broken out by operating system. Submissions are due six weeks or so after the end of each quarter. The SPA then tabulates the results and distributes them to participating companies in large, detailed spreadsheets about three weeks later. Three to four weeks after that, the SPA announces the results to the press, although with far less detail than participating companies receive.
<http://www.spa.org/research/default.htm>
Flawed Research -- The system seems reasonable on the face of it, but it has flaws:
Unknown sample size: The SPA says that SPA members are responsible for 85 percent of all North American software sales (and the numbers in this particular survey and article, by the way, are only for U.S. and Canadian sales), but they refuse to disclose how many of those members participate in the voluntary sales surveys. If the responding companies represent significantly less than 85 percent of all software sales, the survey results are correspondingly less important.
Restricted responses: Only SPA members can participate, and since dues start at $750 per year and go into the hundreds of thousands of dollars for the largest companies (depending on software revenues), many smaller developers - the kinds that traditionally do innovative Macintosh work - choose not to join, and are therefore ineligible to have their sales tabulated.
- Few uniform guidelines: The SPA tabulates data exactly as companies report it - so if those companies have little information on where their software goes, the SPA has just as little. Take the well-publicized case of hybrid CD-ROMs that contain Macintosh and Windows software. The SPA makes no attempt to figure this out, and they shouldn't have to. Instead, they rely on data from reporting member companies. If a software company ships 10,000 hybrid titles in a year but has no idea how many went to Mac users and how many went to Windows users, they'll either not report the sales or they'll estimate percentages. The SPA doesn't provide guidelines for collecting accurate information, and Apple executives have publicly speculated that many more hybrid CD-ROM sales should be counted as Mac OS sales than are reported that way. With the SPA's voluntary survey, there's no way to tell.
Fast But Inaccurate -- However, the biggest problem is that the results have, for the past several years, been shown by the SPA itself to be highly inaccurate. Some companies don't have the data the SPA wants by the deadline, so they don't turn it in. If they come up with it a month or two later, they'll submit it then, and the SPA will assimilate it into the results. Then, when the SPA needs numbers for the same period, the revised and ostensibly more accurate figures are used.
It makes sense to use the most accurate information available, says the SPA. However, over the past few years, some figures available a year after initial reporting have been vastly different from the originally announced figures. The SPA then commits a sin of synchronization - they compare the fast-but-incomplete results (which I call "preliminary" numbers) to the year-old, far-more-complete figures I call "final" results.
With few exceptions, the final results for Macintosh software sales have been higher - often significantly higher - than preliminary numbers. By comparing the preliminary numbers to the final numbers, the SPA is comparing apples to oranges, and they get percentage changes from one year to the next that can be invalid and highly misleading.
1995: A Good Year -- That's what we see with the results for software sales in 1995. Remember that the SPA releases numbers twice - preliminary figures about three months after the end of the year being discussed, and final numbers one year later as the basis for comparisons against the current year. We now have final 1995 numbers because the SPA is using them as a base to see the differences in the preliminary 1996 numbers.
Several of the SPA's seventeen categories showed little or no difference between the preliminary results and the final results, meaning the original reports were relatively representative - word processors, spreadsheets, databases, integrated software and presentation graphics were all revised by 4 percent or less. However, other categories saw major corrections. Sales of Macintosh utilities are now believed to be twice what the SPA reported a year ago, with similar gains in the Drawing & Painting, personal information management and home education categories. Overall, the SPA now believes that Macintosh software sales were 44.2 percent higher for 1995 than originally reported.
A year ago, when the SPA released preliminary 1995 results, they said Macintosh software sales were down 13.9 percent from 1994's final figures, meaning Macintosh software market share fell to 14 percent from 18 percent in 1994. Now that final 1995 numbers are available, it turns out that Macintosh software sales for the year actually grew 24 percent in 1995.
Unfortunately, Mac OS software still slipped to 15.5 percent market share. That's not because sales fell, but because Windows software sales grew faster - easy to understand during the year Windows 95 was released.
1996: An Iffy Year -- The recent announcement from the SPA compares preliminary 1996 numbers to the new, final 1995 numbers. The comparison shows what you'll read in the major press coverage of these results - a drop of 24 percent in sales for 1996. However, consider the following. What if we assume the under-reporting for 1996 will be similar to the under-reporting that took place in 1995? Perhaps, by comparing the two sets of preliminary numbers (an "apples to apples" comparison), we might find a better estimation of how things are really changing.
Such a comparison shows yearly growth of about 11 percent instead of a decline of 24 percent. This is a lot easier to swallow. Other research I've done showed that Mac OS market share actually rose during the last part of 1996 (see TidBITS-369), and it's hard to believe that computer sales were up while software sales for them were down. Historically, new computer sales have been a large driving force behind new software sales; new software, especially major software, is typically purchased only with new computers or when new versions are released.
Interestingly enough, the SPA's preliminary and final numbers for 1994, a year earlier, were close to each other - within about 3 percent. If we try to average 1994 and 1995 differences to get a sense of how far the SPA's preliminary numbers might be off, we come up with the idea that preliminary numbers are about 21 percent below final numbers. If we go with that idea, we could expect to see 1996 Macintosh software sales drop a bit as a 21 percent margin of error doesn't quite match the 24 percent drop the SPA reports in its preliminary 1996 numbers. However, don't put too much faith in this "average error" approach - the two margins of error for 1994 and 1995 are very different, a good indication that any speculations based on their average may not mean (no pun intended) much.
Past experience with the SPA's numbers and an understanding of the Mac OS market leads me to believe that the final numbers for 1996 will eventually reveal a small increase in sales, but nothing like the large increases for Windows-based software. 1996 was a difficult year for Macintosh software and hardware developers alike, and though customers did eventually return to the platform in small but significant numbers, I think that 20 percent or 30 percent gains would be unrealistic. It's more likely that the platform saw about 8 percent growth or less during the year. The hardware market showed similar moderate growth during 1996, and the "apples to apples" comparison of preliminary numbers is also in that ballpark. We won't know for sure until a year from now, when the SPA releases final numbers for 1996.
Why Does the SPA Do This? I spoke extensively on this matter last fall with SPA public relations director David Phelps, who confirmed my description of the process with research director Jim Sanders. Phelps vehemently insisted that the SPA was not anti-Macintosh and was, in fact, almost an entirely Mac OS shop - it's just that the numbers show declines, and they report them at the time with the best information they have available.
The SPA certainly isn't trying to deceive people about the Macintosh specifically, but the organization does show a marked and lamentable tendency to put the value of its own numbers above the good of the developers they're supposed to serve. The discrepancies mentioned above also affect other SPA categories - Windows (both 16-bit and 32-bit) and DOS - but in those cases, the differences are of magnitude, not of sign. DOS sales in the final figures didn't fall as much, and Windows sales in the preliminary figures didn't rise as much as later information showed. Only the Macintosh sales actually change direction, going from an announced loss to a quiet gain. When the SPA's final figures are 150 percent to 200 percent of the originally reported figures, as is often the case here, the preliminary announcements become significantly misleading.
This then is the SPA's problem. The group is determined to release the numbers on schedule, which invites these inaccurate and misleading reports. Requests that the SPA disclose how many member companies report, disclose the full results that reporting companies receive, and point out when revised figures make the original announcements invalid have all been rejected. There is only one crack in the armor - in this latest press release, touting how all application software sales topped $10 billion for the first time (although a year from now it will probably be significantly higher), the SPA said that the 1996 total was "an 8.3 percent increase from a revised $9.8 billion in 1995" (let me emphasize "revised"). Last year at this time, the SPA said 1995's total was $7.53 billion - an error, again, of nearly 25 percent.
The group appears afraid that if they point out how inaccurate their results are shown to be a year later, the press and public will stop paying attention to their cries that the software industry is healthy. But ironically, these same reports are making the Macintosh software market less healthy. Venture capitalists and other investors who read these results in the popular press will never hear about the corrections, or about the difference between "preliminary" and "final" numbers. The numbers keep going up and the SPA keeps announcing that sales are falling - if they were to point out the paradox in this proposition, they'd have to admit that they don't have accurate data three months after a year ends. They seem completely unwilling make that admission.
There's always hope that this trend will change. After MDJ reported in January (reprinted in TidBITS-363) that Mac OS hardware market share was flat to improving and noting how major research firms didn't bother to track Mac OS market share differently from Apple Computer's market share, some research firms started tracking Mac OS market share and the press was surprised to discover that it was growing. I'm not claiming cause and effect here, but the truth has a way of catching on once it escapes. The truth is that the Mac software market, while suffering through a bad year, has been significantly healthier than the SPA's preliminary numbers have indicated, and there is no reason to believe that situation has changed. Perhaps a different group will come along with scientifically sampled surveys that more accurately represent software sales, so the business community can accurately judge the validity of an investment in Mac OS technology.
[This article is reprinted and updated with permission from MDJ, a daily Macintosh publication covering news, products, and events in the Macintosh world. If you can't get enough insightful Mac news, sign up for a trial subscription to MDJ. TidBITS readers who had a free trial before 01-Apr-97 are invited to accept another one and see how this publication has changed since its beta-test period last year. For more information, visit the MDJ Web site.]
<http://www.gcsf.com/>
