Hide full article
The TidBITS staff has been spending a vast amount of time and energy thinking about how we want to recast our collaborative editing system, but we haven't been able to come up with a solution that meets all of our needs and wants. In the past, we've gotten great advice from readers that, for example, helped us set up a search tool (see "TidBITS Macintosh Search Tool Shootout" in TidBITS-368) and choose a content management system (see "Help Us Choose Among Content Management Systems" in TidBITS-675). Let me explain what we're contemplating, and then I'd like to see if you can suggest anything.
<http://db.tidbits.com/article/00721>
<http://db.tidbits.com/article/07143>
Our eventual goal is to create a system that enables us to create new TidBITS articles, have multiple people be able to edit them, and post the final result to a content management system. It sounds simple, and in fact, conceptually, none of the pieces is all that difficult. The three parts are:
An easily accessible Internet file storage site for sharing articles
Change-tracking across multiple iterations of an article
A combined format/technology solution for uploading finished articles to a content management system
A company called Near-Time created a program called Flow that addressed all of these problems. It provided an interface to "spaces" that could be shared with other Flow users over the Internet. Each space held documents that could be created with standard Cocoa text editing tools, bolstered by extra-textual comments. Flow handled version tracking by default, updating itself every few minutes and optionally showing the changes from one shared version to another. Once an article was finished, we could post it to a weblog on our Web Crossing server via the MetaWeblog API, and Flow would also pull down articles that others had posted.
Flow was by no means perfect, though. It had an interface that required significant training, many additional (and unnecessary, in our view) features that cluttered the interface, and some key bugs. More worrisome for those of us who like to control our own services, Flow relied upon a Near-Time server and used SMTP and POP to synchronize versions. I refer to the program in the past tense because although Near-Time hasn't officially killed it, you can't find any mention of it on their site and they're focusing all their efforts on a Web-based service that provides weblog, wiki, and shared calendar capabilities.
<http://www.near-time.com/>
Anyway, here's what we do now and possible alternatives we've come up with.
Internet File Storage Site -- Any small group that's attempting to share files needs some sort of centralized file store. Our current solution relies on AppleShare over IP running on the one Mac I have with a static IP address. On that file server, which all of our editors can mount in the Finder, there are two folders: IN and OUT. Files available for editing live in IN; those that someone is working on are moved to OUT. We also add metadata to article file names for more information. In other words, a version number in the file name is incremented by each editor, and each editor appends his or her initials to the file name after editing. So, this file starts out life as DocumentCollaboration-1.ace in IN. When Jeff wants to edit it, he moves it to OUT and renames it to DocumentCollaboration-2.jlc. That way we all know who has it checked out, and we know not to mess with it until he puts it back in IN.
This solution works fairly well, but has some problems. Most notable is that AppleShare over IP can be troublesome when used over the Internet. It's not uncommon to have troubles with the Finder, and having the volume mounted can slow down any activity that queries the available volumes. Plus, it's sufficiently slow that most people choose to copy a file locally before working on it; that's extra work and can engender mistakes.
We've hit upon two solutions to this problem. First is the Subversion version control system. Matt Neuburg, who's familiar with Subversion as a programmer, set up a shared repository for us and has helped train everyone in what's necessary to use the main Mac OS X Subversion clients, svnX and BBEdit. Overall, the Subversion setup works well technically, but despite loud claims in the Subversion documentation that it can be used for any sort of file, the entire system is designed by and for programmers and falls down hard in terms of usability. The free svnX offers a graphical interface, but it's obtuse, and BBEdit's interface is essentially a long list of similarly named menu commands. Most annoyingly, there's no built-in way in svnX or BBEdit of saying, "Just keep my local copies of the files up-to-date at all times," forcing us to update manually over and over again.
<http://subversion.tigris.org/>
<http://svnbook.red-bean.com/>
<http://www.lachoseinteractive.net/en/community/ subversion/svnx/features/>
<http://www.barebones.com/products/bbedit/>
I came upon another possibility that's a bit reminiscent of Flow, the weblog editor Ecto. Once properly configured to see our Web Crossing server via the MetaWeblog API, Ecto automatically showed me all the weblogs we host, and it's trivial to create a new article or edit an existing one, regardless of who first created it. Using Ecto, I could create a new article in a private weblog, Jeff could edit it, and I could add more later. When we were done, there's an easy way to transfer an article from one weblog to another. Although Ecto doesn't automatically refresh its listing of posts on a schedule, it's easy to do and could likely be automated by AppleScript, and overall it provides a nice interface. (Another program that provides roughly the same feature set as Ecto is MarsEdit, by Brent Simmons of NetNewsWire fame, but on the face of things, Ecto seems a bit more full-featured.)
<http://ecto.kung-foo.tv/>
<http://ranchero.com/marsedit/>
Change Tracking -- So what's wrong with Subversion and Ecto? In both cases, it comes down to tracking changes and commenting. Because we have a number of people editing articles multiple times, we need a simple way to view changes.
Right now, we use Nisus Writer Classic, and we all know to mark notable changes in green and to make comments (prefixed with three asterisks) in red. Obviously, this sort of change tracking is more work than the way Microsoft Word does it automatically, but paragraphs aren't cluttered with lots of very minor changes, and we don't have to think about who made any given change. A Word document that's been through a few edit passes is generally a cacophony of color, which makes it hard to read, and it's difficult to tell which changes were made when. If one of us edits an article multiple times, it's impossible for others to tell which changes were made in which edit pass. Also, we've noticed over the years that Word's change tracking often sets up an antagonistic situation ("How dare he change that word!") where none really exists, and its accept/reject approach doesn't match well with what we generally want to do, which is to accept all the good changes (so it would be nice not to have to do this manually) and tweak all the bad changes (seldom does anyone just reject a change).
Flow got this right by storing each version of the document that was shared and providing a comparison feature that could be applied to any two versions. It wasn't perfect; sometimes it declared an entire paragraph was changed when only a word or two were different, but it worked pretty well. Better yet, you could compare versions whenever you wanted, but the changes weren't in your face all the time as they tend to be with Word (and yes, I know that Word's change tracking display can be modified, but it's clumsy).
Subversion, because it's designed for programmers to share code, is great about maintaining multiple versions of documents. But even though the versions are present, there's no good way to see the differences between them. BBEdit can compare files, of course, but it can do so only at a line level and shows the comparisons in a separate window from the latest version of the document. With prose text, a line is a paragraph, and adding a single comma marks an entire paragraph as changed. There's also the FileMerge utility that comes with Apple's developer tools. It can do character-level comparison, but it's such a horrible text editing environment that it's essentially unusable. Plus, neither BBEdit nor svnX offer a halfway decent interface for seeing all the versions of a document and comparing them.
While bemoaning BBEdit's lack of any control over the look of individual characters, we realized that BBEdit can do some colorizing of text when the document in question is in a particular programming or tagging language like HTML. In BBEdit 8, this colorizing is configurable via "codeless language modules," simple XML files that define tags for BBEdit to recognize. I created a codeless language module that understood that .tb files used lines starting with three asterisks as comments and would colorize text between backslashes (a subtle character we essentially never use in TidBITS). In essence, this solution recreates what we had in Nisus Writer Classic, though we must add backslashes around notable changes and remember to remove them at the end. Again, it's a functional solution, but it's more work than would be ideal, and is generally inelegant.
Ecto has no versioning capabilities at all, which means that the only way to track changes in a document would be to use the colored text approach we used in Nisus Writer. That's not unthinkable, but colorizing text via the Colors palette isn't a fluid action. I don't know if there's any way to make a keyboard shortcut for applying a color, but none has jumped out at me. One of the planned aspects of our next-generation content management system is the capability to maintain multiple versions of articles (Web Crossing can already do this via its wiki plug-in); it's possible that this capability would help during the editing phase as well.
Final Format -- The third aspect of the problem is the format in which we write, and how that interacts with our content management system. Currently, we use a variety of styles in Nisus Writer Classic, and when Geoff Duncan has finished the final edit pass on the issue, he uses some age-old Nisus macros I wrote to turn the styled issue file into the setext (structure enhanced text) format we send out in email, and an HTML version that he then imports into his database (which subsequently generates the HTML edition that's also sent out via email, along with the text and HTML announcement editions). Although we have to be highly accurate about the styles so my macros can do the right thing, it's easy to apply them and edit the document with them applied. Unfortunately, this system is entirely dependent on Nisus Writer Classic, and we simply can't be relying on a Classic application any more now that Intel-based Macs don't support Classic.
In rethinking this part of the problem, it would seem to make sense to create articles in HTML, since it has become a common file format and would import easily into our next-generation content management system as well as our current one. But we don't want to have to write in full HTML code, since it makes editing difficult and is prone to error. Plus, HTML can be difficult to render down to something simple like setext for email.
Ecto addresses this issue by allowing styled text editing and sending HTML out when articles are posted, but any conversion to something like setext would have to be a server-side function. In the next-generation content management system, that's likely, but styled text-to-HTML doesn't help us much at the moment.
We've also considered swapping our Nisus Writer styles for writing in something very much like setext - John Gruber's setext-inspired Markdown language. John designed it to be easy to use while writing, and he provides a Perl script that can be installed into BBEdit to convert Markdown format into HTML. Plus, it appears that Ecto automatically understands Markdown as well when previewing articles, so if we can create scripts in Ecto to convert Markdown into setext and add Markdown support to our next-generation content management system, we could have a solution for both the present and the future. This may sound like a good approach, but changing all of our systems will be a fair amount of work, so we don't want to move further until we're certain it's the best approach.
<http://daringfireball.net/projects/markdown/>
Putting It All Together -- If I had to make a decision today, I'd pick a private weblog as the Internet file storage site, with articles edited in Ecto using the Markdown format and colors for change tracking. Our setext and HTML issues could be generated in BBEdit using Text Factories until such time as our content management system can kick those issues out semi-automatically.
But the more I poke at these different programs, the more picky little details I come across, and the more I wonder if there might not be other approaches that could work even better. For instance, although I found that Ecto can send an article to BBEdit for editing, that method of transferring the data always converts characters that are problematic in HTML to their corresponding entities, even though we want angle brackets, for instance, to remain angle brackets in the setext version of TidBITS.
A discussion I had with Chuck Joiner on a recent MacNotables podcast generated a few suggestions from listeners that I haven't yet had time to investigate fully, but which look promising. In particular, we're checking out Writeboard from 37signals and AdventNet's Zoho Writer, though the latter works only in Mozilla-based browsers like Firefox and Netscape. Both are essentially online word processors, leading me to wonder if writing in a Web browser instead of a real Macintosh text editor would be acceptable, if the services would be sufficiently reliable in both daily uptime and in the long run, and if we'd run into troubles with needing to work in too-many offline environments like airplanes.
<http://www.macnotables.com/archives/2006/ 631.html>
<http://www.writeboard.com/>
<http://www.zohowriter.com/>
So I welcome your thoughts and suggestions about alternative solutions that we might be missing. Feel free to send them to me directly, and if appropriate, I may forward some to TidBITS Talk for further discussion. Thanks!
