Mirror 1.0
Written by Alastair Rankine
Copyright ©1997 Progmatics Pty Ltd.
The latest revision of this document and the Mirror software is available on the World Wide Web at the following URL: http://www.progmatics.com.au/mirror/
Written by Alastair Rankine
Copyright ©1997 Progmatics Pty Ltd.
The latest revision of this document and the Mirror software is available on the World Wide Web at the following URL: http://www.progmatics.com.au/mirror/
Mirror is an application that can keep files synchronised between computers on your local network or over the global Internet.
Computer networks are wonderful things, but sometimes there are occasions where information needs to be replicated from one part of the network to another, in order that it be accessible. Pages on the World Wide Web are often edited on one computer and then copied to another computer, which then makes them available to the Web. In this instance, there is a complete copy of the web site on both computers, which needs to be kept synchronised.
Mirror works by allowing the user to create a document which specifies where the original files are stored, and where to copy them. Using this information, Mirror can periodically scan the original files and copy the changed files to the destination. Mirror is designed to make this process as simple as possible, so that you can keep your files synchronised with a minimum of fuss.
There are four basic modes in which Mirror can operate. Mirror can copy files from your computer to a another computer over the Internet, or from another computer to yours. Mirror can also copy files between the disks accessible on your computer. Lastly, Mirror can copy files between two other computers over the Internet. Files on other computers are accessed with the Internet-standard FTP protocol.
In a nutshell: Mirror is free for non-commercial use and cheap for commercial use.
If you are using Mirror for non-profit personal use, or at a non-commercial educational institution, you are licenced to use it free of charge. Bear in mind that Mirror is not public-domain software, but we encourage you nonetheless to distribute Mirror and it's accompanying files as widely as possible.
If you are using Mirror for profit, or at a commercial educational institution, a licence fee is payable. To do so, use the Register application that you should have received with your copy of Mirror. Alternatively you can register on the World Wide Web at the following URL:
The price is as follows:
The complete terms of the licence should accompany your copy of Mirror. Alternatively they can be found on the World Wide Web at the following URL:
Mirror requires a Macintosh or clone running System 7.0 or later. In addition the following software must be installed:
The most obvious use for Mirror is to upload your home page HTML to your web server. It is assumed that you presently do this by using Anarchie, Fetch or something similar to upload your pages en masse using the FTP protocol. This tutorial will show you how to upload your home page with Mirror.
The first step is to organise the files that comprise your web site. You must ensure that the HTML files are grouped inside a single folder. The entire contents of this folder will be uploaded to your server, so you don't want to have any superfluous files in there. You should arrange your files with the same hierarchy as you desire on the server.
In the example above, we have organised our web site files into a folder called "Originals", which contains the files to be uploaded to the web server. Notice that the Mirror document, "Upload Example" is not inside this folder, so it will not be uploaded. This document is created in the next step...
If this is the first time you have launched Mirror, you will be presented with a window asking you to personalise your copy of the application. Please take a moment to select whether you will be using Mirror for commercial or non-commercial purposes. Click the appropriate button, fill in your name (or accept the defaults read from Internet Config), and click OK.
When you launch Mirror, a new untitled document will be opened for you. At the top of the window is a small prompt area and the start button. When you have entered all of the required information, the prompt area will read "Ready to Start" and the Start button will become enabled. Until then, the prompt area is used to indicate which information is missing.
Below the prompt area are four checkboxes. These are explained below, but for now leave them alone.
The two panels that occupy most of the document window are used to enter information about where files are to be copied from ("Original") and where they are to be copied to ("Mirror"). We are copying from a local folder to a remote server, so the popup menus in each panel should be adjusted to reflect this.
In the Original pane, we need to tell Mirror where our web site folder is. Use the "Set" button, or simply drag and drop your web site folder into the gray box next to it. Be sure to use the "Originals" that we set up in step 1. The gray box will change to reflect the path to your chosen folder. Mirror is smart enough to follow the location of this folder, should you move it (go ahead and try it while Mirror is running!). You probably won't have any aliases in your web site folder, but in case there are, click "Ignore Aliases" to do just that.
Next, enter your FTP server details in the "Mirror" panel. You should have been allocated this information from your Internet Service Provider. If in doubt, ask them. Typically, the hostname will be of the form ftp.xxx.com. You probably won't need to specify a directory, but you will need to enter your username and password (usually the same you use to log in with).
Mirror accepts URLs of the form "ftp://..." when dragged or pasted into either the hostname, directory or username fields. You may find this a more convenient method of setting up your Mirror document. Using Anarchie or Fetch, you can simply highlight the highlight the directory you wish to mirror, then copy and paste the URL into Mirror.
Your document should be ready to go, with the Start button enabled. If not, check the prompt area for a clue as to what is missing.
Click on the Start button, and your web pages should be uploaded. A status window will appear while the Mirror is in progress, giving information about transfer times and so forth. If an error occurs, check that your username and password is correct. You may also want to check the log file (located in the Mirror Logs folder) to determine what went wrong.
You can see Mirror in action by modifying one of the files on your web site and clicking the Start button again. Watch the status display (or check the log file), and you should see that only the file you modified has been uploaded.
Check that your web site is up-to-date by using a web browser. Also be sure to save your Mirror document.
This section describes features which will help you get the most out of Mirror.
Sometimes you don't want to copy the entire contents of a folder from the original site to the mirror site. The "Copy New Files" checkbox and "Copy Updated Files" checkbox can help you select which files to copy.
"New files" are defined as files which exist on the original site, but not on the mirror site. "Updated files" are files whose modification date on the original site is later than that of the corresponding file on the mirror site. Both types of files are copied by default, but turning one of these checkboxes off can help you select which files to copy.
For example, a site contains files called Alpha, Bravo and Charlie, and you are only interested in the Alpha and Charlie files. To do this, you could turn off "Copy New Files" and create empty files called Alpha and Charlie on your mirror site. In this case Mirror would ignore the Bravo file.
When you are copying files from a folder on your computer, you have another option for selecting files. By turning on the "Follow Aliases" option, you can select which files to copy, simply by creating aliases to the required files. See the reference section for further information about aliases.
By this stage you should have a Mirror document which you can open, click Start, and have the latest modifications on your Original site copied to the Mirror site. As you might imagine, this feature is most useful only if it is automated, and performed on a regular basis. In the options panel of the Mirror document window are two checkboxes which can help you set up Mirror to operate automatically.
The "Start when Opened" checkbox tells Mirror to start as soon as the document is opened. Using this option, you can set up your Mirror documents so that they will start mirroring simply by double-clicking them from the Finder. Better still, your Mirror documents can be opened using an AppleScript, or a utility such as Chris W. Johnson's cron. The following snippet illustrates how to open a Mirror document from an AppleScript. This might be inserted into, for example, an OT/PPP connection script.
tell application "Mirror" launch open alias "Pierre:Library:Web Sites:Example Web Site:Upload Example" end tell
"cron" is an extension which launches applications at scheduled times. Please see Chris' web site for more information: http://gargravarr.cc.utexas.edu/cron/cron.html
The "Quit when Finished" checkbox in the options panel of the Mirror document window can be used to have Mirror quit when it has finished copying files (or even if nothing was copied). This is also useful for automation purposes because the Mirror application does not have to occupy your computer's memory, instead it can be loaded and unloaded as required. Bear in mind that Mirror will not quit when finished if an error was encountered, or there is another mirror in progress.
The word mirror is computer industry jargon and is widely used as both a noun and a verb. We speak of "mirroring" which is the act of copying files from one place to another such that they form an identical hierarchy, implying also that they are to be kept in sync. For the purposes of this document, the two places are known generically as "sites".
Of the two sites, one regarded as the "original site", which is the place where changes to the files and/or hierarchy is made. By the act of mirroring, the changed files are copied from the original site through to the "mirror site".
A site bay be either "local" or "remote". A local site is an area on your local file system, such as a folder on your hard disk. A remote site is a directory on an FTP server.
When you specify an original and a mirror site and click the start button, Mirror synchronises the contents of the original site with that of the mirror site. It is worthwhile understanding the process that Mirror performs in order to do this.
The first step to be performed is called the "preflight". This is where Mirror walks through the contents of each directory on the original site. For each file found on the original site, Mirror attempts to locate a file with the same name and in the same relative location on the mirror site. If such a file cannot be found, Mirror adds the file to it's list of files to copy. Similarly, if a directory needs to be created on the mirror site, then Mirror stores a the name internally until later.
If a file exists on the mirror site with the same name and relative location as on the original, Mirror then checks the modification dates of both files. If the date of the file on the original site is more recent than that on the mirror site, the file is added to Mirror's internal list. Otherwise the file is ignored. See below for a more detailed discussion of modification dates.
The next step that Mirror takes is to examine it's list of files to copy and directories to create. Mirror ensures that no empty directories are to be created and that Mirror is not to re-visit directories from which there are no files to be copied.
The third step is the mirror itself. Mirror works through it's list of files to copy and directories to create. At the start of this phase, the status window changes from it's barberpole display to a progress bar, intended to give some indication of the amount of data remaining to copy.
Mirror needs to know the modification dates of each file on the original and copy site so that it can determine which files are to be sent. This is fine for local sites, but FTP sites pose something of a problem. When a file is sent using the FTP protocol, the modification date is not sent with it. So when a file is uploaded to an FTP server, the modification date on the file is usually the date the file was uploaded. Most FTP clients use a similar strategy when downloading files from an FTP server - the file usually ends up with a modification date set to the date the file was downloaded.
This can sometimes lead to unexpected behaviour in the context of a mirrored directory. For example, if you have some files on an FTP server and you mirror them to a local folder on your computer. If you then modify one of the files and attempt to mirror the files back to the server, you find that not only is the modified file uploaded, but all the rest as well. The reason is that the downloaded files have a date which is more recent than the files on the server. However the result is that you're better off not using a strategy based on modification dates to copy these files around, and reverting to some other process (such as a manual one).
Mirror takes two steps to minimise these sorts of problems. The first is to reset the modification date of downloaded files to match the date given by the server. Mirror can do this because unlike most FTP clients, Mirror examines the names and dates of files on the server before downloading them. This strategy solves the problems encountered when mirroring from, then to, an FTP server as given by the example above.
In addition to setting the correct modification date of downloaded files, there are some circumstances under which Mirror can also set the modification date of an uploaded file. If the FTP server understands the MacBinary protocol (specifically, the MACB command), Mirror will use MacBinary encoding for all files sent to/from that server. As the name implies, MacBinary encodes lots of Macintosh-specific information about the file, including the modification date. A file uploaded with MacBinary will have the same modification date as the original.
When mirroring to or from an FTP server, Mirror does a directory scan to extract the file names and modification dates in order to determine which files are to be copied. It is common for FTP servers to supply such information to clients, however it is not specified as a requirement by the FTP standard. As such, there is no common standard format for the information.
The defacto standard on the Internet, and that supported by Mirror, is the UNIX directory listing format. This is essentially the same as the "ls" command on a UNIX system. Mirror interprets the results from such a directory listing and extracts the file names and dates.
This obviously causes a problem for other kinds of FTP servers, and there are plans for Mirror to support these servers in a future version. In the meantime Mirror is incompatible with FTP servers that do not support a UNIX-like directory listing format. Note that this does not mean that the FTP server must be running on a UNIX operating system, merely that can it pretend to be one when asked to list the contents of its directories.
The UNIX directory listing format has a slight drawback however in that it does not list the file modification dates down to the last second. For instance, a file dated 12:00:30 would be listed on the FTP server with a modification date of 12:00. Further still, the modification date of some files is listed as the day and year.
When Mirror is scanning directory listing formats, it takes note of the degree of precision in the modification date and takes it into account when comparing dates from the other mirror site. For instance, the dates of local files are known to the last second, but this component of the date is ignored when comparing against a file on an FTP server.
The result of all this is that Mirror sometimes does not have access to all of the information it needs to determine whether or not to copy a file. Bear this in mind when using Mirror. It would be unwise, for instance to perform two mirrors in the space of a minute and expect changes made to files in the interval between these mirrors to be propagated through.
Mirror treats aliases differently from other files on the local file system. If you are mirroring from a local site, you can choose to have mirror either follow the alias or ignore it. Aliases are always followed when mirroring to a local site.
Following an alias means searching for the original item to which the alias points to. If the original item is a folder, its contents are examined during the preflight. If the original item is file, the name and modification date of the file are examined during the preflight. Note that the alias itself is merely a pointer to the original item, and that it's name is not significant. This can have some unanticipated side effects.
For example, consider the case Mirror is mirroring to a local folder. During the preflight, a file called Arnold is encountered on the original site, and so Mirror dutifully looks for a file with this name on the mirror site. Suppose an alias with the name Arnold is encountered, which points to the original item, a file called Dave. Mirror has looked for a file called Arnold and found only a file called Dave. Hence Mirror will copy Arnold across to the mirror site, replacing the alias called Arnold.
Another potential pitfall can occur when mirroring from a local site. Mirror walks the folder hierarchy when performing the preflight, recursively examining all the folders contained within the local site. If Mirror is told to follow aliases, then aliases to folders will be examined, as well as real folders. You must ensure that Mirror is not sent into an infinite loop by making aliases between two folders that are to be part of the local site. That is, if Mirror is to follow the aliases in your folder hierarchy, is it going to end up in another part of the hierarchy? Mirror does not automatically guard against this eventuality, so it is suggested that if you are in doubt, tell Mirror to ignore aliases.
Mirror uses the URL name encoding scheme as per RFC 1738. This means that filenames and directory names with "special" characters in them are changed such that the special characters are encoded. For example, files with spaces in their names are changed such that the spaces are replaced by "%20". Many other characters are changed to their encoded representation of a percent sign followed by the ASCII value of the character in hexidecimal.
The reason for this encoding is that most operating systems have "reserved" characters which may not occur in a file or directory name. For example, the backslash character "/" cannot be used in a filename under the UNIX operating system. Under MacOS however, this is a perfectly legal character. Mirror encodes filenames under the guidelines set down in RFC 1738, which is adequate for the most commonly-used operating systems today.
The pitfall to this scheme is that some FTP servers don't understand the encoding scheme. So if Mirror attempts to create a directory on the FTP server with the name "Folder%201", then the folder is created with the name as specified, and not with the desired name "Folder 1". Unfortunately there are still many FTP servers which do not understand the encoding, and there is not much that Mirror can do about this. Upgrading the software used on the server can often fix the problem - for instance the "wu-ftpd" server package for Unix servers understands encoded filenames and is a lot more functional than the standard FTP server package for most Unix platforms.
Attempting to mirror large sites, or several sites simultaneously can require more memory than Mirror is allocated by default. If you are experiencing out of memory errors, try increasing Mirror's preferred memory size from the Finder.
Mirror will intentionally not delete files which exist on the mirror site but not the original. Such obsolete files need to be cleaned up manually for the moment. An update to Mirror 1.0 is expected shortly after the public release, in order to address this limitation.
To resolve operational errors with Mirror, it is best to consult the log files which are automatically produced and placed in the "Mirror Logs" folder in the same folder as the application. One log file is produced for each calendar day of operation.
It is worth noting that these log files can accumulate over time, and you should delete old log files periodically in order to free up space on your hard disk. It is expected that a future version of Mirror will perform this task automatically.
We regret that no direct support for Mirror is available. Progmatics will however endevour to answer any queries emailed to mirror@progmatics.com.au, or posted to the comp.sys.mac.comm newsgroup. Preference is given to paid licence users.
Thanks to: Peter Marks, George Bray, David Wareing, many others...