The home for this Modem FAQ, such as it is, is http://www.actrix.gen.nz/faq/modem.html.
This FAQ is undergoing a major rewrite, and will be updated over the course of the next few days (he says, optimistically!)...
Feedback to arnim@littek.actrix.gen.nz
The word modem is an abbreviation for MOdulator/DEModulator. Modems are used to modulate, or mathematically speaking to multiply, the data signal onto a carrier signal for transport over a medium where the data signal does not happily go.
In our case here, we are talking specifically about modems for modulating RS232 serial data from the serial port of our computer onto an audio carrier for transmission on the telephone line, but there are MANY other kinds of modems, just to confuse things.
If you're bored already, skip on, otherwise, here's a bit of historical background/explanation of the above.
Historically, modems started out by "multiplying" a slow, simple data stream onto a set of tones. One tone represented a logical binary ON, the other a logical binary OFF. By way of example, we can use the A below middle C at 440Hz, as one tone, and the next higher A, at 880 Hz, as the other tone. If the serial bit stream from our computer looks like:
1 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1for example, this modem would send a sequence of:
1 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1
| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | |
440 880 880 880 440 440 440 880 440 880 440 440 880 440 880 440
using a simple method called frequency shift keying. This sounds
like just what it is, a pair of tones flipping back and forth. They
are flipping back and forth at the rate of the data being sent. In
the case of my example here, if we sent data at more than about 100
bits per second (bps), we would start to land in hot water, but
that's another story.
The other end, the demodulator for this simple arrangement, needs only two tone detectors, which are looking for 440 and 880 Hz. The signal is reconstructed from this, and then sent on to the computer attached to it.
This basic example is at the heart of modem theory. There are other ways of doing modulation than this simple multiplying. This is also called amplitude modulation, like AM radio, which carrier an audio signal mixed onto a radio carrier frequency. And like radio, there are good reasons why old AM is not necessarily the most effective way to carry a signal.
This example above shows that we are encoding our data signal straight on, with no bit encoding. Bit encoding is used for all higher data rates, because if we used simple frequency shift keying, we could only use data rates up to about 1200 bits per second (bps) before Telecom would start complaining that we are using more frequency than we are allowed on a telephone channel. There are important technical reasons for keeping our signal within the width of one telephone channel, ie. between 300 and 3400 Hz.
So, to use a really simple example of bit encoding, we could use 4 tones to encode two bits. We take the following coding, which is only illustrative, to indicate the mechanism:
bit1 bit 2 freq
0 0 440
1 0 880
0 1 1320
1 1 1760
and now we re-encode our bit stream:
1 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1
\ / \ / \ / \ / \ / \ / \ / \ /
880 440 1760 880 880 1760 1320 1320
And you can see where, last time we had to send 16 pieces of
information to encode our 16 bits, this time, we only needed to send
8. This information piece is called a baud. We are sending, in this
example, two bits per baud, whereas in the first example, we were
sending one bit per baud.
Extrapolating, with 8 different tones, we could send the same 16 bits of serial information in only 4 baud. Further, we could send it with only two baud if we used a 16-level encoding scheme.
This sounds wonderful, but there are tradeoffs. For this simple example, using only frequency shift keying, we cannot send data any faster than a few hundred bits per second, otherwise the tone multiplication overlaps the next tone so much we could not decode it at the other end, but I'm getting a bit technical.
In order to get away from that, these discrete levels are sent in real modems separated in both frequency and a second dimension called phase. This is getting analogous to FM radio. In radio you can send a signal in both AM and FM on the same carrier, or you can even send two separate radio signals on the same carrier, one modulated AM and the other FM. So too, we can send signals modulated in two fashions, and this lets us get more levels onto the carrier, ie. more bits per baud. Correct combination of a different amplitude and phase for each baud can give us a modulation scheme called Quadrature Amplitude Modulation (QAM).
For a 2400 bps modem, the QAM modulation scheme is called V.22bis, according to the Recommendation of the same name from the International Telecommunications Union (ITU) which is the international body looking after this sort of standard. This is a 16 level scheme, ie. four bits per baud. The information is sent at 2400 bps, but at 600 baud. For a 9600 bps V.32 modem (operating in standard non-trellis coded mode - another trick) it likewise uses 16 levels, ie. 4 bits per baud, but uses a baud rate of 2400.
Bell 103 or | Although Bell 103 and V.21 for instance use
V.21 - 300 bps | the same data rate, they have slightly
Bell 212 or | different modulation schemes.
V.22 - 1200 bps |
V.22bis - 2400 bps |
V.32 - 9600 bps | All of these are for standard dialup lines
V.32bis - 14400 bps | ie. 2-wire full duplex service.
|
V.FC or | There are others for other kinds of
V.34 - 28800 bps | service levels too.
V.34bis - 33600 bps |
Things get a bit more complicated after this, as we get into coding
tricks to make the most out of the available signal-noise ratio, and
as we approach issues like echo cancellation to sort of double the
available bandwidth on the line, but I'm a little loathe to start
getting into details like that here. People write entire books on
the theme.
If someone starts tossing around numbers like V.29, V.17, then they are talking about fax stuff, which is a separate kettle of fish again. George Pajari maintains a pretty useful FAQ on fax stuff, for further pointers for the curious. I haven't checked recently, try the source, at http://www.faximum.com/faqs/fax
This is also complicated further by the fact that the newer protocols, since V.32 (9600), actually have mechanisms whereby they don't have to operate at their nominal top speed, but if the conditions warrant it, they "fall back" to a lower speed, to be able to live with a particularly bad line or some such.
Recommendation: Speeds:
V.32 9600, 4800
V.32bis 14400, 12000, 7200
V.FC 28K8 down to 14400 in 7 steps
V.34 28K8 down to 4800 in even more steps
V.34bis 33K6 down...
I'm showing my biases a bit, but I've not included any material on the
K56 and X2 pseudo-standards for operating a PSTN modem at 56K bps.
Neither of these two is anywhere near through the ITU-T standards process, and the commercial infighting between these two competing proposals for the next increase in speed for modems, is actually splitting the marketplace in a wasteful fashion.
But worse than that, the entire 56K bps technology is, in my opinion being oversold. It ONLY works at 56K in one direction, from your ISP to you. It ONLY works at 56K under ideal circumstances, in real world situations, you're lucky to get any more than 33K6. And it is being badly misrepresented by sales people in more situations than I care to recount.
If you've got a REALLY good line, if you live quite close to your local telephone exchange, if you have no other gear on the line, if your ISP has one of the two forms of 56K bps modems, and if your current modem hits 33K6 EVERY time, then maybe its worth trying 56K. Maybe. But don't bet the farm on it. There are better technologies, with much better speed improvements, like ISDN or the xDSL technologies.
On the other hand, if you're buying a modem as an upgrade from an existing 9600 bps or 14400 bps modem, then, here in NZ anyway, its getting hard to find a modem on sale in the glitzy shops that is not advertising one or the other 56K bps protocol. These modems still fall back to the 33K6 speeds etc, so you won't be worse off, and the 56K technology that you get won't actively hurt you... No expectations = no disappointments.
In summary, though there is a physical limit to how much data can be sent down a given line - this is well expressed formally as Shannon's Law. Telecom gives us a specific bandwidth, and within that bandwidth, we can make up increased bits per second by increasing the bits per baud. But, for every baud, there is less margin for error between the individual baud. This means that the potential for error gets larger. The lines have to get better and better as we use more bits per baud. This margin is called the signal-to-noise ratio. You can hear it sometimes yourself, how much or little signal-noise ratio you have, on a particularly bad line.
Of course, the lines don't get better just because we get a better modem, as we all know. It means, in the end, that higher quality lines achieve better data rates, mostly by stuffing more bits per baud. So if you have a noisy line or a line with low line levels (high signal losses), you will not be able to send data as rapidly as say, I would on my relatively good line here at home.
For those of you playing with the V.34 or V/34bis modems, you should not be alarmed if you are not achieving 28K8 or 33K6 connections. The modems are deciding between them which line speed is going to achieve the best line rate - bear in mind here that faster is not always better - if you go too fast, and the line is error prone, you spend more time doing retransmissions than if you had started out one or two notches slower... In the end, it is your throughput that matters, not the given line speed.
What can you do about a lousy line? Well, one of the worst things for modem line quality tends to be a PABX. If you are running through a PABX, but can argue your way out of the situation, do so. Many PABXs have a marked impact on line quality. Then we have other equipment on the line - or features you have from your local telecom provider, like call waiting - not good for modems. Eliminate as much of this interfering equipment as you reasonably can. I've got my modem at home on its own line, but not everyone has that luxury.
Inherently, there may not necessarily be that much you can do if you are saddled with a poor line at home. Telecom does not guarantee ordinary phone lines for data service, and to get a conditioned line costs $$$. But the quality of the modem on the end of the line makes a difference too. If you are having trouble with a particularly bad line, it is a good idea to at least try another modem which someone is happy with, to see how well it operates on your line. The more sophisticated modems will have a setting where you can actually get line levels from the modem by an AT command (more on that later).
I'll try not to spark any religious wars here. This stuff applies principally to modems for the IBM-PC (and clones).
For the PC, you can buy a modem card which fits into your PC, for perhaps 10-25% less $$$ than the equivalent standalone modem. This has a few advantages, and a host of disadvantages. Don't let me bias you though...
On the one hand, it is cheaper to buy the card modem. It is also nice and tidy and fits into your PC, away from any prying fingers etc. No power supplies to fiddle with, and no serial cable. That's the good stuff.
But, invariably, there is a performance hit. The signal-noise ratio is not as good on an internal modem - it is closer to a whole lot of electrical noise (your PC) and is not shielded at all well, generally speaking. That's the biggest thing. If you are playing with slow, ancient modems, this may not be such a big deal, but I for one would not touch an internal V.34bis or V.34 modem. They need to be reasonably sensitive to be reliable, but the inside of the PC is not at all suitable for sensitive electrical devices.
And if you dump your PC before you dump your modem, well, the internal modem is only good for use in PCs. Not in a PowerMac, or a Sun or an SGI Indy or anything else you might desire. But external modems are good everywhere, and in a whole lot more applications, like synchronous, or leased line or other applications. (At least they used to be... )
If you live in a lightning prone area, internal modems are somewhat worse off. They are a hazard to the internals of your PC. So what if you fry your external modem, you will be more likely able to read your hard disk. There's no guarantee even with an external modem, but the odds are simply worse with an internal modem.
Additionally, if the going gets tough, it is nice to have the modem indicator LEDs to tell you what is going on. As a someone with a technical bent, I depend on them for feedback, but I am sure the rest of humanity is not necessarily as strong on this as I am.
There are more pros and cons, but those are the big issues, in my eyes. But, I'm not biased, right?
Because of some of the difficulties in making a PC work with a serial port in a one-bit-at-a-time fashion, some ambitious folk have been pushing the concept of a modem running on the PC parallel port, ie. shoving around 8 bits at a time in parallel.
If only it were that simple. Maybe there is something to be gained by this (I fail to see where, especially against a UART with a FIFO) but I think most of the gain is in a custom software driver rather than in any attribute of the parallel port itself. Most implementations of parallel ports on PCs are at least as stupid as the stupidest serial cards, ie. offering no hardware assistance to lighten the PC's workload.
If that isn't enough, all the myriads of software used in communications applications, from drivers through to applications with in-built serial port activity, will not work with a parallel port modem. It needs special software, not much of which exists.
Take it for what it's worth - I'd be happy to be proven wrong. (Note that I have no experience with or desire to use EPP)
What does that mean?
The modem takes RS-232 serial data from your computer. Serial data is information sent one bit at a time on the same line. If I take the example up above (section 1.1 etc.):
1 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1and assume that the bit on the left comes out first, and the one on the right last, then this can represent a serial bit stream coming out of my computer. Strangely enough, this kind of stuff can actually convey a great deal of information.
For a little more on how this relates to connectors etc, see 2.5.1. below.
For instance, let us assume that our information is sent as ASCII text. This is the most common way to send information, including email etc. Some ASCII characters are:
A 1000001 a 1100001
B 1000010 b 1100010
C 1000011 c 1100011
D 1000100 d 1100100
E 1000101 e 1100101
etc. up to
Z 1011010 z 1111010Note these are 7-bit characters. Looking at our data stream as 7 bit characters, from above:
1 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1
\______________________/ \______________________/ \___
we note we have two characters with the first two bits of the next
character. As it happens, the first ASCII character is a "G", next
one is a "+". Note also, that if we took the same data stream and
broke it up into 8 bit chunks instead, it would mean something else
entirely different. You have to be sure that whatever you are
talking to is assuming the right number of bits in a character...
This is called setting data bits, and is for most RS232 applications
today set to either 7 or 8 data bits. Actrix, our host, uses 8 data
bit format.
By the way, ASCII stands for American Standard Code for Information Interchange. There are international standards based on essentially the same character set, as well as extensions for the unusual letters in some of the European languages etc.
But the base of 7 bits only allows 128 characters, not nearly enough to accommodate languages like the pictographs of the Oriental languages. These are fully accommodated in a coding scheme which uses 16 bits, called Unicode. Unicode is a relatively new thing, but should become more widespread over coming years. A 16 bit coding scheme can account for 64K characters, ie. 65536 characters, so this takes care of the thousands of Chinese pictograms, the non-Roman alphabets like Arabic and Thai and many others, with some space to spare.
There are two basic formats for serial data - synchronous and asynchronous. There are lots of spinoffs and variations on these, but these are the basic.
A synchronous data stream is the easiest to understand. Say we have a data line which changes in level, and we have a clock line to tell us when the data means something.
1 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1 <-- our data
__ ____________ ___ _______ ____ ____
____________ _____ _____ ____ ____ <-- signal
| | | | | | | | | | | | | | | | <-- clock
If we put a meter or measuring instrument on the signal line, we see
the signal moving up an down, or ON and OFF, or whatever you want to
call it. Each time the receiver sees a clock edge, the last line,
it says AHA, now I can test the line and take a reading, because now
it means something. This is a really simple way to operate.
But, PCs, Macintoshes and other such computers do not use this method generally for their standard serial communications. Sync operation generally means you need a separate clock line for each signal line (ie. one for the data in each direction - to and from the PC). There are other reasons too why sync data is not used so commonly at low serial data rates.
Notice though, that there is no fiddling with figuring out how fast the data is coming. The clock tells you how fast the data is getting there, and that's all there's to it.
Now, for asynchronous (meaning "without clock") serial ports, the receiver has to figure out how fast the data is running, or else it has to be set up to ASSUME how fast it is running. And to make sure that the data is not sampled just exactly at the edge, when the receiver might make an incorrect assumption about the real data state, the receiver samples each signal multiple times.
In real asynchronous receivers, data is sampled 16 times (generally), and the best fit value is taken. Asynchronous data is also framed in a start bit and a stop bit, so that the data is always "marked" with an additional start bit and a stop bit (or two, in the old days). Otherwise you might never know it was there, particularly if the data happened to be all 0 0 0 0, which happens a lot.
If you know enough information about the data stream you are going to receive, you can figure out what the data rate is by looking at the data. For instance, if an asynchronous modem is first turned on, it doesn't know what speed the incoming data will arrive at. But, modern Hayes-compatible modems such as we are discussing here, all know that the first thing they SHOULD receive is an instruction from the computer. And every instruction in a Hayes-compatible modem is prefixed with the letters AT. So, if the modem knows this, it looks for the data pattern and reads it, and then figures out the speed because it "knows" that the first data it sees is supposed to be an "AT" in ASCII.
Modern modems can have autobauding turned off if you always want to operate at the same serial port speed, but the command to do this in the AT command set for your modem may be very hidden and hard to find. A good modem manual will almost certainly have this option somewhere in it.
Now things get a little complicated. Back in the bad old days, modems talked at the same speed on both ports. The line speed, ie. the bps rate that went to the telephone line, was exactly the same as the serial bps speed that came from your PC. Note I did not say baud rate, since that is different depending on the line coding.
So, if you had your modem dial a line at 2400 bps, the PC was supposed to work at 2400 bps. Very simple, in the ideal setup. Very primitive.
But, the modems got a bit smarter, and then they discovered that for some bad lines, it was a good idea to have the modem "fall back" in speed. This meant that when the modem discovered that the line was not good enough to operate at one speed, that it was supposed to arrange with the other modem to go at a slower speed to accommodate the worse line. If done appropriately this rate renegotiation can be done without having to "retrain" to the line conditions, a process which can take several seconds or even more.
Initially, the modems only got a little bit smarter. That meant that if your modem got a bad line and fell back from 2400 to 1200, you had to suffer, and had to live with a serial port speed of 1200 bps. Bad stuff. All you could do with these primitive modems was hang up and redial, hoping to get a better connection and the data rate you really wanted.
But it is not very hard to make the microprocessor in a modem talk at one speed on the serial port and at another speed to the line. It needs to have a little memory in between, to store the difference in the data, but this is all quite do-able with asynchronous circuits.
So nowadays we have modems that operate at 33600 bps (V.34bis) on the line but which can talk at 115200 bps or even 230000 bps on their serial port. This operation is technically called speed buffering, and is almost universally only available on modems that have another feature - error correction, which we will come to later.
As an aside, in formal terms, RS-232 is specified up to 20K bps. So, these data rates are, umm, well beyond the RS-232 specification. But, more recently, some smarty-pants standardised RS-560, which is a derivation that is specified for these high asynchronous data rates that originated in the PC world.
As soon as we have differing speeds on one side of the modem from the other, we NEED something called flow control. Using an example of the 33600 line speed and the 115200 serial port speed, we have a difference of 80K bps. Since the memory in a big, smart modem almost never exceeds about 128K bytes, which is 1M bits, if all of the memory was used to save this 80K bps we could hold out for about 12 seconds before the memory would fill up. In practise the memory allocated for this "buffering" task is much smaller. So, the modem has to be able to tell the PC to "STOP". Then, when the line speed of 33600 has managed to send out the backlogged data, the PC can start up again. This is flow control. The modem is flow-controlling the terminal or computer, to say when there is room in its buffer to store more data.
There are two separate ways to do flow control:
If you have a serial cable from your modem to your computer with more than three signal lines in it, which almost all do, your modem should probably be set up for hardware flow control.
The RS232 (or V.24) standard sets out quite a few signals. We don't care about synchronous stuff in this discussion, so we will forget about clock lines. But there are other lines besides the two directions of data:
Connector DB25 IBM-AT
Standard DB9
Shield - 1
Transmit Data TxD 2 3 data ---> modem
Receive Data RxD 3 2 data <--- modem
Request to Send RTS 4 7 control ---> modem
Clear to Send CTS 5 8 control <--- modem
Data Set Ready DSR 6 6 control <--- modem
Ground GND 7 5
Derived Carrier Detect DCD
or \
Received Line Signal RLSD - 8 1 control <--- modem
Detect /
Data Terminal Ready DTR 20 4 control ---> modem
There are more signals on the standard DB25 connector, but most PC
users will only be familiar with the deviant DB9 connector on the
serial port of IBM-AT clone computers, which has only these signals.
I apologise for not addressing the issue of Macintosh serial ports. They make no pretense at being RS-232 compatible. Please refer to the very comprehensive FAQ on Macintosh Communications available on the Net: ftp://rtfm.mit.edu/pub/usenet/comp.sys.mac.com/c.s.m.c.FAQ[1-4]
The lines of relevance for flow control in the PC-modem world are RTS and CTS. All widely used PC software for instance, assumes that flow control is done with RTS to indicate that the terminal can accept data, and with CTS to indicate that the modem is happy to accept data.
Each modem which has error correction will have a specific AT command to set up hardware flow control. Recommended, if you have any choice in the matter. It should be a factory default setting, but you might want to check to be sure. (Factory default means that it should be set up that way out of the box... if you get it out of the box.)
For those who must, there is an alternative. Software flow control, or XON/XOFF flow control, assigns one ASCII character to tell the flow controlled party to "STOP" sending (XOFF) and another (XON) to start.
This has only limited applicability. Firstly, your software must only ever send data which does NOT include XON or XOFF in the data stream. This means if you use software flow control, you will not be able to do most binary file transfer.
It generally requires more work from the microprocessor in the modem or your computer, and hence is generally slower than hardware flow control. Forget it for speeds over about 19200.
This can be useful for applications using only ASCII text transfer, and in situations where for one reason or another, your serial port is only providing TxD, RxD and GND lines, a poor man's serial port.
A terminal is a video screen reflecting the input from a serial port. Simple enough, on the surface, but there are any number of ways to control how to use that screen - control meaning things like special commands to clear the screen, or move the cursor around, or make things appear in colour, or make beeps and so on. There are some widely used sets of these special commands, from terminals that are or were very popular.
The only "standard" of a sort is the way screen handling is done on a basic IBM-type PC, in the sense that this ANSI terminal handler is labelled with a standards body name. I presume this means that ANSI has actually put something in place, but I can't at this time point you to a specific ANSI standard. (ANSI = American National Standards Institute)
The most widely used terminal emulation is called VT100, from DEC. Most modern comms software should be able to deal with either one of these two, at least enough to get a communications session started. After that we get into all manners of other terminal emulations like Wyse, Data General, IBM3270 and so on. The latter is a bit special, and will not run on your basic async serial port hardware, but let's not confuse the issue.
The end effect is that if you are using an inappropriate terminal emulation, you will be quite likely to end up with funny characters on your video screen, or worse, not be able to see what is supposed to be happening. And forget any function keys if your terminal emulation isn't right. They only work on the right terminal/emulator. Lastly, be aware that even if a software package claims to do VT100 emulation or ANSI, it may not necessarily do a good job of it. Obscure or even common commands may be left out, or worse, implemented completely incorrectly. Such is the way of life with software, especially low-cost/no-cost stuff.
Modems were, believe it or not, not always as good as they are now. We used to assume, when the first 1200 bps modems came out, that this was hot stuff, but that it would not always work on bad lines. Similarly when 2400 first happened.
About the time of the appearance of the 2400 bps V.22 bis modems, Microcom was putting more microprocessor into modems, and came up with the ideas of:
If the data arrives correctly, fine, but if not, the sending modem is forced to retransmit the erroneous packet of information, until it gets it right. This is a much simplified description of the process, but essentially, that is what error correction does.
It only works like this for asynchronous serial ports, and only with speed buffering, because, you see, every time it finds an error and asks for a retransmission, it must stop the data coming from your computer, to catch up first. I won't confuse the issue with sync protocols or windowing protocols here...
The packets which are used to "frame" your data incur a little extra overhead, so that makes for another reason to require flow control.
Fortunately, the process of packetising the information allows a form of asynchronous-to-synchronous data conversion, from your serial port to the modem. This conversion means that the modem no longer needs the bits on the asynchronous data that mark the bit - the start and stop bits. Synchronous data is inherently a little more efficient this way. In fact, on 8 bit data, an asynchronous serial port sends 10 bits - one start, 8 data and one stop bit. So, the same data in a synchronous format takes only 8 bits - this is a saving of 20% of the data to be sent.
As it happens, with an appropriate choice of framing, Microcom realised that the async-sync conversion could actually produce a net GAIN, something of the order of 8%, ie your async data is sent in (100 / (100 + 8)) x 100% of the time over this packetised link. This async-sync framing is called MNP Level 3 or 4, depending on the type of framing.
While MNP was (sort of) a proprietary framing technique, Microcom donated it to the world, and it became a most effective de facto standard.
Shortly after that, the international telecommunications standards body, the CCITT (now called ITU-T) streamlined MNP a bit, and released a standard called V.42, which is in essence very similar, if somewhat more efficient. This more efficient protocol is technically called LAP-M - Link Access Protocol for Modems - but V.42 also consists of a fallback mechanism whereby it will use MNP Level 4 if LAP-M is not permitted.
Error correction on modems is not perfect, but when a good implementation is working well, it drastically reduces the chances of any errors ever showing up on your computer. Not all implementations of error correction are good but finding bad ones is not so easy nowadays.
The process of checking and even retransmission of data should be transparent to you as a user, ie. you will not even be aware of it without a suite of fancy test equipment to watch what is going on. Unless you have a very bad line and there are a lot of retransmissions, in which case the transfer of data will slow down perceptibly.
Its important to note, the error correction applies ONLY between the two modems. There is no requisite error correction going on at this level between your PC and the modem, or at the other end. If you've got dickie cables or a high noise environment, the error correction between the modems may be all for naught. True enough, its not as likely, but don't ever forget it can be a problem!
Now, error correction takes up processor power in a modem, and it requires some memory. This is precisely the kind of thing that a PC can do too, given half the chance. Rockwell, the people behind the commodity modems dominating much of the marketplace, have placed a protocol in the market which takes this processing power out of the modem and puts it into the PC. Unfortunately, they didn't make this a public specification, so it is quite difficult to find software that does the PC end of the RPI. Not very good marketing.
And, it is not very difficult, in terms of processor power, to leave that function inside the modem. Indeed, in these days of power hungry users, I am inclined to think that the more power we put into a peripheral, the more we leave left over for the computer to use as we want it to use on our application.
The world seems to think the same way, and RPI is not catching on. It is an orphan specification, the prime justification is squeezing the last penny out of the cost of modem hardware. It seems only the cheapest of the cheap modems are considering this mode of operation. It is in the same camp as parallel port modems in my opinion - I won't touch them. But, you can choose to ignore my opinion, I won't be insulted....
A soft modem is one way of describing the provision of the modem protocols by a completely reprogrammable processor. If it is done properly with a good, grunty DSP engine, then it will do as good a job as any other internal modem.
There's also a school of thought (school of cheap?) that ascribes to using the main CPU, ie. your Pentium or PowerPC, to do this. This class of modem consumes a disproportionate amount of your CPU's processing power, so, I would be tempted to avoid them. The cost of modems now is so low, there's no justification in burning off precious CPU cycles doing a lot of the grunt work necessary for modem signal processing. But that's clearly my opinion.
This is a much overblown feature of modern modems, but it does have its place. Now that we know about speed buffering (ref. back to section 2.4), flow control and error correction (above), we can make a bit of sense of this, I hope.
With an error corrected link working, we can assume the data is arriving intact. Almost every piece of data has some redundancy in it. Thnk of thse wrds. You can still read them, because there is enough information there for you to understand what the missing letters are. They are, in that sense, redundant information.
Data compression uses a host of tricks to remove redundancy from data. Like replacing long strings of the same character with a character and a count. And many more sophisticated schemes. The topic of data compression is much researched.
In addition to the framing async-sync conversion process in MNP Level 4, Microcom also came up with a data compression scheme, MNP Level 5, which on some data reduced the amount of information on the line by up to as much as 50%, ie. the compression was working at up to 2:1.
This was something of an ideal figure, and is seldom ever achieved. For MNP to achieve a 2:1 compression, the data has to have a LOT of redundancy in it. Graphics is good for this sort of compressibility. Text will achieve perhaps 1.4:1 or 1.7:1 under good conditions. And your precompressed files like ZIP, ARC, SIT or CPT files, they will not compress with MNP Level 5 at all. In fact, by the nature of MNP, you are better to deactivate data compression on pre-compressed files because MNP isn't smart enough to figure out that it is dealing with packed data. Don't turn off MNP Level 4 error correction though, it's still working well...
MNP Level 5 also became something of a de facto standard. It is quite useful, even if it might not be perfect.
Again, the CCITT worked toward an international standard, building on V.42, with V.42 bis. This has a very clever bit of data compression technology in it (and is patented by IBM, Unisys and British Telecom in many countries), is very suitable to modem technology, and is a lot better at getting out of the way when presented with incompressible data. In general, use it where you can in preference to MNP Level 5, for these reasons. As with MNP, your mileage will vary with the quality of the implementation in the modem you are using.
Since then, Microcom has answered with some "higher level" data compression schemes, some of which may be better under some circumstances than V.42bis. In general, these have not been so well accepted in the marketplace, and compatibility is a problem. You must always have two modems capable of the data compression before it will work, so if your modem does not talk MNP 7, or if the other modem does not talk MNP 7, then it will not work.
The whole process of data compression is transparent to you as an end user, you should not ever notice, except that the effective serial port data rate will be faster than without it. Or it should be.
So, we've got the above bits straight? Let's put into action. You will need communications software, and the simplest thing to start with is a program called generically a terminal emulator.
Telix Zterm TERM
Telemate Eudora Kermit
Procomm VLT Seyon
are some of the examples I know of. With patience, one can even use
the Windows Terminal program, packaged with every Windows PC, to do a
basic job. Even the dialers for your PPP or SLIP software will sometimes
give you a manual mode to let you type AT commands, although sometimes
how to arrange this is not obvious. It is hard to be specific with
so many software packages around. And don't anyone waste my time with
questions about Windoze '95. Its against my religion.
I don't have the expertise to coax each of you through getting the serial port of your computer to talk to the modem, there are too many types of computer around. But, start up the communications software package and it should put you into a blank looking screen. If your cable and modem are hooked up right, you will be able to type
AT <CR>where <CR> is "ENTER" or "RETURN" or whatever that large key is on the right of your keyboard.
and the modem should answer with
OKalmost no matter what setting your comms software is set up for. If you've got this far, you have the cabling right, the serial port right and the modem is ready to set up. If not, well, back up, read the manual on your computer to figure out how to use your serial port. Or else be sure you have a good cable. Or that the modem is in fact powered ON. Or that it is a Hayes "AT" modem at all.
If you have an external modem, check to see that when you hit the <CR>, that the TxD or Transmit Data or TD indicator flashes on the modem. If you are getting an OK back, the RxD or Receive Data or RD light will also flash briefly, while the modem sends the "OK". (If you have an internal modem, you are sort of stuck for that information.)
If neither is happening, suspect the cable or the serial port or that the modem is powered ON. If only the TxD light flashes, the modem is either not understanding that you sent an AT (in which case it might send an:
ERRORmessage, or it has been specifically instructed to say nothing. If you think this is the case, try restoring the original factory configuration:
AT&F <CR>which should get you an
OKif the modem had been put into "quiet" mode, ie. set up to not reply. Otherwise, the modem may also be in a "dumb" mode, which is different for every modem manufacturer. You will have to find the manual to get any modem out of dumb mode. It may not be easy.
Now I have to assume you have received your first response from the modem. Somewhere on the screen of the comms software, it will say that you are operating at a serial port speed of 1200, or 9600 or 38400 or some sort of legitimate data communications speed. The list of options will probably not exceed:
50 14400
75 16000
110 19200
150 24000
300 25600
450 28800
600 32000
1200 38400
1800 48000
2400 56000
3600 57600
4800 64000
7200 76800
9600 115200
12000 230400
and if you have even half of these, you have a
pretty capable software package on your hands.
If your modem makes any mention of MNP or V.42 in the documentation, you can operate, no, you SHOULD operate at a higher serial port speed than the fastest line speed your modem can handle.
Using the manual or the on-line help of your comms software, you can change the speed at which you are talking to your modem. Do this, and each time you do so, retype the
AT <CR>again, and get an OK. The modem is now autobauding, ie. automatically detecting the speed and data format of the AT that you are sending, and when it knows, it answers with an
OKat the same speed and in the same data format.
If you have a V.22bis modem capable of a line speed of 2400, but with MNP or V.42 capability, set the serial port speed to 4800 or 9600.
If you have a V.32 modem capable of 9600, almost certainly you will have MNP or V.42 capability, so set your serial port speed to at least 19200, maybe even 38400 or higher.
If you have a V.32bis modem capable of 14400, you will have MNP or V.42 capability, so set your serial port speed to 38400 or higher.
If you have a V.34bis or V.34 modem capable of 28800, you will have MNP or V.42 capability, so set your serial port speed to 57600 or higher, if your software will go that fast.
Sad to say, some software does not talk very quickly. For one, if you are running Windows 3.1 with the original Microsoft COMM.DRV still in place, no matter what comms software you are running under Windows, it will not go faster than 19200. Newer versions, from Windows for Workgroups 3.11 on, has that fixed. Some software, particularly older software, will not run even this fast, and some older computers won't run this fast either...
I cannot know all the software packages, so I won't be able to tell you which ones are capable of what. The manual/documentation for your software SHOULD tell you this information. Otherwise, just experiment.
You can, if you feel so inclined, also fiddle with the parity and data bits settings on the communications software package, since any decent autobauding software will be able to see through that and keep up with your changes. But if you are going to do any file transfer etc. afterwards, you're better off to leave it in 8 data bits mode (see section 2.1 for explanation).
UART - Universal Asynchronous Receiver/Transmitter - the serial port controller chip. PCs in particular seem to have a problem with getting them right. The original PCs ran on NS8250 UARTs, which were slow and cranky and buggy. The AT class PCs ran on the next generation NS16450 and a few clone UARTs, which started to appear. As serial ports climbed past 19200, the one-byte-at-a-time scenario was too hard for a PC to keep up, the serial ports were bogging down processors, particularly if multiple ports were working.
A well known solution from other parts of the data comms world is the provision of a FIFO - a First In First Out memory to store up a few bytes of information and THEN call the processor, so it handles 4 or 8 at a time, cutting the overhead of fetching by a factor of 4 or 8. The first issue of the FIFO version, the NS16550, was buggy, and in fact the FIFO didn't work, but after the version went to NS16550A, things have been right. There are quite a few clones of this IC now, and most of them seem to work, in my experience. If you are now going out to buy a PC or a serial card, and want to use it with these fast V.34bis or V.34 modems, its a good idea to insist on getting a serial port with a 16550 or equivalent serial controller chip in it. I had to spend a few extra dollars to get a serial card put in a PC specifically with some 16550s, but there was no choice... the built-in UARTs only emulated 8250s, and that's just not good enough.
For more detailed information there is a FAQ on just UARTs somewhere on rtfm.mit.edu, the massive, wonderful News Archive ftp site.
For those of using Windows 3.1 or earlier, the ability of that software to run any communications software was pretty limited. Anything over 9600 bps is a waste of time, Windows cannot keep up. The situation seems to be much better in Windows for Workgroups 3.11, Windows '95 or NT.
There are fixes for Win3.1. If you replace your COMM.DRV (in your
C:\WINDOWS\SYSTEM directory, or wherever your equivalent lives)
with one of the following, and then use a 16550 UART as described
above, your problems could be over.
Both of these packages used to be widely available, but are less so
nowadays. However, a month or two ago, I managed to dig up a copy
for a friend still running Win3.1 on his laptop, so using a search
engine on the Web will work:
CYBERCOM
cyberc.zip or cyberdrv.zip
WFXCOMM.DRV
wfxcom.zip
The Cybercom offering goes to 115K2 bps, while wfxcom only to 57K6.
However, the wfxcom package is a little more tuneable, I am told.
Some modems have a "view" command which lets them tell you how they are set up. Good modems will tell you more information, and lazy manufacturers will tell you less. Look to the manual for detailed information, but the two commands I have seen the most to give this information are:
AT\Sor
AT&VThe first one often gives lots of detail, whereas the Rockwell modems and the others that use the latter command, just give you a naked list of command and register settings which is hard to interpret without a manual at hand.
For variations try
AT\Sn where n is a number, say, 1, 2, 3, etc
AT&Vn where n is a number, say, 1, 2, 3, etc
to get additional information.
Try also the following where n is a number, say, 1, 2, 3 or more.
ATIn
If you have any choice, use hardware flow control, ie. RTS-CTS flow control, or out-of-band flow control. Refer to section 2.5 for details.
If you absolutely must, use in-band flow control, or XON-XOFF flow control, but remember this cannot work with pure binary data transfer.
If you have a dead stupid modem or must operate in the most basic direct mode of the modem, then you will be able to work with no flow control. And only then.
Flow control of this type only applies to asynchronous serial lines. It applies to every line where two sides of the modem are operating at different data rates, such as we are recommending here.
Generally, a good factory configuration, via the
AT&For
AT&Fn where n is a numbercommand should set up a configuration with flow control enabled.
The factory configuration should be pretty sensible, it is a good place to start. If your software package is expecting to hang up the modem for you, it will probably want to do so via the DTR line. The modem is set up to do this via the
AT&D2command. When the software wants to hang up, it forces the computer to "drop" the DTR line, which tells the modem in this mode that the software is finished and would like to break an existing connection.
One of the more intimidating aspects of setting up a software package that talks to modems is the aspect of INIT STRINGs. When you see something like:
AT&FS0=0&D2E0V1\V1\Q3\N3S11=7&C1you start to get worried. What does all that gobbledygook mean?
For starters, the easy bit is that all that gibberish translates to a whole bunch of separate commands, each of which should be found in your modem manual.
AT&F
ATS0=0
AT&D2
ATE0
ATV1
AT\V1
AT\Q3
AT\N3
ATS11=7
AT&C1
The hard bit is that although all dial modems are
sold as Hayes-AT compatible, there is no standard AT
command set, and so basically every modem manufacturer
has inflicted at least some proprietary commands on
you.
So, init strings are either modem-specific or really only set up an absolute minimum functionality. If you have a V.34 modem and it gets set up as a minimal Hayes-compatible, you will be severely underutilising it. Like, no error correction or data compression.
One way around this is to set up the modem in the best fashion you can via the factory default settings and flow control etc., and then store this in the modem's long term storage, or non-volatile memory, and then just tell your comms software to use these startup settings. How do we do this?
Well, after you have made a selection of things YOU want done in your setup, you store it with the AT&W command. Using my setup string as an example:
AT &F2 &D2 S0=0 &W
^ ^ ^ ^
| | | |
| | | -- write command stores MY init setting to
| | | modem non-volatile memory
| | ------- turn off auto answer
----------- DTR disconnect
--------------- factory default setting #2
which is V.42bis, RTS-CTS flow control etc.
I can now put the simple "restart" sequence in my init strings:
ATZwhich calls up the settings I saved above.
There are all kinds of other kinks and twists in this procedure, but because modems are such flexible beasts, and because people use them in such different ways, and because no two modem manufacturers implement the same AT command set, it is difficult to be more specific than this.
Its a good idea to avoid using the &W command in an init string itself. The init string gets called every time you open a program with your modem - ie. every time you fire up your terminal emulator, or your SLIP or PPP dialer, etc. This means hundreds or thousands of times a year, if you are really into it. But the chips that store the &W stuff are NOT built for lots of writing. You can read them as often as you like, but the garden variety non-volatile memories often have a lifetime guarantee of only 10,000 or perhaps 100,000 write cycles. While that sounds like a lot, it is prudent to avoid writing unnecessarily to them, hence the advice.
Not meaning to play favourites, or to intend any endorsement, we have had some experience in getting Dynalink V.32bis modems configured, and we can recommend, for the 144VE, for example, the init string
AT&F&D2&C1S95=255And there are all kinds of twists on the small subset of "standard" commands I quoted above - like US Robotics modems which don't use the ATS0 command, but rather a DIP switch, or a few modems which use volume controls instead of AT commands. Some of these things are so obviously more sensible than AT commands...
I was amazed to find out that there is an ANSI/EIA/TIA standard covering (thanks George) the Hayes command set - EIA602. I will get a copy and summarise. In the meantime, here are those basic commands which I have seen implemented pretty much on every one of the many dozens of "Hayes compatible" modems I have ever had the misfortune to play with...
ATA - force answer mode
ATB - select Bell or CCITT modes (all but obsolete now)
B or B0 = CCITT
B1 = Bell
ATD - dial - see your manual for details as this is a flexible and
powerful command with lots of neat options.
ATE - echo on/off - echo is the copying of a command from the
terminal or computer back to it
E or E0 = off
ATH - hook status, ie. line state
H or H0 = on-hook = off line
ATI - product code or identity code, some manufacturers put in
options to provide more or less information about the modem
being interrogated, for example:
I or I0 = ID and version number
I1 = checksum of EPROM
I2 = OK or ERROR on validity of checksum
ATL - speaker volume
L or L0 = low volume
L1 = medium volume
L2 = high volume
ATM - speaker function
M or M0 = speaker off
M1 = speaker on until carrier detected (normal)
M2 = speaker on while modem on line
etc.
ATO - originate - force originate mode or resume data state when on
line.
ATQ - result code output
Q or Q0 - normal result codes sent to terminal
Q1 - result codes not returned to terminal
ATS - set or read S register - see below
ATV - result code format output
V or V0 - single digit result codes generally most suitable
for a computer to interpret
V1 - "english" result codes suitable for human interpretation
ATX - enable result code formats and tone detection such as busy and
dial tone
X0 - most basic, just indicate CONNECT
X1 - full message indicating line speed, eg. CONNECT 9600
etc. See your modem manual.
ATZ - reset modem to startup condition
Z or Z0 - restore to conditions set in &W0
Z1 - restore to conditions set in &W1
etc.
AT&C - DCD/RLSD mode
&C or &C0 - DCD always indicated on to terminal
&C1 - DCD follows modem carrier
AT&D - function of DTR line
see your modem manual
AT&F - set up factory configuration
see your modem manual
AT&W - store present configuration in modem's non-volatile memory.
see your modem manual and the ATZ command above.
These represent only a fraction of the commands available in any one
modem, but beyond this, implementations of the AT command set begin
to diverge widely and wildly.
When implementors of modems are too lazy to conjure up a new AT command for a new function, the function is instead left in something called an S-register. These vary even more widely between manufacturers than do the AT commands, so you are best off to remain away from these if and when you possible can. Good manufacturers will have most or all of the functions in any S-register replicated in an AT command, with the exception of the first 12 "classical" S-registers:
S0 = number of rings before modem is permitted to grab the line
(0 = no answer, ie. AutoAnswer off)
S1 = ring count - read only, cannot be set
S2 = escape code character - normally set as "+" to use +++
S3 = carriage return, or RETURN or ENTER, character
S4 = line feed character
S5 = backspace character
S6 = wait after going off hook, before dialling
S7 = length of maximum wait for carrier after dial
S8 = pause for "," in dial string - see details in your modem's
ATD command
S9 = carrier detect response time
S10 = lost carrier to hangup delay
S11 = tone duration during tone dialling
S12 = escape code guard time
If you really want to have a further explanation of any of these,
email me, and I will walk through it with you, but when you do so,
quote me VERBATIM what it says in YOUR modem manual, to make sure
that it is at least close to what I understand the function to be.
Anything else is, as far as I am aware, completely open slather for the modem manufacturers to implement any which way they see fit.
Dennis Hayes and Co. started off the Hayes command set more than a decade ago. They released most of it into the public domain. What they did not do was release the command state exit method.
A modem operates in one of two modes on the serial port. When you start up, it is in command mode, and will accept the AT commands. Then, when you dial, it becomes transparent so your data can pass without changing the modem. That's the data state.
To get back to command state, one method is to use an escape character sequence, which whenever occurs, throws the modem back into command mode. Hayes patented their method of sending three characters (generally now using the +++ sequence) surrounded by a minimum "dead" time, called a guard time.
If you've reverted to command mode via +++, typing
ATOat any time will then allow the modem to resume the data mode, if you're still "on line".
There is a small, but finite possibility that such a sequence could show up in user data. In order to avoid dropping out of data state, some applications, such as the modems at Actrix, have their +++ capability disabled, by setting the escape mode character to a disallowed state instead of the "+" character.
Chris Blum's TTY1-3.FAQ stuff is considerably more detailed than this, if you are into it. It looks pretty accurate, by and large too. I have a copy if you want one, or ftp://ftp.phil.uni-sb.de/pub/chris to go to the the source, as it were.
If you need more than that, you're getting into serious serial port programming, and the quintessential source on this low level stuff for PCs, MS-DOS etc, has to be Ralf Brown's most excellent Interrupt list, available on SimTel and the related good ftp sites around the world.
Enough for now, let's get your suggestions as to what to add/correct or improve in this. If you read this far, you deserve a medal.
Arnim.
Arnim Littek arnim@actrix.gen.nz
Actrix Networks Ltd. fax +64-4-801-5335
tel +64-4-801-5225