An unnamed customer of mine called one day complaining of terrible performance on one of the company's NT servers. When I arrived on-site, the customer explained that the company had just installed a new server and had tuned the system as the white
papers had recommended. Unfortunately, users were complaining of delays in running the client-server application that attached to the server. On further examination, it was very clear that the system was severely overloaded and lacked some type of system
resource. The paging frequency and low-memory warnings in the Event Viewer showed that the system did not have enough physical memory. It turned out that in its planning sessions, the customer had made a small miscalculation in determining memory
requirements, which gave 32M of RAM rather than the 128M needed. Needless to say, the next system put into production was thoroughly tested and piloted, with simulations of estimated system load after tuning was done.
In my opinion, Exchange Server performance tuning is like selecting a very fine wine and cheese combination: you keep sampling and testing until you find the combination that works exceptionally well together. Selecting the right hardware options to
complement your software poses the greatest challenge, because the hardware performance options change almost weekly. Purchasing the right system to meet today's needs must be balanced with the needs for growth and expandability as you add more users and
business processes to your messaging system. Underestimating current and future requirements can adversely affect the users' perceptions of the system's capability to meet their ever growing needs.
Every system has some type of bottleneck that limits some aspect of system performance. Hardware tuning involves finding and eliminating these bottlenecks until you reach an acceptable level of performance. Determining where the bottlenecks are in any
system requires good tools and structured testing methodologies. Simulating system load in a lab environment is not a good way to find a system's limits; real messaging systems are confined by external forces such as networks and user behavior that can not
be simulated in a "clean" lab environment. The planning and pilot phases of any implementation require good benchmarking and logging applications to assist in documenting the actual performance of the system.
This chapter discusses the concepts and methodologies used to tune your Exchange system to an acceptable performance level. You need to have a good grasp of what your current messaging system is doing before you can begin to shop for hardware for your
Exchange messaging system. A practical approach to help you select servers and performance options is included, with tips for optimizing network access. The text covers the tools included with the Exchange Server that enable you to gain access to the
numerous configuration parameters within the registry and databases. The best way to prove that your design will meet your company's needs is to run a pilot project with a cross-section of your user community, and use simulated clients to determine how
well the servers will perform under the estimated load.
The best way to know where you want to go is to fully understand where you have been. Would a doctor prescribe a treatment without first reviewing a patient's past medical history? Would an architect draw up plans to remodel a building without viewing
the previous plans and performing a site survey? Would you really want to be the customer in either of these scenarios if the proper planning and research was not done? The same importance is put on a proper analysis of your current messaging system and
the habits of the end-users. By using the valuable information contained within your current e-mail system, you will be able to design and implement your next-generation messaging system.
The process of documenting your current system involves gathering, parsing, and publishing the information for your planning sessions. The methods you use to gather data from your current messaging system depend on what tools are available. The parsing
of the data can be done with many different software analysis tools. The tool could be as complex as a customized application specific to this purpose, or as simple as using office automation products such as Microsoft Excel and Access. To publish and
present the data to the project team, you can utilize any of the most popular network modeling tools, or just use your favorite word processor and spreadsheet.
Within your current messaging infrastructure are valuable messaging statistics just waiting to be uncovered. The way you extract that information might be very formal, or might require some imaginative methods of using features that were not designed to
perform data collection. If you are currently not using a messaging system, this section will still be helpful because you will be able to create virtual data derived from your current messaging needs. You need to collect five main categories of data:
Message delivery time is the total amount of time it takes to move a single message from a sender to the intended recipient, averaged by hour, day, week, or other time frame. This total includes the time it takes for the message to move from the client
to the post office, the transfer time from the sender's post office to the recipient's post office, and the time it takes for the recipient to retrieve the message from the destination post office. The time it takes for delivery of any message is affected
by its size, complexity, route, and number of conversions. A difference also exists between actual and perceived message transfer times because of the human factor involved.
This is probably the easiest information to extract from your current system. Many of the most popular messaging systems have counters you can use to calculate this data. You should fully enable system logging within the system, as well as turn on any
external logging tools. You can parse these logs, depending on the format, to determine the start and end times for each message sent to determine the delta time of message transfer. This task could be as simple as searching the message headers for the
keywords Sent and Received within Excel to use in a formula that calculates the delta time. It could also be as complicated as writing a custom program to parse the logs into a more readable format. You should select which period of time to average the
data on and be consistent throughout your analysis.
For example, if you are using an MS-Mail messaging system, these logs can be created by the message transfer agents. By enabling the options in the MTA configuration for logging of sent and received messages in verbose format, the files SENT.LOG,
RECV.LOG, and SESSION.LOG will be created in the \LOG directory on each post office that MTA services. You can combine this information with the estimated polling frequency of the clients to determine the total time it takes for message transfer. A simple
Excel macro, created by an import wizard, can be created to parse these logs and run selected formulas to generate the transfer times.
Message layover, or latency, is the amount of time a message is stored at a post office when it is not in transit. Because most systems are based on the store-and-forward delivery model, every message must spend some time in a static state within a post
office's area used for "storage" of messages. Calculating this time is more challenging because you must determine the time the message arrived and the time the message left. Many systems use unique identifiers for each message, which can be used
to get the in and out times for each message. You might have to use the subject line combined with the time stamps to uniquely identify each message.
Expanding on our example, you would use the same sent and received logs at each MS-Mail post office to calculate the layover time. If a particular message has entries in both the received and the sent logs, the message "hopped" through the
post office by an indirect routing method. Excel can also be used to import and parse these logs to compare each message that shows up in the received log that has a match in the sent log to determine the amount of time the message stayed at the post
office.
The message volume is the total number of messages sent and received between two distinct points during a specific time period. Most messaging systems have some type of message transfer agent that can be used to get this information. You should identify
each point of interest for tracking purposes, and calculate the message volume between all these points. As the number of points increases, the number of connections you must track will increase by a factor of the number of points. For instance, if you
have 4 destinations, you would have 16 different connections to calculate the message volume for; 5 destinations would give you 25 connections, and 10 would give you 100.
Continuing on the example, MS-Mail has a system utility included in the resource kit that performs this function. This utility, called the MS-Mail Postoffice Traffic Analyzer, or TRAFFIC.EXE, needs to be run at a dedicated workstation during your
collection period. It calculates the number of messages sent to each externally defined post office. You then can export this information into a format that Excel can use to create graphs and calculate the average message volume data you need.
The preceding example for MS-Mail is good at getting message volume information for inter-post office traffic, but it will not get any data on intra-post office messages sent. You can get this data by setting up an automated process to copy the file CONTROL.GLB from the \GLB directory on a post office to a time-stamped file. By comparing values in these files, and subtracting the number of messages sent inter-post office, you can get a good estimate of the number of messages and attachments sent within the post office. This procedure is detailed in Microsoft's tech note #Q122852.
The average size of messages transferred over a connection is a value in bytes divided by the total number of messages you calculated in the preceding step. You can best determine the average size of messages by analyzing the logs created by your
system's message transfer agents. Many systems give you the total amount of data transferred by some time period, and you should use the number of messages transferred during that period to calculate the average size. Some systems also give you the maximum
and minimum message sizes, but you should discard these values if they skew your results or have a distinct pattern.
In the MS-Mail example, this information is calculated and generated by the message transfer agent and stored in the SESSION.LOG if the option LogMessageVolume is present in the EXTERNAL.INI file. This provides message volume summaries for the number of
messages sent and the total number of bytes transmitted and received for a given time period. The default time period is once per day, which should give you a good idea of the average size of the messages sent through this post office.
The average number of errors or problems for a connection is a numerical value of the number of message delivery errors per connection divided by a given time period. This information is almost always logged in any messaging system because it is used
for troubleshooting purposes. It is recommended that you stick to using the actual logs from your system, and should not attempt to generate this data from your help desk or problem-tracking system. The error logs are an actual representation of the errors
for a connection, but your help desk is only human perception of error conditions.
To finish out the MS-Mail example, you could extract error information from the SESSION.LOG for each post office and match the instances to identify a problem for a particular connection between two post offices. This task requires some advanced parsing
capabilities, which is a good job for PERL (Practical Extraction and Reporting Language). Those of you who are supporting and maintaining Web sites will be very familiar with PERL. You can use PERL to parse each line of the logs, to determine whether the
line is relevant through pattern matching, and to keep arrays and totals for significant statistics. You could also use PERL in any of the previous examples because it is very good at working with these types of files.
Data is truly relevant only if it is organized and presented in a readable and comprehensible manner. The volume of data you parse from your system might complicate your options for tools to present the data. Certain tools have limitations on the amount
of data they can handle, so you should be sure that your selection is capable of working with your volume of data. The way these programs present the material also differs, with some choosing a more graphical approach and others using the technical essay
format. Choose a program that meets your requirements for presentation and ease of use, because you might not have the time to switch to another program or learn a new program during your documentation phase.
Custom written applications you can use to present your information include such products as Crystal Reports, Visio, and other third-party programs. None of these is specifically targeted at working with the data collected from your system, but they
have powerful graphical display or reporting capabilities that can use data from external sources. It is best to load your data into some type of ODBC-compliant database, because these programs can link directly to this type of data source to get their
content. Crystal Reports has direct data-extraction options for many of the most popular messaging systems, which reduces the time you need to extract and parse the data. Visio is a better solution for generating diagrams for your post offices and
connection, displaying the content in a more visual format.
If you have a very small system or decide that you do not want to learn a program you are not familiar with, you can use office automation products such as the Office95 product suite. Microsoft Access has some very rich importing and reporting
capabilities and is best for larger volumes of raw data. Excel is good at importing and parsing various log files and can be used with Access and Word. Word is the application that can pull data from Access and Excel to present it in a technical essay
format. The integration between these products, as well as most people's familiarity with them, will prove more useful in small to medium-sized documentation projects.
Whatever solution you use to present the information obtained from your data collection process, the data should be verified to be accurate and relevant. You can apply some practical knowledge of your system to verify that the data you have collected
properly depicts your system. Only you and your technical staff can properly validate the statistics you collected because it is your job to keep the system running. If you are using this section to plan for a new Exchange messaging system with no current
system to replace, you should import your "virtual" data into these same tools and discuss them in your planning sessions.
Hardware planning, the process of selecting the fastest "toy" that your budget can afford, can be very complicated and time-consuming. Actually, the process of hardware selection is not so much a search for the most bang for your buck, but a
collection of concepts and methodologies that can assist the system designer in meeting practical performance requirements. I don't know many computer engineers, including me, who would not jump at the chance to get their hands on the latest technology.
Just buying the "fastest" server on the market, however, does not guarantee that it will meet your performance expectations or requirements.
The hardware industry is constantly in a state of flux, with new and faster hardware options coming out every day. Although you should attempt to keep up-to-date on the hardware industry's newest innovations, don't discount the tried and tested systems
that have a well-established history. You might also want to see whether you can utilize your current server hardware to meet your Exchange messaging needs, and with a few upgrades you might be able to get performance numbers to your liking. Keep in mind
that your messaging needs of the future will require much more hardware than you need for today, so plan for a system that has scaleability and upgrade options.
Don't kid yourself—Exchange Server is very resource intensive and will possibly require more hardware than is available in your current system. But you should consider the number of different systems that it can replace while still meeting your
company's messaging needs. Many of you are downsizing from a multimillion-dollar mainframe-based messaging system to a server platform that costs only in the thousands of dollars range. There are no cooling towers or lumbering air handlers for these
systems, because most will fit under someone's desk.
You can break down the planning for your server hardware requirements into the following five hardware categories:
There is one very solid rule you should keep in mind when looking at memory requirements for NT and Exchange Server: you can never have too much memory. Exchange Server, like any good NT-based application server, is a series of multithreaded processes
that allocate and deallocate system memory as needed. Each of these multithreaded processes can utilize any memory that the NT Server software can see, up to maximum hardware limitations somewhere around 1–2G.
If your system does not have enough system memory to meet the requirements of the operating system services running on the server, including Exchange, the system begins to page physical memory to a virtual memory swap file located on the disk subsystem.
This action not only creates a resource problem with memory availability, but it also competes with database file operations on the disk subsystem for I/O bandwidth, causing an even greater system slowdown. Running out of physical memory and creating a
"disk thrashing" situation is the worst situation you can be in if your server is in production mode.
To determine the amount of memory you should have in your servers, you need to make some decisions about what additional services will run on the server. Following are some examples of server processes that will increase a system's memory requirements:
Each of these services increases your memory requirements above those of the operating system and Exchange services. As a general rule, you should plan for at least 16M of RAM for the operating system, 16M of RAM for other services such as the ones
listed, and at least 16–32M of RAM minimum for the Exchange services. As your number of users increases, the amount of memory for Exchange will increase approximately 100–150K for each connected client, not including background tasks for message
transfer and replication processes. Use the guide later in this section to help determine a rough estimate of the amount of memory your system might need.
The Achilles heel of hardware performance is the disk subsystem, because even the fastest SCSI controllers can sustain transfer rates of only 20M per second. Even though network cards can only achieve around 12.5M per second transfer, multiple cards can
sustain much higher transfer rates when using PCI technology. Taking into account that the memory and bus are much faster than the disk throughput in most server class hardware, you can see how the disk subsystem can truly be the limiting factor of
hardware performance. New RAID controllers can help by writing to many spindles in a RAID5 configuration, spreading the load over many drives working at the same time with a large memory cache to speed the total throughput.
This discussion assumes that the disk subsystem is utilizing SCSI-based technology because IDE interfaces are currently limited to just under 5M per second of bandwidth. It is also assumed that you will use an EISA or a PCI bus due to serious
performance problems with ISA-based systems. Because FAST-WIDE SCSI-2 drives and controllers are readily available, settle only for this type of SCSI technology, unless you are upgrading an existing server that has a non-wide drive built-in controller.
Getting the best performance out of your server's disk subsystem involves planning for adequate drive space and multiple disk channels. Exchange Server requires only approximately 120M of drive space, but your messaging data will use much more. The
current release of Exchange supports message store sizes up to 16G, with the directory service also having a maximum of 16G for its database. For this reason you should plan your drive space accordingly and allocate the right amount of space for each
user's mailbox in the Information Store settings at the site on the server. For instance, if 100 users will use an Exchange Server, and your message stores are located on a drive with 4G free space, each user should not have more than 35–40M of space
allocated. Single message instance, where a message is stored once in the database with the user’s mailboxes just pointing to that message, will reduce the actual amount of space available for each user, but it is better to have some extra room for
growth.
Because even the fastest PCI-based hardware caching controllers rarely exceed 25M per second of data transfer bandwidth, you can greatly enhance disk performance by increasing the number of controllers and drives within the system. If you increase the
number of SCSI controllers to three, with one being a hardware RAID5 controller, your system performance will be enhanced by the creation of different I/O channels for the Exchange services to use. Optimally, one channel should have a mirrored pair of
1–2G drives for the operating system, Exchange executables and tracking logs. The next channel should have a 1–2G drive dedicated to the DS and IS transaction logs. The RAID5 controller is used for the public and private message stores and could
have 16G or more of storage for each message store. This is an optimal solution for the best level of performance, but you might decide to only use two channels to keep your costs down.
Raw speed and data-transfer times are not the only factors you should consider when designing your disk subsystem; fault tolerance also plays a very important part. By using the built-in fault tolerance options available in NT Server, you can guard your
system against drive failures and improve your system performance, depending on the option you select. Table 25.1 lists the fault tolerance options available in the disk administrator for NT Server, as well as the best way to use each for Exchange Server
storage requirements.
| Option | Description | Exchange Storage Options |
| Disk Mirroring | Data is read and written to a pair of identical disks, allowing failure of one | NT operating system, Exchange executables, and tracking logs |
| Volume Set | A proprietary method of concatenating hard drives to create one virtual drive | Can be used to expand storage for the IS and DS databases, but performance will suffer |
| Disk Striping | Data is written across two or more drives with no fault | Used for the personal and tolerance for a drive failure (RAID0) private Information Stores with very fast performance |
| Disk Striping with Parity | Data is written across three or more drives with a parity stripe calculated and written for each data segment, allowing for a single drive failure | Used for the personal and private Information Stores with a good balance between speed and fault tolerance |
Although NT Server has these options available for performance and fault tolerance, the best choice for high-speed reliable access are RAID hardware caching controllers. These Fast/Wide SCSI-based controllers have RISC-based processors that
perform the striping and parity calculations without putting extra load on the system processor. They also usually support features such as hot-swap and on-the-fly rebuild of replaced drives. If you are building a server that will support more than 500
concurrent users, it will require very fast hardware caching controllers with many hard drives to get acceptable performance.
One of the best ways to increase the performance of any Exchange Server is to use a dedicated drive for the transaction logs. Both the Directory Service and the Information Store databases use a transaction log to store transactions before they are written to the actual database. When MAPI data is received by the server for storage in the DS or IS databases, it is written from memory to a fast, sequentially organized transaction log file. When the IS database process is not busy, the transaction is written from memory to the database, and a checkpoint is incremented for the transaction log. This technique not only increases performance because it can write to a sequential file very quickly, but also can be used for crash recovery of the databases. If you dedicate a SCSI channel to this purpose, database writes and transaction log transfers will not compete for bandwidth on the same channel. For an extra two to three percent increase in speed, format the dedicated transaction log disk with the FAT file system. The smaller sector size and reduced security storage inherent with FAT will give a slight amount of performance increase on heavily used servers, but will eliminate the advanced security checking and error recovery available in NTFS.
The speed and number of processors will affect the overall performance of your Exchange Server, but you will actually need less processor horsepower that you think. Throughout the history of computer evolution, software performance problems have been
fixed with faster and faster processors. This was due in part to operating systems that were not utilizing the advanced features of these new processors. OS/2 and Windows NT were the first operating systems to run on Intel processors that could actually
utilize the enhanced performance options available in the fastest processors. Then came SMP and multiprocessor systems to push the performance ceiling even higher, and the advanced operating systems such as NT stepped up to the challenge with
multiprocessor support.
The Exchange services fully support and utilize the fastest processors on single- and multi-processor–based systems. The amount of processor your server will require can differ, depending on the number of users, the types of services, and the
number of messages handled. The following Exchange services will increase the need for processor performance:
In planning for your server hardware, consider purchasing systems that support dual processors or full SMP expandability. Not all Exchange Servers need multiple processors to perform acceptably, but as your system grows and new demands are placed on it
for business processes, you will need scaleable processor resources. The general rule for selecting processor type and speed says that Exchange will run on a 486/33 processor with adequate amounts of RAM, but you will get the best bang for your buck if you
use Pentium- or Pentium Pro-based systems.
All client and interserver connections must be made through the network card and the associated physical network. This can be the one area where system performance can suffer because of bottlenecks created by forces outside the server. Networking
technologies such as Ethernet and Token Ring are based on shared bus or token passing access methods, and they can run at speeds from 2 megabits to 100 megabits, depending on the cabling and timing specifications. Network cards are the server's link to the
physical network, as well as the connection medium for all external connections. Typical Ethernet network cards are capable of supporting the network traffic associated with approximately 250 users simultaneously transferring data, even though many more
could be logged on not transferring any data at the same exact time. You can increase the amount of network bandwidth on your server by installing multiple network cards connected to different physical segments, and use protocols such as IPX/SPX or TCP/IP.
Token Ring and FDDI network cards can also be installed in multiple numbers, but these technologies have much higher bandwidth capabilities than Ethernet so fewer would be needed.
When it comes to Windows NT, the performance of any network card is dependent on the quality of the drivers that support it. Most of the newest cards come with drivers that allow you to set up the options for the card such as interface, interrupt, and
memory address, but the driver should also support high-speed direct transfer modes. Most of the EISA- and PCI-based cards support busmastering to reduce the card's processor utilization. Many Ethernet cards support full-duplex mode connections, and this
option is highly recommended if your hubs or switches support this mode. You might need to use standard network benchmarking software to determine the aggregate network bandwidth, and use that information to gauge the requirements for your type of network
conditions.
If you decide to use multiple network cards, remember to plan which protocols are bound to which cards. If your network primarily uses the TCP/IP protocol, you must assign a unique address to each card. With the NetBEUI protocol, you can bind it to one
or more cards as long as they are not on the same segment and cannot "see" each other. The IPX/SPX-compatible transport can be bound to multiple cards; however, each must have a unique internal IPX network number, and they should be on different
network segments. You will get the best performance with multiple network cards if one protocol type is bound to each card, enabling you to isolate network traffic by protocol.
The system backplane and bus make up the I/O subsystem that connects all the system components for data transfer. Older motherboards for 486-class systems did not have very fast backplanes even if they used a PCI or EISA bus for add-in cards. Because
the backplane connects all the system components together, you should look for high bandwidth architectures such as Compaq's Tri-Flex or Intel's Triton or Orion chipsets.
Whether you select an EISA or a PCI bus for your server hardware will depend on your budget and selection of add-in cards. The EISA bus is a proven technology that supports data transfer rates of 33M per second, just above the sustained bandwidth of
most disk controllers. PCI, a newer technology that can run as high as 155M per second, is geared toward systems with very fast network controllers and multiple high-speed cards. You will probably want to select a system that uses both technologies to give
you a better choice of add-in cards for network and disk controllers. The ISA bus was not mentioned because it should not be an option for your server; it is a desktop-system technology with very poor performance and increased processor utilization under
heavy loads.
You might be wondering how you begin the process of selecting which options you will need for your server to support x number of users. This question is very subjective because one population of 100 users will use e-mail completely different than
a separate population of 100 users will. The best way to gain confidence in your selection of server hardware and the number of your company's users it will support is by using the LoadSim utility that comes with the Exchange Server software. This utility
is covered in the section Running a Pilot Test later in this chapter, but Table 25.2 gives you a very rough guideline for the hardware options you should use to support various numbers of users.
| Range of Concurrent Users | Processor | Memory | Disk Subsystem | Network Subsystem |
| 0–100 | Single Pentium 90MHz | 32–48M | Single channel with 1 or 2 drives | Single Ethernet 10BaseT or 16M Token ring card |
| 100–250 | Single Pentium 100MHz with dual processor option | 48–64M | Single channel with three or more drives | Dual network cards from 10M duplex to 100Tx Ethernet or Token Ring |
| 250–500 | Dual Pentium/Alpha/MIPS 100MHz or higher | 64–128M | Dual channel with one software or hardware RAID for the IS | Dual network cards from 10M duplex to 100Tx Ethernet or Token Ring |
| 500–1500 | Dual Pentium/Alpha/MIPS 133MHz or higher | 128–256M | Three FAST-WIDE SCSI-2 channels with one being hardware RAID5 | Multiple network cards from 10M duplex to 100Tx Duplexed Ethernet or Token Ring optimized for protocols |
| 1500 and up | Two to four Pentium Pro/Alpha/MIPS 200MHz or higher | 256–1024M | Three or more Advanced Hardware Caching Controllers using RAID technologies (PCI) | Multiple network cards from 10M duplex to 100Tx Duplexed Ethernet or Token Ring optimized for protocols |
Your actual hardware configuration will differ from these recommendations, depending on your hardware vendor of choice and your budget for procurement of this system. Two years from the date this book is published, these recommendations will
seem ridiculous compared to the hardware technology available at that time. Don't settle for a hardware configuration that you know will not meet your users' messaging needs, because whatever money you saved in hardware costs will be spent in support
costs. Use a practical approach to selecting and testing your server hardware—this is your messaging system we are taking about.
You might have selected the fastest server hardware available to run Exchange on, but if the underlying network is having bandwidth or stability problems, you might have just purchased a very expensive coffee table. Networks can be very complicated in
design due to the numerous technologies and connectivity options. A company's network infrastructure can be broken into the local area network (LAN) at a location, and the wide area network (WAN) for connectivity to other LANs. Also, dial-in and remote
access are an extension of WAN technologies, but are geared toward supporting remotely connected users.
This discussion uses Ethernet technologies in the examples, but similar concepts can be applied to other network technologies, such as Token Ring and Asynchronous Transfer Mode (ATM). If you currently do not have a LAN or WAN and are planning to design
and implement a new one, you should consider the recommendations in this section as a part of your total requirements. Network design should be done by individuals who specialize in this area to properly meet your company's requirements.
All the data that is transmitted between the users and Exchange Servers is carried over the physical network. Information that is transferred between servers and sites by MTAs is transmitted and received over the same physical network, but is usually is
carried across routers or bridges that connect physical LAN segments. Before you begin to map out your Exchange site topology, you need a detailed diagram of your company's entire network for reference. As mentioned earlier in this chapter, you should use
these network diagrams to analyze the various connectivity options for optimization of network bandwidth usage.
How well is your local area network performing today? Does your network experience frequent transmission delays due to overload or high error rates? Do you currently have network analysis tools available to determine the health of your physical network?
You should be able to answer all of these questions before considering ways to optimize and increase performance of your LAN segments. All the computers within a physical location are connected to the network by a network card and physical cable or wire,
and they must share network bandwidth with the other network nodes for the transmission and reception of data. A good understanding of your current network performance can help in determining which technologies can be used to increase bandwidth and
decrease transmission times.
Ethernet networks use an access method called CSMA/CD, or Carrier-Sense Multiple Access with Collision Detection. Carrier-Sense refers to the process the network card uses to "listen" for a quiet period it can use to send data. Multiple Access
refers to many computers sharing the same physical network cable or segment. Collision Detection is the process by which computers detect collisions on the shared wire and wait for a random period before they retransmit their data. Network collisions are
normal for any typical Ethernet network, but if experienced in high volumes, they can indicate bad network cards or physical wiring problems. You can use the following guidelines to increase network bandwidth and reduce collisions and retransmissions:
Your options for upgrading or enhancing your existing network will depend on the type of hubs and cabling within your network. Network management utilities such as sniffers and network monitors can be used to identify segments that require modifications
to increase performance. Microsoft SMS and Windows NT 4.0 have a program called Network Monitor with server-based agents that can be used for this purpose. You should analyze your network during normal business hours to determine performance when the users
are connected. You should also separately analyze the network after hours to determine performance when nightly processes such as backup or database maintenance are running when the users are not connected. A six-month history of network statistics in an
easy-to-read graph will give you a good idea of your overall performance and help identify times when the network is under a heavy load.
Communications between Exchange clients and servers use synchronous remote procedure calls (RPCs) for all data transfer. RPCs do not require a dedicated amount of network bandwidth, but you will get the best performance when your LAN is operating at
less than 25 percent utilization. Because Exchange is based on the client-server architecture with connectivity over RPCs, you should see a decrease in the amount of network utilization compared to your current messaging system. You should also group users
who frequently mail to each other on the same Exchange Server to limit the amount of message transfer between servers or sites.
Physical locations that have one or more LANs can be interconnected by WAN connectivity options. The core components of a WAN connection are a bridge or router, DSU/CSU or equivalent, and the physical connection provided by a carrier's network. Due to
the options available for WANs today, they are referred to as slow links. Connections range from the slowest speeds of 8K per second to hundreds of megabits per second. Typically, most companies use connection speeds of 56K–1.5Mbit, depending on the
bandwidth needs between physical locations.
Tuning the performance of the WAN link involves setting up filters on the bridges or routers to weed out unnecessary protocol traffic between the connected sites. You should take careful consideration when setting up these filters to your specific
protocol requirements to allow them to work across the WAN link. If you are using NBT (NetBIOS over TCP/IP) to connect two Exchange sites, make sure that UDP ports 137–139 are not closed to traffic, because they are required for connectivity between
the Exchange Servers. The NetBEUI protocol is not even a choice for a WAN because it is not routable, and it would require a source-route bridge to forward broadcasts across the WAS. The IPX/SPX protocol is not the best choice for WAN links due to the
"chattiness" from the Server Advertising Protocol (SAP) broadcasts, but updated implementations from Novell can reduce the effect these SAPs have by decreasing their frequency from one minute to hours or more.
The various Exchange site connectivity options have minimum available bandwidth requirements for reliable message transfer. The term available bandwidth refers to the amount of bandwidth left available after the requirements of other
applications are met. For instance, a 256K frame relay connection might have only 10–15K of bandwidth available after an enterprise database's replication needs are met, whereas a 56K connection might have 40–45K available due to limited
inter-site traffic. If you have a WAN connection of less than 56K, you should consider an upgrade of the connection speed. The minimum available bandwidth needs of each connector were outlined in Chapter 24, "Interfacing with
Other Mail Systems," but the connectors normally require 56–128K of available bandwidth, depending on the message volume expected between the sites.
By examining the network traffic patterns of your WAN, the dollar costs associated with each link, and the estimated message transfer volumes between Exchange sites, you can design your message routing topology to best utilize your network resources. By
using a limit on message size for a particular connection, you can eliminate the ability of a single user to effectively take down your WAN by sending a message with an enormous file attachment. By assigning scaled costs to your connections and scheduling
connections, you can control the impact that message transfer will have on a particular WAN link.
Is the number of employees at your company who work from home or live on the road increasing? The nature of business is changing to incorporate "virtual offices" and telecommuting users who might rarely go into the office, but still perform
the same job functions. Salespeople are spending more and more time on the road as the size of their sales territories expand. The challenge of providing secure and reliable access to the home office has to be balanced against the current available
technologies. Modems have hit a theoretical brick wall when it comes to link speed, and new technologies such as ISDN and cable modems are still finding their place in the industry. Efficient use of these types of connections is the best way to gain better
performance for message transfer.
Remote access solutions for connecting users to a branch or central office must support Remote LAN Node (RLN) connections for the Exchange client to operate correctly. Remote control is a completely different concept, and it does not work as well when
it comes to the messaging needs of users today. The Exchange client uses the remote access software included within the operating system for Windows 95 and Windows NT and provides a Shiva remote access client for Windows 3.x and DOS. These clients can be
used to make connections to the Windows NT Remote Access server or a third-party remote access solution, such as Shiva LANRover or 3COM access builder. This discussion uses NT's RAS for performance tuning examples.
Depending on the size and resources of your server hardware, the best performance will be realized if the RAS server is installed on the same physical server as Exchange. This setup gives you the option of limiting the RAS traffic to only that computer,
effectively increasing your dial-in security. Also, this reduces the effect that LAN bandwidth problems or error conditions will have on the users who are making remote connections. Because the actual connect speeds with typical compression ratios will not
exceed 33.6K, you should be extremely careful about what is transmitted over the modem link. Users can realize better performance by using the remote access features built into the client, allowing them to set up filters for what messages will be
downloaded during a session.
You should use high-speed modems combined with advanced serial port I/O port connections to keep the link speed as high as possible. Modems on the server should support at least v.34 and v.42/MNP compression and error control, with similar modems on the
client's workstations. Resist the temptation to use the on-board serial ports of the server, and make the investment in a high-speed serial port card such as a Digiboard with an EISA or PCI interface. As the number of modems on your RAS server increases,
you will need to use a high-speed serial port solution that will not use more processor resources. Many third-party solutions work with NT RAS to have a minimal impact on the server's valuable processor resources.
If you have made the switch to ISDN, your users will get much better connection speeds. NT RAS supports ISDN-type connections, and connection speeds can range from 64–128K, depending on the hardware and service you use. You will most probably have
to provide solutions for both ISDN and modems because ISDN is a good solution for someone working out of her home, but it is not practical or available for users with laptops who are constantly on the road. The NetBEUI protocol is the best choice for a
protocol if you decide to limit user access to only the Exchange Server, and TCP/IP is the only good solution for connections that need to connect across routers or bridges.
This section title, "Software Tuning Secrets," is a little misleading because we all know that every performance tuning option available in Exchange is published and readily available. Actually, my sarcasm has some merit, because the
performance options for Exchange are documented in many white papers and Microsoft technical articles. I will highlight the most common options for tuning performance of Exchange Server services, including the performance optimizer wizard, because it does
a very good job at tuning most of the registry parameters for the various services. These "secrets" are well documented by various Microsoft sources, and I will point out specific articles where applicable.
Most configuration parameters for tuning and performance of the services within the Exchange Server can be modified with the Exchange Performance Optimizer, also called the performance wizard. Even though many of these parameters are contained within
the Windows NT registry, it is best to access and modify them through the performance wizard interface. Any time you add or remove hardware from your Exchange Server or want to modify the parameters used to scale the server, you should immediately run the
performance wizard. The first thing you should do after installation is to add the -v option to the icon in the Exchange Server group from Program Manager. This option runs the wizard in verbose mode, allowing greater control over the parameters and values
that will be changed, as well as giving you the option to exclude drives from the disk testing process. When you start the performance wizard from the icon in the Exchange program group, you are first asked to confirm that the services will be stopped
during the process by clicking Next; then you see the screen shown in Figure 25.1.
FIGURE 25.1. Performance Optimizer wizard questions and answers screen.
The question concerning the range of users that this server will host should be set to the actual number of users, not the number of concurrent users. This will change the values in the registry for worker threads and maximum concurrent users. Any time
you use the wizard to change this value, make sure that you have adequate memory and disk resources to support those users, because the optimizer will notify you if there are inadequate resources to support the number of users you have selected.
The checkboxes for the types of services this server will provide help on determining the parameter changes that should be made on the MTA, connectors, and Information Stores. Descriptions of the effects of checking these options are as listed here:
The option to select a range of how many users are in your entire organization will control the parameters associated with the directory service. This will change parameters such as the directory cache buffers and number of threads for the directory
service to support directory replication and queries. Changing the range to a higher value will allocate more system resources to the directory service, so you might want to set this value higher than your actual number if you expect more directory lookup
and read requests than normal. The last option on this page enables you to limit the amount of memory the Exchange services will see. This option is helpful if you will have other applications such as SQL Server or SMS running at the same time, or if you
want to reserve some memory for the disk cache. Click Next to move to the screen for analyzing the hard disks, shown in Figure 25.2.
FIGURE 25.2. Performance Optimizer wizard disk analysis screen.
From this screen you can use your mouse or the spacebar to select or deselect the drives you want to test. The wizard then uses a series of reads and writes that emulates the I/O processes of the Exchange databases to determine which drives are the
fastest. This process is looking for which drive will be best for the transaction logs, which will be best for the databases, and which will be the best for the MTA and connectors. You then see a list of the statistics gathered from each drive. Click the
Next button to get to the screen shown in Figure 25.3 with the suggestions on which drives to move the Exchange database, log, MTA, and connector files to.
FIGURE 25.3. Performance Optimizer wizard suggested locations for Exchange components.
From this screen, you can override any of the suggested locations by changing the path manually, and you might have to carry out this action if you have designated certain drives for the components listed. At this point, I routinely run a full tape
backup of the drives, because the Exchange services are stopped and all associated files are closed. I had one system that lost power while moving the files, which forced me to restore everything from the previous night's backup. When you click the Next
button, you get a screen that warns you to back up the files, and it has a checkbox selected to move the files automatically. If you uncheck this box, you will need to manually move the components to their appropriate locations before the process is
finished. After the files are all moved, you can click the Next button to view the changes to system tuning and registry parameters on the screens shown in Figures 25.4., 25.5, 25.6, and 25.7.
FIGURE 25.4. Performance Optimizer wizard parameter recommendations screen #1.
FIGURE 25.5. Performance Optimizer wizard parameter recommendations screen #2.
FIGURE 25.6. Performance Optimizer wizard parameter recommendations screen #3.
FIGURE 25.7. Performance Optimizer wizard parameter recommendations screen #4.
You should review the values on these four screens, but unless specifically instructed, you should not change any of the values. They are calculated using the answers you gave in the previous steps, and are specific to the memory and disk speed of the
system. Instead of changing the values from these screens, modify the answers on the first page to have the wizard calculate values closer to what you want. After you have reviewed the four pages of proposed changes to system parameters by clicking the
Next button, you are shown the final screen of the performance wizard that saves the system parameters and restarts the Exchange services. This completes the procedure for using the performance tuning wizard that comes with Exchange Server.
The performance tuning wizard is good for scaling the parameters for services to use enough resources to support a specific number of clients, but other changes can affect overall system performance. These changes include, but are not limited to,
disabling circular logging for the transaction logs, disabling the loopback messaging for the IMC, and enabling or disabling message tracking.
The circular logging feature for the directory service and Information Store transaction logs is enabled by default. This feature prevents the buildup of transaction log files on the hard drive, but it removes the capability of performing differential
or incremental system backups. For systems that will be under a heavy load, this feature also gains some additional performance because the system does not have to take the time to reset the log files after a predetermined number of checkpoints. If you
disable circular logging, make sure that you run a backup program such as NTBACKUP to back up the system each night, resetting the transaction logs automatically. You can disable the circular logging feature from the advanced properties page of any server
object by un-checking the appropriate boxes and then allowing the system to stop and restart the DS and IS services. Please note that the users will not be able to access the server temporarily while these changes are being made, so you might want to
perform this operation after normal business hours.
The Exchange Internet Mail Connector (IMC) by default checks to see whether a message is being sent to SMTP addresses that reside on the same site as the IMC. It attempts to send those messages to the Internet, which causes the message to be returned to
the IMC, creating a loop condition. Because this process requires computing cycles, it is best to disable this process and let the message be delivered or not delivered from within Exchange. To disable loopback messaging, execute the Registry Editor
(regedt32.exe) and find the subkey
HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeIMC\Parameter
from Registry Editor. Then add the following value:
Value Name: DisableLoopbackConnections Data Type: REG_DWORD String: 1 (Default is 0)
When done, exit the Registry Editor and stop and restart the IMC to have the change take place. This affects only messages that are addressed to recipients with SMTP domain names that match the address space of the site. If other SMTP based systems will
receive messages for the same domain name configured in Exchange, create custom recipients for each of the recipients on those systems. This operates in a similar manner to an alias file for UNIX sendmail, and requires that SMTP host to have an
"A" or "MX" record in DNS.
The message tracking feature is available for the server's MTA, Information Stores, and connector components. The tracking logs enable administrators to use the Track Message feature in the administrator program to trace the route that specific messages
takes through the site or external connections. You must enable the Track Messages selection for any of these components to allow the message tracking feature to read the tracking file for each server. By enabling this feature, you create a small delay for
the time it takes to write the header information to the tracking logs, which might heavily affect performance on systems that are backbone or bridgehead servers. Just be aware of the small performance hit you take by enabling message tracking so that you
can disable it if you need to tune a server that is performing poorly.
Just after you finish your planning and design stages, and just before you move to a production exchange environment, you will want to take some time to develop and run a pilot project. This step enables you to confirm the functionality and design from
your planning stage while getting a limited number of users involved in shaping the system to meet your company's messaging needs. You should already have the server hardware in place before you move into a pilot environment so that it will be easier to
transition into production.
To create a controlled pilot for your Exchange Server messaging system, your project team should implement the following steps in this order:
The pilot project is the last chance to make any major modifications to the system without affecting a large number of users. Take the time during this process to check and double-check all system functions, and don't be afraid to reverse decisions made
during the planing process. Take the recommendations from the users involved in the pilot to your implementation meetings. Have the project team evaluate and vote on which problems will be corrected during the pilot, and which will have to wait until after
implementation. A proactive approach to handling the problems that come up during the pilot will help make the implementation of your new messaging system less of a fire-fighting endeavor and more of a successful product deployment.
As mentioned previously in step 10, you should use this opportunity to run the LoadSim utility included with Exchange Server to validate your system design. At first the LoadSim utility seems complex, but with a little patience and some realistic user
loading data, this process should be very useful in analyzing your system's performance. The utility creates a file that you need in order to use directory import to create the simulation accounts, and I suggest that you create them in a separate recipient
container for easier cleanup later. You will also define the servers that will participate in the simulation, defining the sites and number of users per server. After you have entered the parameters into the program, you will begin the load simulation to
run for a predetermined amount of time, usually 8–24 hours. Remember to have the NT performance monitor logging all the system objects during the simulation so that you can determine system load and response times. Also use the LsLog.exe utility to
merge, truncate, and generate the 95th percentile response times and weighted averages (scores) from the raw log files from each client simulation machine. Refer to the document that comes with LoadSim for guidelines on using and running a simulation in
your environment. See Figure 25.8 for an example of the LoadSim application interface.
FIGURE 25.8. The LoadSim application.
How do you measure the Exchange Server's performance throughout the process of tuning and testing? How will you keep an eye on the operation of the system during the pilot project and implementation phases? Exchange and Windows NT include system
utilities to monitor and collect data from the Exchange Server messaging environment. Which monitoring options you use will depend on what portion of the system you are attempting to get information on and what you are going to do with that information
after you have it.
The monitoring options for Exchange Servers can be broken into four areas:
The event viewer is very helpful in troubleshooting problems or tracking a process from beginning to end. All error and warning messages are written to the application event log, but you can use the diagnostic logging properties page for many of the
directory objects to increase the detail of logging per process. For example, if you were having problems delivering to hosts on the Internet, you would increase the level of diagnostic logging for the message transfer option to get more detailed
information on what the IMC is doing during message delivery. The event viewer is the first place to look.
The performance monitor is a helpful tool for charting, logging, reporting, and setting threshold alerts for any of the counters within the Exchange services. The Exchange Server installation process will create icons in the Exchange program group with
many useful performance monitor charts to track portions of the Exchange services. One chart monitors server health, and others chart things such as number of messages in the MTA queues, MTA delivery times, and IMC statistics. Normally, these performance
monitor charts are running on a machine dedicated to monitoring the Exchange messaging system.
An Exchange link monitor is a very useful tool that can be used to determine the speed and reliability of connections to external systems or other Exchange sites. Link monitors bounce messages off of a recipient to determine round-trip message transfer
time and link status. You must have an NT workstation or server logged onto the network running a link monitor from the administrator application for this feature to work. This can be the same machine running server monitors or performance monitors, or it
can be used during a troubleshooting process to keep tabs on a specific link.
The Exchange Server monitor is used to monitor specific services on an Exchange Server(s), and perform actions that the administrators define when those services stop running. This monitor also requires a dedicated NT workstation or server to be running
the administrator program for constant monitoring. When it is determined that a service is no longer running, the monitor can perform different actions on the first, second, and subsequent attempts. The actions it can perform are to take no action and just
alert, attempt to restart the service, and even reboot the server if specified by the administrator. Because the server monitor looks at the services running on a specific computer, it can be used to monitor non-Exchange services such as backup software or
core NT Server services.
Here are a few notes about link and server monitors for those of you who have very large enterprise messaging systems. Link monitors require no special connectivity between two sites to test the link. As long as a connector is defined and a route to the
mailbox is used to bounce messages, the link monitor will perform its duty. The server monitor does require that the machine running the administrator program with the server monitor must have synchronous RPC connectivity to work correctly. This means that
if you are in the United States and want to monitor a server in Asia, you need to be able to RPC-ping that server first. There are also practical limitations to the number of servers that can be monitored by a server monitor. Because this type of monitor
polls the servers on a routine interval, the more servers you add for it to monitor, the longer the polling period will take, until it begins to miss the window for servers at the end of the list. As a practical design, server monitors should be used for a
geographical region or on a per-site basis for better performance and manageability.
Planning for a messaging system that is scaleable and will deliver performance to meet current and future requirements can be a challenge. The hardware industry is constantly upgrading system options to meet the increased demands placed on servers by
new applications and business processes. Messaging systems are unique in their requirements for high I/O bandwidth and scaleable performance options that can expand to meet ever-increasing messaging demands. As companies move to client-server messaging
systems such as Exchange, they are truly realizing the challenges they face in system design and optimization. They must be able to proactively address new requirements as business processes are migrated from legacy systems into a client-server messaging
environment.
Designing a hardware solution to support the messaging requirements of the user community requires a careful analysis of the current system. It involves gathering and presenting current statistics on messaging traffic for use in planning sessions to
come up with a proposed system design. Your project team should have a good understanding of your company's core messaging requirements, with careful consideration of how network resources will be impacted by processes such as message transfer and data
replication. In an Exchange messaging system, site and routing topology should be tailored to have a minimal impact on the network infrastructure, and be flexible enough to handle network outages.
After you have a solid system design from your planning sessions, you must validate it to ensure that it will meet your company's messaging requirements. This task involves testing and loading the system with simulated and "real" users in the
form of a pilot project. Exchange Server includes utilities such as LoadSim and system monitors to determine how the system will operate under load in a production environment. The pilot project identifies areas that need modification, and enables the
users to become more involved in evaluating the overall system design to meet their needs. If you follow the recommendations and concepts presented in this chapter, your Exchange messaging system should move smoothly into your production environment to
serve your company for years to come.
The next chapter discusses how Exchange Server integrates with the Internet. Topics include using public folders for listservers and newsgroups, SMTP mail integration, and enhancements that will be available in future upgrades. The next version of Exchange will include many features targeted directly at integrating with Internet technologies. By far, the Internet is the hottest thing sweeping the computer industry, and it will continue to drive new business application development into the next century.
