There are two basic steps to optimize upstream bandwidth. First we have to find a way to prevent the ADSL modem from queuing packets since we have no control over how it handles the queue. In order to do this we will throttle the amount of data the router sends out eth0 to be slightly less than the total upstream bandwidth of the ADSL modem. This will result in the router having to queue packets that arrive from the Local Network faster than it is allowed to send them.
The second step is to set up priority queuing discipline on the router. We'll investigate a queue that can be configured to give priority to interactive traffic such as telnet and multiplayer games.
The final step is to configure the firewall to prioritize packets by using fwmark.
Although the connection between the router and the modem is at 10Mbit/s, the modem is only able to send data at 128kbit/s. Any data sent in excess of that rate will be queued at the modem. Thus, a ping packet sent from the router may go to the modem immediately, but may take a few seconds to actually get sent out to the Internet if the queue in the modem has any packets in it. Unfortunately most ADSL modems provide no mechanism to specify how packets are dequeued or how large the queue is, so our first objective is to move the place where the outbound packets are queued to somewhere where we have more control over the queue.
We'll do this by using a simple implementation of CBQ (class-based queuing) to limit the rate at which we send packets to the ADSL modem. Even though our upstream bandwidth may be 128kbit/s we'll have to limit the rate at which we send packets to be slightly below that. If we want to lower the latency we have to be SURE that not a single packet is ever queued at the modem. Through experimentation I have found that limiting the oubound traffic to about 90kbit/s with CBQ gives me almost 95% of the bandwidth I could achieve without CBQ. With CBQ enabled at this rate, we've prevented the ADSL modem from queuing packets.
At this point we still haven't realized any change in the performance. We've merely moved the FIFO queue from the ADSL modem to the router. In fact, with linux configured to a default queue size of 100 packets we've probably made our problem worse at this point! But not for long...
Unfortunately, the priority queuing discipline included with linux traffic control was not meant to be a queuing discipline of CBQ. Although it can be used as a leaf queuing discipline, there is no way to classify packets into bands. I solved this problem by changing the way the existing queue classifies packets. The new queue classifies packets into queues based on their fwmark. The lowest fwmark (0x00) is the highest priority queue. Higher fwmark'ed packets will be placed into lower priority queues. Packets fwmark'ed out of range will be placed as if their mark was 0x00 (an incentive to not incorrectly mark packets!) By using the modified queue (sch_fwprio.o) you can classify packets using ipchains (2.2.x) or netfilter (2.4.x) to set the fwmark. The new module must be installed prior to using the prio queuing discipline with tc. This prevents the old priority queue from being loaded by the kernel.
The final step in configuring your router to give priority to interactive traffic is to set up the firewall to define how traffic should be classified. This is done by setting the packet's fwmark field.
Without getting into too much detail, here is a simplified description of how outbound packets might be classified into a 4-band priority queue:
Mark ALL packets as 0x03. This places all packets, by default, into the lowest priority queue.
Mark ICMP packets as 0x00. We want ping to show the latency for the highest priority packets.
Mark all packets that have a destination port 1024 or less as 0x01. This gives priority to system services such as Telnet and SSH. FTP's control port will also fall into this range however FTP data transfer takes place on high ports and will remain in the 0x03 band.
Mark all packets that have a destination port of 25 (SMTP) as 0x03. If someone sends an email with a large attachment we don't want it to swamp interactive traffic.
Mark all packets that are going to a multiplayer game server as 0x02. This will give gamers low latency but will keep them from swamping out the the system applications that require low latency
Obviously, this can be customized to fit your needs.