Spots InterConnect: What's New? [What's New?]

What's New?

This page is where all our outage information, upgrade information, and other important stuff is listed. If you cannot get out from our network, or if something else just doesn't seem right, look here before getting too worried. We try to keep our equipment up as much as humanly possible, but sometimes things go wrong. Other times, there may be problems elsewhere on the Net that you may wrongly attribute to being a Spots problem. This page should set the record straight. Thanks for reading it!

October 26, 1997

I'm adding this a little in advance (the 23rd) so I don't forget to mention it here. On the 26th, the ATM link from iSTAR Toronto to iSTAR Vancouver will be down for "a minute" for upgrading. These one-minute outages have a habit of getting longer if things don't go too well. The upgrade is supposed to be sometime between 3am and 6am *Eastern* time. Nearly all our traffic leaves iSTAR's network over their Calgary to Seattle T3, so I don't think you'll notice any problems.

August 12, 1997

It seems that one of MCI's routers in Seattle is eating up packets. Traffic is barely trickling through it. Hopefully this will be fixed come morning, but if not, there's nothing we can do, as we're not MCI customers. Even if we were, I doubt we'd be able to make them work faster!

August 3, 1997

Once again, iSTAR's news server is not letting us connect to it. We won't know anything useful until Tuesday, and I'll be surprised if it's working before then. I don't know about you, but I'm getting tired of their piss-poor service.

July 23, 1997

I've put together an unattractive display of HTML to keep you all apprised of the 56k modem situation. You can get there from here. I'll try to keep it as up-to-date as I can.

July 19, 1997

Cheetah, our main server, has been upgraded from 80 megs of RAM to 128 megs. I think that's as much as it can handle, but I'm not sure. It's also got a shiny new Exabyte Eliant 820 tape drive. This is the "marketing droid" name for a new model of their 8505c drive, as far as I can tell, except that it likes the 160m tapes, which can hold 7 gigs instead of 5 gigs on the 112m tapes. Whatever! It works, and that's more than I can say for the old one.

June 20, 1997

Wow, a problem that we can't attribute to anyone's ineptitude other than our own *8-). Our router to the Internet went south last night (very early this morning, actually), and didn't get reset until this morning. Sorry about that!

May 29, 1997

For the last few days we've had our 28.8k ringing with no answer. AGT seems to have disconnected one of our lines from the middle of that rotary, and they're not being overly helpful about either fixing it or busying that circuit out until it can be repaired. We're going to be hounding them all day today, so hopefully we'll have the problem fixed by this evening. If you keep hitting the missing line, you can call in with your voice line (the circuit number is 262-7632, and you should get RING RING RING rather than a modem sound), and then call the regular 28.8k number with your modem. While the missing number is ringing, that circuit is busy, and the rotary works the way it should.

May 24, 1997

You've all received spam. It's a daily occurance, and I wish there was something I could do about it. Unfortunately, it's almost impossible to keep spam from hitting your mailboxes. Some of the more Internet-savvy among you may know what the 'leading' spammers are doing to prevent system administrators like me from blocking their junk-mail. For those of you who don't know, you may be shocked to find out the answer. Basically, they steal the services of an internet provider's mail server, using it as a relay to bounce their spam from, making it very difficult indeed to track. This act of theft causes the mail server to be over-stressed and crash. As if this wasn't bad enough, the mail headers are often forged to make it appear as if the email really was sent from the relay site, so the poor system admin guy gets flooded with hate mail, large file attachments, flood-pings, and SYN attacks. Not much fun at all.

I'm telling you this because this evening I am going to be installing some configuration patches for our mail server that will prevent it from being hi-jacked in the way I describe above. I hope that it will be a simple task, and that none of you will notice. However, I've been a system admin for long enough to know that that's rarely the case! I hope that, if you have email trouble, you'll read this page before calling our support line (which is closed on the weekend) and understand that any temporary interruption in mail service is for the ultimate good and well-being of our happy Spots network *8-).

May 15, 1997

iSTAR is having some problems again... Calgary seems to be isolated from the rest of the world. It's been up and down and up and down since noon. Hopefully a clueful person will be able to tell us what's going on before long, but I'll be surprised. Cluefull iSTAR people are in short supply today, it seems.

Newsflash!

At approximately 10:30h Pacific Daylight Time, a routing failure in the iSTAR network isolated customers in Calgary, Edmonton, Vancouver and Victoria from the rest of the Internet. The failure was isolated to a cross-connect switch located in our computing facility in Vancouver.

Dedicated access and dial access customers in Calgary and Edmonton were unable to connect with locations outside of iSTAR's network in western Canada from 11:30h- 17:30h (local time). Customers in Vancouver and Victoria were similarly isolated from 10:30h to 19:00h (local time).

The following is a synopsis of the trouble resolution process.

10:27h PDT: The failure was seen and logged in the iSTAR Network Management Centre.

11:34h PDT: iSTAR network management staff isolated the problem to an enigmatic routing problem relating to the core routers in our Vancouver computing facility.

11:43h PDT: At this point we escalated to Cisco and a team of engineers were brought in to work with our staff. The team quickly established a set of possible causes for the failure. After systematic testing eliminated all of these, The iSTAR/Cisco team used on-site staff to test network components one at a time to localize the failure. This involved reconfiguring the network around suspect devices, re-seating cards, reloading micro-code, etc.

15:50h PDT: The team tracked the problem to a cross-connect switch.

16:30h PDT: Having identified the problem device, we re-structured our routing architecture for Edmonton and Calgary to bypass the switch and bring these two cities back onto the Internet.

18:16h PDT: After more extensive testing, the team tracked the problem to a failing supervisory card in the cross-connect switch. We were then able to take advantage of redundancy in the computing facility to re-engineer around the faulty card, bringing Vancouver back on line at 18:56h.

The nature of the incident (a card that failed in a manner which degraded operation of the device and triggered failures elsewhere in the network, but which exhibited no direct symptoms and failed to raise any alarms) required our technical staff to isolate the problem device by device. While this prolonged the outage, we recognize that an outage of this duration is not acceptable to our clients. Here's what we'll be doing differently to avoid this type of problem in the future.

Having now identified the cross-connect switches as a potential single point of failure in our current network topology, we will eliminate them from the wide area portion of the network.

We will reconcile the routing architecture with the physical topology of the network to minimize the impact of a failure. This will be repeated after every significant change to the network.

We are firmly committed to providing our customers with high quality Internet services, and regret any inconveniences caused by this outage.

David Ralston,
Chief Operating Officer
iSTAR

April 6, 1997

Apparently Telus's voicemail service is offline for the day. Sorry for the inconvenience! If you have any questions, email is the best way to get ahold of us today.

January 30, 1997

Well, the intelligence of some people is overwhelming. Today at roughly 3:00pm, the City turned off our power. Why? They didn't think there was anyone in our office, and they couldn't figure out who was drawing all that power. Did they think to look? To call? To inquire at all? No, of course not. They thought the 70-amp electrical panel with the word "Spots" on it meant we were just running one HELL of a big spotlight. I'm serious!

Anyway, power was restored, and all the servers and modems came back up (save for one poor hard drive) within about 35 to 40 minutes after the 'incident'.

January 11, 1997

I was re-installing majordomo, the mailing-list software, and it made cheetah's load shoot up into the stratosphere. It took literally an hour for me to su to root to kill that process off and reboot the machine. I wasn't thinking, though, and as soon as the machine came back up the mail queue was processed, and majordomo went through the roof again. I know why, and it's fixed now, and I apologize for being such a pain in the ass to you all! All this *despite* having enough coffee...

December 24, 1996

Merry Christmas Eve! Istar blew up another router card in Calgary, so the Net was unreachable for a few hours. Apparently they got locked out of their office space while the security guards were enjoying their Christmas Cheer at home. Or something. They are investigating, and I'm sure won't let this happen again.

December 21, 1996

It's the holiday season, so most of us are going to be taking some vacation time to visit friends and family in far-away places. For this reason, please accept my advance apologies for the lapse in customer service that you may notice while Spots is running with nothing more than a skeleton crew. Happy Holidays to you all!

December 11, 1996

Well, I'll be. Cheetah locked up late this afternoon. I'm not sure what knocked it on its keester, but the symptoms indicate that the SCSI bus locked up. Of course, the SCSI subsystem will give that indication with either a filesystem or kernel panic, so more investigation is in order. It's been literally months and months since this happened last. July, you'll note, if you read down far enough.

November 19, 1996

"Going Around In Circles," or "How Best To Deal With AGT". Once again, they have buggered our phone system. If you call our support line tonight, and receive a "Thanks for calling the Telus Message Manager" recording, simply tap in 571-7768 on your touch-tone phone to leave us a message which will be promptly returned. If you don't have a touch-tone phone, you really should get one, since AGT is actually charging you extra money for your rotary-dial line. Sorry for the inconvenience.

November 17, 1996

The building that our office is located in needs to turn off the power for three hours starting 8am on the 17th. They say it's sure to be less time than that, but better safe than sorry. During this time, no Spots services will be available, and there will be no access to our site from anywhere.

November 9, 1996

We just received notification from Istar that InternetMCI needs an outage at their West Orange, New Jersey site, to replace a temporary fibre optic cable. The time in their notice says the outage will be from 07:00 to 10:00 "ZULU time" whatever that's supposed to mean. Either way, I don't think there will be a major impact for us here, since most of our traffic exits Istar's network via Seattle.

October 30, 1996

Wow, we finally received an official explanation for the outage on the 11th.

"The failure was due to 2 main processor cards failing. Due to the unlikelyhood of these cards failing, there were no maintenance spares on location in Calgary. The cards were flown from Ottawa and replaced on spec. All has been stable since. The process of replacing such cards is under review with hopes of creating a more efficient means of distibution.

iSTAR is sorry for the inconvenience."

October 11, 1996

It's been a while since my last problem-update, thank goodness! Last night at about 12:30am, IStar, our upstream provider, had one of their routers blow up. Unfortunately, their spare was also bad, so they had to get a new one flown out from Ottawa. For some reason, this whole process took 14 hours to resolve, most of which were during prime-time. If I get real answers, I'll post them here.

August 8, 1996

The hard drive problems are gone, thankfully, but they've been replaced with a very annoying ethernet problem that makes cheetah disappear off the network until it's rebooted. Until the problem is solved, I've whipped up a little script that will reboot the machine when it's unable to reach other hosts, so downtime should be quite minimal.

August 5, 1996

After an aborted attempt to replace the drive yesterday, that task was completed this evening. My apologies for bringing the system down semi-unannounced, but it was an emergency case. The filesystem problems SEEM to be gone now, but I'm presently knocking on my wood desk.

August 4, 1996

Due to me finally getting pissed off enough at the 1.6G EIDE drive, I will hopefully be pulling it out today. Expect some outages on cheetah, as well as missing/renamed files in your home directories while I compress things and make room for what's on the 1.6G EIDE drive. If I'm lucky, I'll have a new SCSI drive in there to replace it today, though buying that kind of thing on a Sunday is typically no fun for anyone involved.

July 28th, 1996

Looks like MCI has some Big Problems in Seattle. Nothing's leaving their FDDI network there right now. Long-term router problems on IStar's Toronto border router are not helping things, either, I don't suspect. Stay tuned for more information as it's available, if any.

Our problems with the 1.6G EIDE drive for /home/users and other tasks seem to be continuing. Your data's safe, though, as it's located on a 2.1G SCSI drive, but the links from /home/users to that location will disappear every time the EIDE drive decides to go away.

July 26th, 1996

I-star needs YET ANOTHER outage on the 26th from 4:00am to 7:00am our time. They apparently didn't finish what they needed to in Vancouver, and need to open another maintenance window to finish it up.

July 25th, 1996

I am going to attempt to move all the directories in /home/users to a new drive that has more space on it. This is a stopgap measure, until we get a really brand new drive in place. If you notice odd things happening, let us know!

July 23rd, 1996

I-star needs another outage to upgrade their Calgary network between 3:00am and 6:00am. Expect that you won't be able to get out from the Spots network during those times.

We had some trouble with the new drive we installed the other day, so some of you may have seen some trouble in the /tmp partition over the past short while. We've 'fixed' the problem, but there's no way to be sure it won't come back. We're monitoring the situation.

July 19th, 1996

Our internet provider, i-star, has announced an outage to perform an upgrade on their network in Vancouver. So, between 4:00am and 7:00am, expect connection problems to other sites.

June 28th, 1996

As per the email I sent everyone, the T1 install was postponed until the 27th of June. They tried to get it pushed back to JULY 25, but that was not acceptable.

This morning at 01:30, after spending five full hours on the phone with our internet provider, the new T1 went live. It's SPEEDY! Have fun with it; you all deserve it for being so patient.

June 16th, 1996

This isn't exactly new news, but on the 20th our fractional T1 is supposed to be upgraded to a full T1, meaning a 1200% increase in bandwidth, give or take. If AGT holds up their end of the bargain we'll be back up to a reasonable speed by Thursday.

June 1st, 1996

Our upstream provider has been cut off from the Internet due to an unscheduled outage on their fibre-optic connections. This will be fixed as soon as AGT can repair the fibre; hopefully some time late this afternoon or early this evening.

Update: The T1 was back in operation by mid-afternoon.

February 27th, 1996

Spots InterConnect now has a new head office!

We have moved to SUITE 400, 604 1st STREET S.W. CALGARY, ALBERTA T2P 1M7

January 21st, 1996

Spots is offering a list of Macintosh Internet resources in our Spots' Apple Orchard