Spots InterConnect: What's New?
What's New?
This page is where all our outage information, upgrade information, and
other important stuff is listed. If you cannot get out from our network,
or if something else just doesn't seem right, look here before getting too
worried. We try to keep our equipment up as much as humanly possible,
but sometimes things go wrong. Other times, there may be problems elsewhere
on the Net that you may wrongly attribute to being a Spots problem. This
page should set the record straight. Thanks for reading it!
October 26, 1997
I'm adding this a little in advance (the 23rd) so I don't forget to mention
it here. On the 26th, the ATM link from iSTAR Toronto to iSTAR Vancouver
will be down for "a minute" for upgrading. These one-minute outages have
a habit of getting longer if things don't go too well. The upgrade is
supposed to be sometime between 3am and 6am *Eastern* time. Nearly all our
traffic leaves iSTAR's network over their Calgary to Seattle T3, so I
don't think you'll notice any problems.
August 12, 1997
It seems that one of MCI's routers in Seattle is eating up packets. Traffic
is barely trickling through it. Hopefully this will be fixed come morning,
but if not, there's nothing we can do, as we're not MCI customers. Even if
we were, I doubt we'd be able to make them work faster!
August 3, 1997
Once again, iSTAR's news server is not letting us connect to it. We won't
know anything useful until Tuesday, and I'll be surprised if it's working
before then. I don't know about you, but I'm getting tired of their piss-poor
service.
July 23, 1997
I've put together an unattractive display of HTML to keep you all apprised
of the 56k modem situation. You can get there from here.
I'll try to keep it as up-to-date as I can.
July 19, 1997
Cheetah, our main server, has been upgraded from 80 megs of RAM to 128 megs.
I think that's as much as it can handle, but I'm not sure. It's also got
a shiny new Exabyte Eliant 820 tape drive. This is the "marketing droid"
name for a new model of their 8505c drive, as far as I can tell, except
that it likes the 160m tapes, which can hold 7 gigs instead of 5 gigs on
the 112m tapes. Whatever! It works, and that's more than I can say for
the old one.
June 20, 1997
Wow, a problem that we can't attribute to anyone's ineptitude other than
our own *8-). Our router to the Internet went south last night (very
early this morning, actually), and didn't get reset until this morning.
Sorry about that!
May 29, 1997
For the last few days we've had our 28.8k ringing with no answer. AGT seems
to have disconnected one of our lines from the middle of that rotary, and
they're not being overly helpful about either fixing it or busying that
circuit out until it can be repaired. We're going to be hounding them all
day today, so hopefully we'll have the problem fixed by this evening. If
you keep hitting the missing line, you can call in with your voice line
(the circuit number is 262-7632, and you should get RING RING RING rather
than a modem sound), and then call the regular 28.8k number with your modem.
While the missing number is ringing, that circuit is busy, and the rotary
works the way it should.
May 24, 1997
You've all received spam. It's a daily occurance, and I wish there was
something I could do about it. Unfortunately, it's almost impossible
to keep spam from hitting your mailboxes. Some of the more Internet-savvy
among you may know what the 'leading' spammers are doing to
prevent system administrators like me from blocking their junk-mail.
For those of you who don't know, you may be shocked to find out the
answer. Basically, they steal the services of an internet provider's
mail server, using it as a relay to bounce their spam from, making it
very difficult indeed to track. This act of theft causes the mail server
to be over-stressed and crash. As if this wasn't bad enough, the
mail headers are often forged to make it appear as if the email really
was sent from the relay site, so the poor system admin guy gets
flooded with hate mail, large file attachments, flood-pings, and SYN
attacks. Not much fun at all.
I'm telling you this because this evening I am going to be installing
some configuration patches for our mail server that will prevent it
from being hi-jacked in the way I describe above. I hope that it will be
a simple task, and that none of you will notice. However, I've been a
system admin for long enough to know that that's rarely the case! I
hope that, if you have email trouble, you'll read this page before calling
our support line (which is closed on the weekend) and understand that any
temporary interruption in mail service is for the ultimate good and
well-being of our happy Spots network *8-).
May 15, 1997
iSTAR is having some problems again... Calgary seems to be
isolated from the rest of the world. It's been up and down and
up and down since noon. Hopefully a clueful person will be able
to tell us what's going on before long, but I'll be surprised.
Cluefull iSTAR people are in short supply today, it seems.
Newsflash!
At approximately 10:30h Pacific Daylight Time, a routing failure in the
iSTAR network isolated customers in Calgary, Edmonton, Vancouver and
Victoria from the rest of the Internet. The failure was isolated to a
cross-connect switch located in our computing facility in Vancouver.
Dedicated access and dial access customers in Calgary and Edmonton were
unable to connect with locations outside of iSTAR's network in western
Canada from 11:30h- 17:30h (local time). Customers in Vancouver and Victoria
were similarly isolated from 10:30h to 19:00h (local time).
The following is a synopsis of the trouble resolution process.
10:27h PDT: The failure was seen and logged in the iSTAR Network Management
Centre.
11:34h PDT: iSTAR network management staff isolated the problem to an
enigmatic routing problem relating to the core routers in our Vancouver
computing facility.
11:43h PDT: At this point we escalated to Cisco and a team of engineers
were brought in to work with our staff. The team quickly established a set
of possible causes for the failure. After systematic testing eliminated all
of these, The iSTAR/Cisco team used on-site staff to test network
components one at a time to localize the failure. This involved
reconfiguring the network around suspect devices, re-seating cards,
reloading micro-code, etc.
15:50h PDT: The team tracked the problem to a cross-connect switch.
16:30h PDT: Having identified the problem device, we re-structured our
routing architecture for Edmonton and Calgary to bypass the switch and bring
these two cities back onto the Internet.
18:16h PDT: After more extensive testing, the team tracked the problem to a
failing supervisory card in the cross-connect switch. We were then able to
take advantage of redundancy in the computing facility to re-engineer around
the faulty card, bringing Vancouver back on line at 18:56h.
The nature of the incident (a card that failed in a manner which degraded
operation of the device and triggered failures elsewhere in the network, but
which exhibited no direct symptoms and failed to raise any alarms) required
our technical staff to isolate the problem device by device. While this
prolonged the outage, we recognize that an outage of this duration is not
acceptable to our clients. Here's what we'll be doing differently to avoid
this type of problem in the future.
Having now identified the cross-connect switches as a potential single
point of failure in our current network topology, we will eliminate them
from the wide area portion of the network.
We will reconcile the routing architecture with the physical topology of
the network to minimize the impact of a failure. This will be repeated
after every significant change to the network.
We are firmly committed to providing our customers with high quality
Internet services, and regret any inconveniences caused by this outage.
David Ralston,
Chief Operating Officer
iSTAR
April 6, 1997
Apparently Telus's voicemail service is offline for the day.
Sorry for the inconvenience! If you have any questions, email
is the best way to get ahold of us today.
January 30, 1997
Well, the intelligence of some people is overwhelming.
Today at roughly 3:00pm, the City turned off our power.
Why? They didn't think there was anyone in our office,
and they couldn't figure out who was drawing all that
power. Did they think to look? To call? To inquire
at all? No, of course not. They thought the 70-amp
electrical panel with the word "Spots" on it meant
we were just running one HELL of a big spotlight.
I'm serious!
Anyway, power was restored, and all the servers and
modems came back up (save for one poor hard drive)
within about 35 to 40 minutes after the 'incident'.
January 11, 1997
I was re-installing majordomo, the mailing-list software, and it made
cheetah's load shoot up into the stratosphere. It took literally an hour
for me to su to root to kill that process off and reboot the machine. I
wasn't thinking, though, and as soon as the machine came back up the
mail queue was processed, and majordomo went through the roof again.
I know why, and it's fixed now, and I apologize for being such a pain
in the ass to you all! All this *despite* having enough coffee...
December 24, 1996
Merry Christmas Eve! Istar blew up another router card in Calgary, so the
Net was unreachable for a few hours. Apparently they got locked out of
their office space while the security guards were enjoying their Christmas
Cheer at home. Or something. They are investigating, and I'm sure won't
let this happen again.
December 21, 1996
It's the holiday season, so most of us are going to be taking some vacation
time to visit friends and family in far-away places. For this reason, please
accept my advance apologies for the lapse in customer service that you may
notice while Spots is running with nothing more than a skeleton crew. Happy
Holidays to you all!
December 11, 1996
Well, I'll be. Cheetah locked up late this afternoon. I'm not sure what
knocked it on its keester, but the symptoms indicate that the SCSI bus
locked up. Of course, the SCSI subsystem will give that indication with
either a filesystem or kernel panic, so more investigation is in order.
It's been literally months and months since this happened last. July,
you'll note, if you read down far enough.
November 19, 1996
"Going Around In Circles," or "How Best To Deal With AGT". Once again, they
have buggered our phone system. If you call our support line tonight, and
receive a "Thanks for calling the Telus Message Manager" recording, simply
tap in 571-7768 on your touch-tone phone to leave us a message which will
be promptly returned. If you don't have a touch-tone phone, you really
should get one, since AGT is actually charging you extra money for your
rotary-dial line. Sorry for the inconvenience.
November 17, 1996
The building that our office is located in needs to turn off the power for
three hours starting 8am on the 17th. They say it's sure to be less time
than that, but better safe than sorry. During this time, no Spots services
will be available, and there will be no access to our site from anywhere.
November 9, 1996
We just received notification from Istar that InternetMCI needs an
outage at their West Orange, New Jersey site, to replace a temporary
fibre optic cable. The time in their notice says the outage will be
from 07:00 to 10:00 "ZULU time" whatever that's supposed to mean.
Either way, I don't think there will be a major impact for us here,
since most of our traffic exits Istar's network via Seattle.
October 30, 1996
Wow, we finally received an official explanation for the outage on
the 11th.
"The failure was due to 2 main processor cards failing. Due to the
unlikelyhood of these cards failing, there were no maintenance spares on
location in Calgary. The cards were flown from Ottawa and replaced on spec.
All has been stable since. The process of replacing such cards is under review
with hopes of creating a more efficient means of distibution.
iSTAR is sorry for the inconvenience."
October 11, 1996
It's been a while since my last problem-update, thank goodness! Last
night at about 12:30am, IStar, our upstream provider, had one of their
routers blow up. Unfortunately, their spare was also bad, so they had
to get a new one flown out from Ottawa. For some reason, this whole
process took 14 hours to resolve, most of which were during prime-time.
If I get real answers, I'll post them here.
August 8, 1996
The hard drive problems are gone, thankfully, but they've been replaced
with a very annoying ethernet problem that makes cheetah disappear off
the network until it's rebooted. Until the problem is solved, I've
whipped up a little script that will reboot the machine when it's unable
to reach other hosts, so downtime should be quite minimal.
August 5, 1996
After an aborted attempt to replace the drive yesterday, that task was
completed this evening. My apologies for bringing the system down
semi-unannounced, but it was an emergency case. The filesystem problems
SEEM to be gone now, but I'm presently knocking on my wood desk.
August 4, 1996
Due to me finally getting pissed off enough at the 1.6G EIDE drive, I will
hopefully be pulling it out today. Expect some outages on cheetah, as
well as missing/renamed files in your home directories while I compress
things and make room for what's on the 1.6G EIDE drive. If I'm lucky, I'll
have a new SCSI drive in there to replace it today, though buying that kind
of thing on a Sunday is typically no fun for anyone involved.
July 28th, 1996
Looks like MCI has some Big Problems in Seattle. Nothing's leaving their FDDI
network there right now. Long-term router problems on IStar's Toronto
border router are not helping things, either, I don't suspect. Stay tuned
for more information as it's available, if any.
Our problems with the 1.6G EIDE drive for /home/users and other tasks seem
to be continuing. Your data's safe, though, as it's located on a 2.1G SCSI
drive, but the links from /home/users to that location will disappear every
time the EIDE drive decides to go away.
July 26th, 1996
I-star needs YET ANOTHER outage on the 26th from 4:00am to 7:00am our time.
They apparently didn't finish what they needed to in Vancouver, and need to
open another maintenance window to finish it up.
July 25th, 1996
I am going to attempt to move all the directories in /home/users to a new
drive that has more space on it. This is a stopgap measure, until we get
a really brand new drive in place. If you notice odd things happening, let
us know!
July 23rd, 1996
I-star needs another outage to upgrade their Calgary network between 3:00am
and 6:00am. Expect that you won't be able to get out from the Spots network
during those times.
We had some trouble with the new drive we installed the other day, so some
of you may have seen some trouble in the /tmp partition over the past short
while. We've 'fixed' the problem, but there's no way to be sure it won't
come back. We're monitoring the situation.
July 19th, 1996
Our internet provider, i-star, has announced an outage to perform an
upgrade on their network in Vancouver. So, between 4:00am and 7:00am,
expect connection problems to other sites.
June 28th, 1996
As per the email I sent everyone, the T1 install was postponed until
the 27th of June. They tried to get it pushed back to JULY 25, but
that was not acceptable.
This morning at 01:30, after spending five full hours on the phone
with our internet provider, the new T1 went live. It's SPEEDY!
Have fun with it; you all deserve it for being so patient.
June 16th, 1996
This isn't exactly new news, but on the 20th our fractional T1 is
supposed to be upgraded to a full T1, meaning a 1200% increase in
bandwidth, give or take. If AGT holds up their end of the bargain
we'll be back up to a reasonable speed by Thursday.
June 1st, 1996
Our upstream provider has been cut off from the Internet due to an
unscheduled outage on their fibre-optic connections. This will be
fixed as soon as AGT can repair the fibre; hopefully some time late
this afternoon or early this evening.
Update: The T1 was back in operation by mid-afternoon.
February 27th, 1996
Spots InterConnect now has a new head office!
We have moved to SUITE 400, 604 1st STREET S.W. CALGARY, ALBERTA T2P 1M7
January 21st, 1996
Spots is offering a list of Macintosh Internet resources in our
Spots' Apple Orchard
[ HOME |
SPOTS |
MEMBERS |
WEB PRESENCE |
TEXT |
WHAT'S NEW? |
HELP! ]
Copyright © 1995, 1996 Spots InterConnet, Inc.