Maintenance and Downtime

[bmn]

My apologies for the downtime between 8:50am and 3:05pm today (9 September '09).

The silver lining is that this one was totally unrelated to the server itself - in fact we were at 40 days uptime and still going strong. With the mess of organising the Summer of Sonic (and a few other things I had on my plate at the time), I managed to miss the scheduled billing date and we were suspended. Took me about 20 minutes to do everything necessary to get us back.

[bmn]

You may have experienced more downtime today. This was caused by a configuration problem that made two conflicting DNS services start on reboot - the one we don't use took priority and the first nameserver went offline.

What annoys me personally is that whenever the main nameserver is unavailable, that seems to automatically break DNS for the site, regardless of the fact that the backup nameserver is working perfectly fine and returning the correct response to anything that asks for it.

Edit: I believe I've found the cause. This particular problem shouldn't happen again.

[Biafra Republic]

Odd... I thought it 9 september was next week (today being 9/2 as I post...)

[bmn]

I always label dates as YMD. Thus, 09/09/01 = 1st September '09.

[bmn]

Between approximately 12:50pm and 1:20pm today (November 5th '09) the server apparently crashed and had to be restarted by on-site staff at our datacenter.

If you are experiencing any new problems on the server, please post them here.

[bmn]

At about 10:20pm yesterday (16 December 09) the server had to be restarted, causing about 10 minutes of downtime. SSR 24/7 was down from then until 5:20pm today.

[bmn]

Sometime before 4:50am Saturday SSR 24/7 went down. This was caused by a failure of the language parser for the app that decides what track to play when the previous one ends. After failed attempts to fix the issue without site downtime, the server was rebooted at 6:30am, and full service was restored by 6:35am.

[eXtaticus]

I was in a car at the time, so it didn't affect me; I guess I'm just lucky!

[bmn]

There was approximately 30 minutes of database downtime at around 1am this morning which affected most of the sites. This was caused during a software upgrade when two minor problems that are close to harmless on their own converged causing a bigger problem that couldn't be readily solved or reverted. Once the problem was fixed the upgrade went ahead successfully.

[Jetronic]

There was approximately 30 minutes of database downtime at around 1am this morning which affected most of the sites. This was caused during a software upgrade when two minor problems that are close to harmless on their own converged causing a bigger problem that couldn't be readily solved or reverted. Once the problem was fixed the upgrade went ahead successfully.

Another thing... I went to the front page just now, about 5 minutes ago, and it said "You are banned" in bold black letters at the top of the page when I went to look at some news.  What's up with that D= ?

[bmn]

At 17:10 GMT 8th, the server died causing about 20 minutes of downtime on all services.

[bmn]

1:40am BST 12th, 20 minutes of downtime on all services.

[bmn]

3:52pm BST 29th, 10 minutes of downtime on all services.

[bmn]

Around 5pm BST 25th. Approximately an hour of downtime on PHP only, causing downtime of parts of TSS, SSR site, SoS and SSN; all of SSMB and SoS registration. The cause was a major issue arising during a routine upgrade of the PHP software.

[bmn]

Today the 17th, at about 8:50am BST, about 20 minutes of downtime on all services due to a system reboot to correct an email delivery issue.

[bmn]

From 12:30am to 3:45am this morning all services were down. The cause was a corruption of the server's firewall configuration that closed all ports when the server was rebooted.

[eXtaticus]

The SSR IRC Chatroom went down approximately 15 minutes before this post; it progressively kicked all users - including ChanBot - in chunks of 5-10 every 15 or so seconds. The stream stayed up and continued working fine, and the DJ commented on the IRC "going crazy" - so I guess that it wasn't just me. It still isn't letting me in; it keeps saying "Unable to connect : java.net.ConnectException : Connection timed out: connect", though the stream is still okay...

[bmn]

The IRC server is hosted and run by another organisation (it's one of the few things we have that isn't hosted here), so I can't really help on that front.

[bmn]

About 30 minutes of downtime earlier on 13/8/11. Cause was heavy system slowdown from an unknown cause, possibly related to some work I was doing at the time.

[Heliumdere]

About 30 minutes of downtime earlier on 13/8/11. Cause was heavy system slowdown from an unknown cause, possibly related to some work I was doing at the time.

Edited July 13, 2011 by Heliumbunnette

[Dreadknux]

Didn't you know? Bmn can travel through time.

[Heliumdere]

Didn't you know? Bmn can travel through time.

He can? Can he tell us what Generations will be like? Or when the Sega Forums will be back up? O:

Edited July 13, 2011 by Heliumbunnette

[Pelly <3's U]

Ask away ye, any question of the future can be unveiled to you.

*asks sonic related question*

[bmn]

8/8/11 at about 1pm BST, the PHP daemon running on the server died or hung, breaking all PHP-powered pages for about 3 hours.

[bmn]

26/8/11 about 10am BST - a software issue stopped the server's OS being able to mount the main hard drive partition, causing at least 7 hours of downtime for all services.