STATIC DISCLAIMER: All the stuff in here is purely my opinions, and they tend to change depending on what mood I'm in. If you're going to get bitter if I say something about you that you don't like, then maybe don't read. I avoid using names as much as possible, and would request that people who know me do the same in their comments. Basically, I often vent my frustrations on here, so if you happen to be someone who frustrates me, expect to read a description of someone very much like you in here!

Monday, January 23, 2006

Just when you thought it was safe to go back in the free-media pool...


{gasp, pant, pant, choke, gurgle, splutter, wheeze, inhale...}


This is the sound of me, today, as once again our server had a conniption. To be fair, this time it wasn't anybody's fault other then perhaps the manufacturer of our RAID controller, who should never have put external SCSI ports on the back of the thing if they weren't intended to be used. Meanwhile, I have a cactused Exchange organization once again. I've heard of this magical thing called "brick-level backups" with Exchange, whereby you backup each individual mailbox separately. This would be a stupid waste of time, except for the fact that it's the only way to actually ensure that no server disaster is too great to recover from. Bloody Exchange. Bloody Microsoft. Bloody stupid fricking tape library that is supposed to make life easier but instead has made me stay back at work until stupid hours yet again.

In a strange turn of events, it turns out that I am Exchange's biatch.

PS: The only plus to all of this broohaha is that my boss shouts us pizza once again. Mmmmm... pizza... there's nothing like cheezy lard in a disaster recovery situation. I love pizza far too much.

Thursday, January 19, 2006

A short follow-on...

As of this afternoon our Exchange server is back up and running, sans all data created prior to today. Databases will be going to at least one data recovery company, and my final 5 days of work at St Paul's begins tomorrow. Life is peachy.

Wednesday, January 18, 2006

Work looks different at 3am

It all started at about 10am on Tuesday morning. I was sitting at my desk, which is in close proximity to the server cabinet, basking in the success of our recent domain rename operation. Meanwhile, my colleague was messing around at the rear of the stack of humming, very live, servers.
Suddenly, there was a beep.
Not just a short beep, but a continuous "something is very wrong" beep. For a second, I thought it must have just been one of the various desktops/laptops lying around, but I soon realised it was coming from the servers.
"What's that? Is that the file server?" I asked, noticing my colleague's slightly bewildered expression.
I quickly switch our server console over to the machine in question. Blue Screen Of Death. Not good. It was then I realised what had happened - My colleague had just plugged the SCSI cable from our new tape library into the RAID controller on our file server while it was all live. This was going to be a bad day...
My first reaction was the most obvious one: unplug tape library, reboot server. So that's what I do. During the boot up process, I see a line that reads something like this:
Array #1:BIGDATA - Failed

Following that, the server tries to boot, but with no success. Why? Because there's no disk to boot off anymore. It's gone. I load up the array diagnostics, but it tells me what I already suspected. Our array of 10 drives is spread over 2 channels; 5 drives on each. When my colleague had plugged in the tape library, 5 of those drives had simultaneously gone offline, and so the array had immediately become cactus. See, in a 10 disk array, you can lose up to 3 disks at once and still have it continue to work, but if you lose more then that the array fails, and you have to start from scratch.
Now, I didn't want to believe that this had happened at first, so I stuffed around with various utilities and progressed to firmware updates only to come back to what I knew to begin with: 5 drives fail = dead array. The array data was irreconcilably inconsistent, and the controller continued to insist that there was no possible way that sucka was coming back to life.
It's at this point that I discover that my boss is useful for something I'd never though of before: damage control. I quickly set him up to play diplomat so that I can get on with doing the work we're actually employed to do. Meanwhile, I shoot my colleague a couple of pointed comments just to hammer home that this is his fault, and I'm not happy. Bad, I know, but I was cranky. I did apologise later...
So we move on to trying to rebuild. We have the technology, we can rebuild it. However, during the process of our recent domain rename, we'd moved every possible essential Active Directory function on to the server that was now sitting mute in front of me. Crap. This could be a problem. So I ring a friend of mine who is wise in the ways of AD, and after some puzzled musing over why on earth we don't have a backup of our AD database (I SO should have been in charge of backups), he gives me the following solution: Make with the global catalogue, seize the FSMO roles, and pray for a replica.
So I jump onto one of our other DCs and do just that. Luckily for us, there was an up-to-date replica of our policy data on the server, so the GC was able to rebuild itself and for the most part our Active Directory was good from there on out. However, there was still the issue of the dead server...
So I start on a rebuild. Building a server doesn't normally take that long, but having to deal with a panicking colleague in the background can string it out a bit. Couple that with the fact that the server's driver CD decided it should downgrade the baseboard management controller firmware and cause all kinds of trouble, and it took me about 2 hours to get it up to being a fairly bare system with DNS and a DC role. From here, we moved onto the backups...
I don't know if you've ever done much with computers, but when you start talking about shifting 200Gb files around, time seems to come pretty much to a standstill. See, our backup strategy has consisted of three 500Gb firewire hard disks that are cycled on a triweekly basis. So we've got this 200Gb backup file sitting on this firewire hard disk. But here's the catch - apparently, the disks have been failing of late. First I'd heard of it. So although there's a backup file there now, it may disappear if we start restore operations straight off the disk. Well, we can't have that happening, so we better make a copy before we do anything else. At about 4pm, we started the copy process. I'm not 100% sure why, but it finished at 9pm. 5 hours of waiting. During this time, I found out that Winamp has a selection of video feeds you can watch, and there's a feed that plays Futurama 24/7. Nice.
Anyway, after several coffees, a pizza and some Coke, the copy is finished and we start the restore process. People's home directories come back with full permissions and ownership intact. It's a beautiful, beautiful thing. However, this short-lived goodness was not to last. The restore finished about 12:30am, leaving just one thing to do - rebuild Exchange.
Now at this point I'd officially decided that when I got Exchange restored, I was going to go home and not come to work again for at least 24 hours. However, the first part of that preposition turned out to be the problem. Exchange wouldn't restore. It just wouldn't. I tried over and over to get the mailbox stores to mount, but finally at 3am I gave up and headed home.
Midday the next day and I'm back at my console trying once again to make Exchange play nice with all the other friendly services. Over night, Active directory has kept itself replicated and playing nicely without throwing any errors at all other then some bollocks about having duplicate SQL instances registered for the same server, but I'll deal with that later. Meanwhile, my well-meaning boss has organised to pay stupid quantities of money to have a senior tech from Commander come out and have a look. On the phone, this guy spoke to me like I was an idiot, so I wasn't looking forward to the experience. When he arrived, he continued with this level of superiority while he ran through everything I had just done. Three hours, and at least a score of Google searches later, he told my boss that "...Justin has a fairly firm grasp of the issues involved, and maybe talking to Microsoft directly would be our best bet from here." Thanks very much, here's the bill, see you later.
By this time, I decide it's time for me to suggest a course of action. So I do. We'll hive off the old databases, rebuild the information store from scratch, give everyone new mailboxes, and get the system back up and running so we don't have another day of messages disappearing into the ether. We'll then look into extracting mailboxes from the offline database file to PST files (which must be done fairly easily from the number of companies offering the service) so that those PST files can then be merged into the new information store. Brilliant. How come I'm the only one who thinks of these ideas? Oh, and just as a backup, I tell Stuart that tomorrow he should spend the day extracting the cached copies of mailboxes from the Executive's laptops so that we can get their emails back even quicker. Everyone nods and accepts the fact that I am t3h l33t. Exchange is my biatch, I tell you.
Meanwhile the end result of this is that I'll probably leave St Paul's with people singing my praises while scowling menacingly at the other IT members. I like that. It works for me. The other result is that I'm crazy tired, and yet I'm up at 1am tonight just becuase my body thinks it's a good idea. w00t for coffee is all I can say.

Alright. Time for bed.

Sunday, January 15, 2006

Life Update

This program will update "Justin's Life" from version to

  • House
    So we've been trying to buy a house. I've posted on this a few times, but at the moment we've made an offer on a duplex in Bligh Park and we're waiting to hear back. We were pretty pleased with the real estate agent too, which was unusual. However, I think she's a bit worried she might not close the deal, and has tried to divert our interest onto another smaller property. Not interested. Stupid real estates. :P Still, hopefully I can post about us buying this place shortly. We should hear next week. If you are of the praying persuasion, please pray!
  • Work
    So I'm into my last couple of weeks of work. It's pretty weird changing jobs like this. This is my first fulltime position, so change is a little more difficult then I thought it would be to pack up and move on. Still, two weeks and it'll be done. I've tried really hard to leave the door open for me to come back if they can offer me a better position in the future. It's a nice place to work. It was just my situation in the organisation that was problematic for me.
  • Church
    Well, although we piked out this morning, we have been going to Hawkesbury Valley Baptist Church and really, really enjoying it. It's a great church. We're about to get involved in some stuff there like kids ministry and the music, which will be quite refreshing. For me, I'm going to love having the opportunity to drum again. It's been quite a while, and playing at church is a great way to serve. I like it I like it a lot. :)
  • Me
    Someone told me once that by about 30 you've sorted out your life, and are a bit more comfortable in who you are and what your priorities are. Well, although I've got a few years to go to 30, I think the beginning of this year is the start of that for me. I've started thinking about what I need to do to achieve the things I'd like to. I was a bit disheartened when I found some goals I'd written in 2002 that I've completely failed to achieve, so I'm hoping I can actually start putting some honest effort into making things happen.

Anyway... just some updates.

Oh - and has anyone noticed that songs in 6/8 really rock? I never thought about it before...

Friday, January 06, 2006

Song for today

Album : "Going Somewhere" (written by: Colin Hay / Thom Mooney)
Colin Hay

Any minute now, my ship is coming in
I'll keep checking the horizon
I'll stand on the bow, feel the waves come crashing
Come crashing down down down, on me

And you say, be still my love
Open up your heart
Let the light shine in
But don't you understand
I already have a plan
I'm waiting for my real life to begin

When I awoke today, suddenly nothing happened
But in my dreams, I slew the dragon
And down this beaten path, and up this cobbled lane
I'm walking in my old footsteps, once again

And you say, just be here now
Forget about the past, your mask is wearing thin
Let me throw one more dice
I know that I can win
I'm waiting for my real life to begin

Any minute now, my ship is coming in
I’ll keep checking the horizon
And I'll check my machine, there's sure to be that call
It's gonna happen soon, soon, soon
It's just that times are lean

And you say, be still my love
Open up your heart, let the light shine in
Don't you understand I already have a plan
I'm waiting for my real life to begin

Click here to listen to this song.

Thursday, January 05, 2006

Justin's QOTD

If the automobile had followed the same development cycle as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year, killing everyone inside. - Robert X. Cringely

So this is my 100th post. It came and went without me even noticing. Well, it didn't "went" exactly, as I noticed as I was about to post my 101st. Anyway - w00t for my blog!

G14NT PH34RB0T 0f D00M!!!

One of the down-sides to moving on from St Paul's is that there are two things that have only just happened that I've been trying to get to happen for the past two years and I'm going to miss out on them. One is a new office for us that doesn't physically contain the server and comms cabinets also, and the other is a new server cabinet that will actually fit our servers in. Well, I got it, but I'm going to miss out on the w00ty goodness of working with it. However, it does have our servers in it now. Here's a pic:

You wanna know about these servers?? I'll tell you anyway. I know there's a bunch of you who couldn't care less, but this is my blog and you can just choose to not read today if you like.

Starting from the top down (all have Gig ethernet - the goofy Biblical names weren't my idea, btw):

Genuine IBM xSeries 345 - Xeon 2.66Ghz, 1.5Gb RAM, 3x36Gb hotswap SCSI HDD.
This server handles our printing/faxing services, including accounting and our print/copy charging service. It's also a domain controller.

HP DL380 G3. 2x Xeon 2.8Ghz, 3.5Gb RAM, 3x36Gb hotswap SCSI HDD.
This server is our external DNS, main web server and internet gateway/proxy with SurfControl for content filtering. It also currently runs a SMTP relay, but that will change shortly, as will it's RAM. We're going to rip 2Gb out of it for our new database server.

SOLOMON (soon to be JONAH):
HP DL380 G3. 2x Xeon 2.8Ghz, 3.5Gb RAM, 3x36Gb & 3x172Gb hotswap HDD.
This server is a packhorse. It runs - AD, DHCP, DNS, WSUS, SQL, Filemaker server, Sophos Enterprise Antivirus. This was our first server, so it's a bit chunky service-wise. However, we've just bought a new server to replace it. This server will have some of it's hardware removed (RAM, hard disks), and be retasked as a mail gateway and federation proxy for LCS2005. Seeing it will be the beefiest mail gateway ever, it'll probably end up doing other stuff too.

HP DL380 G4. 2x Xeon 3.2Ghz, 3Gb of RAM (soon to be 6Gb). 3x 72Gb HDD (soon to be + 3x 172Gb HDD) - battery backed-up RAID, redundant everything.
This is our "new Solomon". We have 27 SQL databases, serving at least 5 high-use applications and a Filemaker database solution that is in constant use. This is the first server that I've spec'd myself (my boss did the others - which is why our internet gateway is a dual Xeon), and I've made sure that everything except for storage is well covered. Why not storage? Because we need a SAN. As you'll see in a moment...

HP DL380 G3. 2x Xeon 2.8Ghz, 2.0Gb RAM, 3x36Gb hotswap SCSI HDD.
Sharepoint Portal Server of w00t. Also runs our PHP/MySQL eLearning solution, Moodle.

Whitebox Intel. 2x Xeon 2.8Ghz, 3.5Gb RAM, 10x 72Gb hotswap SCSI HDD. Redundant everything.
This is our primary fileserver and Exchange mailbox store. It also is a domain controller, and runs DNS, SurfControl for Email, Backup Exec and a handful of lightweight internal web applications.

Whitebox Intel. Xeon 3.0Ghz, 1Gb RAM, 1.6Tb HDD.
ClickView server for video resource distrobution. Also runs Live Communications Server 2005 Standard Edition.

Whitebox something. Xeon 2.8Ghz, 1Gb RAM, 1.6Tb HDD.
Security server - aggrogates video feeds from security cameras and records to hard disk. We have 8 cameras, and store about a month of footage based on motion detection.

This rack also has a Firebox X1000 firewall, and a Quantum PX502 tape library in it - neither of which appear in the photo.