Reasonable Expenses

My boss assigns me to support a new department, thinking they might need some extra attention.  I get a call from a really, really nice guy having some issues.  Turns out, their whole office has been neglected for ages, they have issues with network connectivity and, possibly some malware.  I can’t be sure without taking a look myself, and I can’t fix anything without logging in as an admin.  I’m very eager to help.

“Yeah, I’m sorry to hear you haven’t had support for a while.  Honestly, in order to even get started, I’d have to check out this computer, run some tests, and if you don’t have the admin login, I’m going to have to be physically at the computer so I can override it.”

“Oh, that sounds great!  There’s just one problem.”

“What’s that?”

“The computer is in Rome.”

“… You know, I really think this constitutes a need for a company-paid trip.”

Of Course It’s My Fault

This story is far enough in my past now, I can finally tell it.  To give some background, I was working at a university doing desktop support.  THAT ALONE probably fills a few of you with dread.  But to be specific, I was working for the nice folks in Dewey Hall.  I say “nice folks” because they had the decency to hire me and set aside some of their budget for my salary.  The deal was, I worked for them, but if my schedule allowed, I was to help out at the neighboring buildings, Screwem Hall and Howe Hall.  Those buildings contributed nothing to IT and basically got free support.

I get a call over to Screwem Hall, and a tearful lady tells me that her “backup drive” isn’t working.  Because this isn’t technically my building, I have no idea what she’s talking about.

It turns out that, years ago, before my time, before my co-worker’s time, before my boss’s time, in fact, before anyone I knew actually worked there, someone had set this lady up with a four-disc RAID-1 array.  A bad storm had come through during the night, and this morning… it no workeez.  Bear in mind, at this point, you readers know more than I did while this lady is breaking into hysterics about her years of lost work.

“It was supposed to be automatically backed up!  That’s the entire purpose of that thing!  And it’s supposed to do it automatically, I’ve never had to touch it!  YEARS OF WORK ON THERE!  IF IT’S LOST I’M GOING TO COLLAPSE!”

Again, this is not my territory and in all my time here, no one ever alerted me to this drive’s existence.  So, I sit down and try to assess the situation… and as you’d all agree, the ideal time to analyze a system is NOT after it’s already had a massive failure.

…Turns out, this is a four-bay enclosure, with four 1TB drives, configured into a 2TB drive with a 2TB redundancy drive.     Obviously, if a disk failed, the data could be reconstructed from the redundancy drive.

…Except it wasn’t one disc that had failed, it was two.  So her chances of success now hinge on the failed drives NOT having the same data.  I decided to look into this possibility.

…Except that the array’s management software hadn’t been loaded onto this computer, since the array had been there so long the user had replaced their computer in the meantime, and no one but the original tech even knew the management software existed.  So I decide to access the menu of the drive directly.

…Except it’s a painful cluster of menus, and I’m afraid to even turn the damn thing off and on, for fear of making any data corruption worse than it already is.  The only thing I CAN tell is that it’s set to automatically rebuild, so if the data is all still on there, it’ll go into auto-pilot mode.

So I tell her the short version: that her main chance of success is to hope this thing works as designed.  I tell her to go buy me two appropriate-sized drives and let me install them.  She does so, I install them, and I get nothing.  No data comes back.

RAID LADY LOSES HER FUCKING MIND.  

There are words.  There are noises.  There are screams.  There are tears.  And that’s just for starters.  She calls my boss, reports me, announces a vendetta against our whole department.  Without even blinking an eye, I calmly list the following:

I had no idea this device existed in our environment, therefore, I had never been able to do any preventative maintenance on it.

Whoever set it up set it up in a highly questionable manner.  First, they did not put it on a UPS, which made the electrical damage from the storm that much more likely.  That was stupid.

Second, they used double the number of hard drives needed to achieve the space needed.  Even years ago, 2TB hard drives weren’t that expensive, least of all if you’re charging them to an academic budget.  By doubling the number of disks, he doubled the potential points of failure.  Stupid.

Third, all the hard drives were the same model and from the same batch, indicating if one had a manufacturing flaw, the others would suffer the same flaw, multiplying the potential for failure.  Stupid.

Fourth, all the hard drives were original to the installation.  None had been replaced proactively.  So all were out of warranty and well past their expected time to fail.  Stupid.

Fifth, the software suite which MIGHT have alerted this lady to the fact that her drives were in bad shape OR that her backups weren’t being done was never installed.  GOD DAMN FUCKING STUPID.

…But no, it’s apparently MY FAULT her data is gone.  Because I couldn’t fix the magic drive that was supposed to never break.

Obey Wheaton’s Law!

As some may recall, a few months back, I pulled the trigger on beginning my own IT cybersecurity company, and it started off rather slowly, despite my best efforts in advertising and word of mouth. In August, I was invited to join a business networking group in my area that met once a week for a breakfast meeting/conference and to pass along buisness, and which only allows one member per business field, meaning in a group comprising major lawyers/law firms, doctors, accountants, real estate agents, and several local and national banks, I was the sole IT person there. They were only too happy to have me aboard, and with very good reason from what I discovered soon enough.

Slowly, I began hearing horror stories that the previous IT company representative they had, who was the owner and head tech of the company… let’s just say he was a certified, Grade-A jackass. He overcharged the other members every chance he got, told them to get equipment that was either unnecessary, a horribly bad and overpriced mismatch for their business and then either refused to set it up or would set it up so badly, they’d be forced to call him back at extra $350 hourly expense, gave vague and contradictory advice, his schpeals every meeting were little more than “Look at me, rah rah rah” sessions with very vague info on what he actually did, and in one case, flat out told one of the other members in the chapter who’s the owner of a prominent accounting firm in the State, with customers covering a good chunk of the Western Hemisphere, and whose soda budget is bigger than half the member’s salaries combined, that he wouldn’t provde tech support for her company any more because her company was now too small for him. Needless to say, when the annual membership committee met, they gave this guy Das Boot, letting the other chapters in the state know not to bring this guy in.

In just the month I’ve been with them so far, they’ve referred some good business my way, and that’s just the beginning. I already have three more major referrals from them, two with the possibility of being $6k-7k each for cybersecurity pen tests, a third from the vice president of the state chapter to do his company’s annual PC cleanup, and yesterday, the lady who’s the head of $accountingFirm was so impresed that I got rid of all the bloatware the previous IT guy had put on her computer, making her computer usable again, she wants me to be her go-to IT guy from here on out and even asked me to do some reasearch for her on a new SAN/NAS system with a virtual AD server so she can store the 45+ years of digitized client files such that her employees are able to better access them, since the previous IT guy refused to take the job. And she told me yesterday as I was finishing up, that she would do everything in her power to steer more IT business my way, especially from some of her prominent clients that have businesses of their own, or who are in need of cybersecurity as well as some good old-fashioned tech support.

Just goes to show that obeying Wheaton’s Law has it’s perks.