Of Course It’s My Fault

This story is far enough in my past now, I can finally tell it.  To give some background, I was working at a university doing desktop support.  THAT ALONE probably fills a few of you with dread.  But to be specific, I was working for the nice folks in Dewey Hall.  I say “nice folks” because they had the decency to hire me and set aside some of their budget for my salary.  The deal was, I worked for them, but if my schedule allowed, I was to help out at the neighboring buildings, Screwem Hall and Howe Hall.  Those buildings contributed nothing to IT and basically got free support.

I get a call over to Screwem Hall, and a tearful lady tells me that her “backup drive” isn’t working.  Because this isn’t technically my building, I have no idea what she’s talking about.

It turns out that, years ago, before my time, before my co-worker’s time, before my boss’s time, in fact, before anyone I knew actually worked there, someone had set this lady up with a four-disc RAID-1 array.  A bad storm had come through during the night, and this morning… it no workeez.  Bear in mind, at this point, you readers know more than I did while this lady is breaking into hysterics about her years of lost work.

“It was supposed to be automatically backed up!  That’s the entire purpose of that thing!  And it’s supposed to do it automatically, I’ve never had to touch it!  YEARS OF WORK ON THERE!  IF IT’S LOST I’M GOING TO COLLAPSE!”

Again, this is not my territory and in all my time here, no one ever alerted me to this drive’s existence.  So, I sit down and try to assess the situation… and as you’d all agree, the ideal time to analyze a system is NOT after it’s already had a massive failure.

…Turns out, this is a four-bay enclosure, with four 1TB drives, configured into a 2TB drive with a 2TB redundancy drive.     Obviously, if a disk failed, the data could be reconstructed from the redundancy drive.

…Except it wasn’t one disc that had failed, it was two.  So her chances of success now hinge on the failed drives NOT having the same data.  I decided to look into this possibility.

…Except that the array’s management software hadn’t been loaded onto this computer, since the array had been there so long the user had replaced their computer in the meantime, and no one but the original tech even knew the management software existed.  So I decide to access the menu of the drive directly.

…Except it’s a painful cluster of menus, and I’m afraid to even turn the damn thing off and on, for fear of making any data corruption worse than it already is.  The only thing I CAN tell is that it’s set to automatically rebuild, so if the data is all still on there, it’ll go into auto-pilot mode.

So I tell her the short version: that her main chance of success is to hope this thing works as designed.  I tell her to go buy me two appropriate-sized drives and let me install them.  She does so, I install them, and I get nothing.  No data comes back.

RAID LADY LOSES HER FUCKING MIND.  

There are words.  There are noises.  There are screams.  There are tears.  And that’s just for starters.  She calls my boss, reports me, announces a vendetta against our whole department.  Without even blinking an eye, I calmly list the following:

I had no idea this device existed in our environment, therefore, I had never been able to do any preventative maintenance on it.

Whoever set it up set it up in a highly questionable manner.  First, they did not put it on a UPS, which made the electrical damage from the storm that much more likely.  That was stupid.

Second, they used double the number of hard drives needed to achieve the space needed.  Even years ago, 2TB hard drives weren’t that expensive, least of all if you’re charging them to an academic budget.  By doubling the number of disks, he doubled the potential points of failure.  Stupid.

Third, all the hard drives were the same model and from the same batch, indicating if one had a manufacturing flaw, the others would suffer the same flaw, multiplying the potential for failure.  Stupid.

Fourth, all the hard drives were original to the installation.  None had been replaced proactively.  So all were out of warranty and well past their expected time to fail.  Stupid.

Fifth, the software suite which MIGHT have alerted this lady to the fact that her drives were in bad shape OR that her backups weren’t being done was never installed.  GOD DAMN FUCKING STUPID.

…But no, it’s apparently MY FAULT her data is gone.  Because I couldn’t fix the magic drive that was supposed to never break.

The Help Desk is there for a reason

Not my story but the guy who sits next to me. He just got a caller who said he wasn’t able to get into his Outlook. He is like, ok, let me look at your account, and we shall try a few things. After going thru some basic stuff with his caller, he found out they didn’t know what their password was. No problem, we can set you up with a temporary one you can change. My coworker said he went into the system to see how long it had been since they last changed their password (it is a max of 90 days and usually tells us the date). It had been so long since he changed his password that the date had completely dropped off. So he asked the caller about it. Caller said he had problems with his password over 7 months ago and just stopped checking his password, but that his boss was irritated he wasn’t responding via email so told him to get it fixed.
How do you keep a job that requires you to communicate via email and for that matter uses the password in all the systems for over 7 months and just not even bother to tell anyone who can help you. Help Desk is there for a reason ya idjit!

Please let the door hit your ass on the way out… repeatedly

Found out today that effective this week, the FNG is no longer employed by our department.

The final straw began a few weeks ago. The FNG, quite literally out of nowhere, sent an email to the entire department, as well as $divisionChief’s boss, and their boss above them. He accused the acting supervisor and another staffmember of making racist remarks, as well as levelling an accusation of $divisionChief calling him an immature asshole, and “berating” him over the phone. Considering who the FNG CC’d on the email, the boss’ boss of $divisionChief ordered an investigation.

Almost immediately, the FNG’s own “evidence” came back to haunt him. The IM chat log the FNG pasted into the email… well, let’s just say there was nothing even remotely close to anything racist in the chat whatsoever on the part of the acting supervisor, the FNG simply blurted out that because they disagreed with him on some article he had read, then they (read: everyone else in the department) were all a bunch of racists, and launched into a tirade about how he was the smartest person in the entire department, how everyone else was beneath him, that we weere all jealous of his genius, and he shouldn’t have to follow all these rules $divisionChief and others were making him follow. You know, rules like standardizing our email signature, making sure we’re logged in on time, answering the phones, replying to emails in a professional manner, those sorts of things.

Quite a few people were interviewed as part of the investigation, and they had a tsunami of evidence against the FNG over the last 5 years concerning his behavior and general inability to do his job, including multiple cases of gross insubordination, deliberately ignoring rules, conflict of interest when he advertised his personal business in his email signature, refusing to do or not even knowing the basic aspects of his job, even AFTER he was told how to do it, and in one case, nearly causing the death of a patient. There were literally 5 years of chat logs, emails, reports of contact, formal reprimands, and otherdocuments that the supervisor and $divisionChief presented, showing that the FNG was, at best, completely incompetent. I and others presented further emails and chat logs, showing the FNG had no clue how to do his job, and when we tried to steer him in the right direction, he would frequently become difficult, if not outright beliggerent, and that the 18 certifications weren’t worth a crumped pile of soiled toilet paper. In addition, those interviewed showed that whenever he was on-shift, he frequently caused several of us to do more work because not only would we try to correct him, we would have to frequently go behind him in a ticket or email, and make sure he actually did that was asked of him. More than a few times, he’d disappear for hours on end, and not a soul could get a hold of him via email, IM or on the phone. When one of us finally did, the FNG would seem annoyed, as if we had interrupted him, and work was a distraction to whatever it was he was doing.

There was also some evidence brought to light that the FNG may not have been entirely honest about his work experience and more specifically, his time in the military. When he was asked about his time in the military, he got cagy, saying he spent 4 years as a medical corpsman/field medic, but couldn’t give basic details on even the simplest procedures performed by one when another staffmember (also a field medic) asked him. Even when presented with a video from a few years ago, in which the FNG was interviewed and himself stated that he washed out of Basic Training due to a broken bone in his foot, he refused to acknowledge the video’s existence. One of the people interviewed also told the investigators that he had previously worked with the FNG at a local facility, and that the FNG was frequently argumentative with supervisors and coworkers alike, would artificially inflate his numbers to make it look like he was doing more work than anyone else, would make unauthorized changes to systems, which sometimes caused them to crash, then deny ever making those changes despite evidence showing what he did and when. In one case, the FNG was asked to tape a note to doors on a medical wing notifying people that they were due for a computer update soon. Instead of creating just a single ticket to log the time spent, he put in a high-priority ticket for every…. single…. door that he put the note on, stating that he took several hours on each ticket. When his boss confronted him about that and told him not to inflate his numbers like that, the FNG became upset because he was already on thin ice for not working on enough tickets, only cherrypicking the easy ones. Oh, and the FNG’s coworker said, flat out, that when the FNG finally transferred out, the entire department went out to celebrate, and were only too happy to be rid of him.

All told, the investigation took only a couple weeks, which is blindingly fast for a gov’t investigation, and he was told that Friday was his last official day with us.

Microsoft says i have 128000 malware and i should call them….,

Got  a message from one of my customers. He says his computer shut down and then gave him a message saying he has 128000 pieces of malware and he needs toc all Microsoft. Now this user instead of calling me went and paid for it despite my always telling him not to fall for these things. I of course called hm back and tld him to power down the pc in hopes it gets them out! Now he is calling his credit card company to report the fraud. Lesson #1 always call your computer guy first!  SMH