Typical calls…

Since moving to my new team, here’s a short smattering of the kinds of calls I’ve been getting…

“Hi, I don’t know if I got the right group but… (proceeds with the Lord of the Rings in length version of her password problem)”

“Hi, I have a major incident, I can’t log into $program all of a sudden”

“I need a new monitor for my computer”

“Hi I need to reset my password”

Bear in mind, ours is the very last option on the call tree and very clearly states that we only handle major incidents affecting multiple sites, states, regions or the whole country, and not problems with individual computers or user issues. People up the food chain have sworn up and down  since March that they will remove our number from the help desk call tree, but have yet to do so, and on a hunch, I called in a few times to one of the other options and the shortest wait time I was quoted was 90 minutes. And when I tell these poor souls they have to call somewhere else, I’m usually met with a mix of indignation, resentment, resignation, requests to be transferred (which we can’t do since our phones have no transfer button), and outright anger since they confuse our inability to help them with an unwillingness to help them. In fact, some local IT shops specifically tell the users to call us directly and avoid the other options entirely, even though we have no way to help them due to our lack of access to many systems.

And yet, between this, the new management that has little to no clue what their doing, the utter lack of direction, being pulled in a half dozen different directions to do things I was never trained to do with programs I didn’t even know existed, the complete absence of anything resembling SOP’s, call lists, or any other written policies on how to perform this job (as well as many other things I haven’t mentioned), it’s no wonder most of the people in my group are actively looking for outside positions, dusting off their retirement papers, or otherwise loooking elsewhere for gainful employment.

The months of Hell

The last few months have been nothing short of baseball-bat-to-the-testicles hell at my day job.

Reason being, someone higherup the food chain than our division chief got it into theit head, without notifying us, clearing it with us, or even bothering to do much of anything else, that our entire department should be transferred from our current duties, and begin working major incidents. This all was because the contract for the people who were doin it previously wasn’t being renewed and they needed someone to take it over immediately, and hadn’t planned on anyone actually taking it over.

So you can imagine the chaos that took place. Literally, we all came in and discovered that the links to our normal dashboads were working with “Access Denied” error messages, we weren’t getting any of our normal email traffic, we couldn’t even log into our phones to contact each other. Our division chief gathered us together to let us know of the change the day after it happened, since no one ever bothered to even tell us (even our division chief until 9PM on the day it happened). So we were dead in the water as a department for a whole day before anyone even told us what happened.

And here’s the best part… the major incidents group was, and still is, a section of the Tier 1 Desk Monkey contractors that remained and were able to renew their contractors. So in essence, all of us who are full time employees working for the Uncle are essentially answering to contractors, ones who have proven time and time again that they have to rise multiple levels before we can consider them utterly incompetent. Currently, my direct supervisor is a contractor that’s been detailed to my shift, and he’s instituted policies which have been roundly groaned at. For example, every single call we get, we have to put a ticket in for it. The problem is, 99% of the calls we get, we have to tell the poor souls to call back into the Help Desk because they picked the wrong option on the call tree, all because these people want their passwords reset, they want a new mouse, their $hardware is broken, they can’t log in, and our team has no access to their accounts or to anything else at the local level.  The only reason they call us is because we’re the only people who pick up the phone in a quasi-reasonable amount of time, because local IT told them to, or because they think a password reset really is a Major Incident. But still, we have to put a ticket in for all of them, even if we just tell them to call the help desk back to pick Option 1, and not us. In fact, in the three months we’ve been doing this, call times to get passwords reset have gone DOWN to just over an hour. All because of the chaos that resulted from this whole contract kerfuffle.

And when we do get a call that is a justifiable major incident, whoever gets the call becomes the “Incident Commander”, even if they have 30 seconds left on their shift. And even though our team primarily consists of people who live in the Southeastern United States, the incident in question could theoretically be in Alaska, Guam, or any other US territory that has an office of ours. Oh, and did I mention that nobody even bothered to give us a list of anyone (or their contact information) that works for any of those other offices? And when our division cheif asked for it from the powers that be, he was told rather tersely to go find it himself.

That reminds me, we were literally given no SOP’s, no training, no lists of callbacks, no nothing on how to do our new jobs, and some of the issues we had to deal with have had a national impact, and in the case of one item I had to deal with just this week, cost millions of dollars in lost revenue, which doesn’t even take into account the sheer amount of overtime requred to get it back up and running.

This entire thing has been universally despised by our entire group, and two people have already quit. Three more are threatening to do the same, and another is in the process of retiring six months early. The only reason I haven’t left is simply because my side business isn’t making enough regular income to justify leaving just yet. Once I do though, I’ll be leaving with a quickness because this is definitely not what I signed up for…

Of Course It’s My Fault

This story is far enough in my past now, I can finally tell it.  To give some background, I was working at a university doing desktop support.  THAT ALONE probably fills a few of you with dread.  But to be specific, I was working for the nice folks in Dewey Hall.  I say “nice folks” because they had the decency to hire me and set aside some of their budget for my salary.  The deal was, I worked for them, but if my schedule allowed, I was to help out at the neighboring buildings, Screwem Hall and Howe Hall.  Those buildings contributed nothing to IT and basically got free support.

I get a call over to Screwem Hall, and a tearful lady tells me that her “backup drive” isn’t working.  Because this isn’t technically my building, I have no idea what she’s talking about.

It turns out that, years ago, before my time, before my co-worker’s time, before my boss’s time, in fact, before anyone I knew actually worked there, someone had set this lady up with a four-disc RAID-1 array.  A bad storm had come through during the night, and this morning… it no workeez.  Bear in mind, at this point, you readers know more than I did while this lady is breaking into hysterics about her years of lost work.

“It was supposed to be automatically backed up!  That’s the entire purpose of that thing!  And it’s supposed to do it automatically, I’ve never had to touch it!  YEARS OF WORK ON THERE!  IF IT’S LOST I’M GOING TO COLLAPSE!”

Again, this is not my territory and in all my time here, no one ever alerted me to this drive’s existence.  So, I sit down and try to assess the situation… and as you’d all agree, the ideal time to analyze a system is NOT after it’s already had a massive failure.

…Turns out, this is a four-bay enclosure, with four 1TB drives, configured into a 2TB drive with a 2TB redundancy drive.     Obviously, if a disk failed, the data could be reconstructed from the redundancy drive.

…Except it wasn’t one disc that had failed, it was two.  So her chances of success now hinge on the failed drives NOT having the same data.  I decided to look into this possibility.

…Except that the array’s management software hadn’t been loaded onto this computer, since the array had been there so long the user had replaced their computer in the meantime, and no one but the original tech even knew the management software existed.  So I decide to access the menu of the drive directly.

…Except it’s a painful cluster of menus, and I’m afraid to even turn the damn thing off and on, for fear of making any data corruption worse than it already is.  The only thing I CAN tell is that it’s set to automatically rebuild, so if the data is all still on there, it’ll go into auto-pilot mode.

So I tell her the short version: that her main chance of success is to hope this thing works as designed.  I tell her to go buy me two appropriate-sized drives and let me install them.  She does so, I install them, and I get nothing.  No data comes back.

RAID LADY LOSES HER FUCKING MIND.  

There are words.  There are noises.  There are screams.  There are tears.  And that’s just for starters.  She calls my boss, reports me, announces a vendetta against our whole department.  Without even blinking an eye, I calmly list the following:

I had no idea this device existed in our environment, therefore, I had never been able to do any preventative maintenance on it.

Whoever set it up set it up in a highly questionable manner.  First, they did not put it on a UPS, which made the electrical damage from the storm that much more likely.  That was stupid.

Second, they used double the number of hard drives needed to achieve the space needed.  Even years ago, 2TB hard drives weren’t that expensive, least of all if you’re charging them to an academic budget.  By doubling the number of disks, he doubled the potential points of failure.  Stupid.

Third, all the hard drives were the same model and from the same batch, indicating if one had a manufacturing flaw, the others would suffer the same flaw, multiplying the potential for failure.  Stupid.

Fourth, all the hard drives were original to the installation.  None had been replaced proactively.  So all were out of warranty and well past their expected time to fail.  Stupid.

Fifth, the software suite which MIGHT have alerted this lady to the fact that her drives were in bad shape OR that her backups weren’t being done was never installed.  GOD DAMN FUCKING STUPID.

…But no, it’s apparently MY FAULT her data is gone.  Because I couldn’t fix the magic drive that was supposed to never break.

Microsoft says i have 128000 malware and i should call them….,

Got  a message from one of my customers. He says his computer shut down and then gave him a message saying he has 128000 pieces of malware and he needs toc all Microsoft. Now this user instead of calling me went and paid for it despite my always telling him not to fall for these things. I of course called hm back and tld him to power down the pc in hopes it gets them out! Now he is calling his credit card company to report the fraud. Lesson #1 always call your computer guy first!  SMH

When do i get my new computer

I installed a new computer for a customer to replace their 6+ year old desktop. It’s clearly a different computer a full sized tower compared tot he micro they had before and it has Win 10 as opposed to Win7. I get done setting it up (join to domain, have user log in etc) and the user turns to me and asks when she is getting her new computer. I’m like this is your new computer… her comment was then why isnt my screen bigger. Turns out she wanted a new monitor with the pc and they didn’t order one for her.