I got a call today, and it was one of those calls where I felt almost like I fell into a version of the Twilight Zone especially for IT people. A Tier 1 Desk Monkey called me to tell me he had a local tech on the phone who wanted to have a ticket put in to have some wireless access points checked, and who claimed he had exhausted all his options an needed Tier 3 LAN support.

I almost had to do a double take when I heard that, for two reasons:

  • Why couldn’t the local tech just create the ticket himself and send it up to our team to route where it needed to go, or better yet, send it himself?
  • Why couldn’t the Tier 1 Desk Monkey create the ticket for him and send it to us?

Both $localTech and the Tier 1 Desk Monkey had the same access I do to the ticketing system so there was literally nothing preventing them from putting the ticket in themselves, except perhaps sheer laziness, since neither of them were new employees, and had been around the block enough times to know that they can put the ticket in themselves.

Boy are my arms tired

Had to do a bit of work on the forum. Apparently, a bunch of zombie nodes added over 3700 users to the forum, each producing a couple posts in the forum. Ended up pruning the user list for any users after the 6th of January. If anyone legit registered on the forum after Ninja, I do apologize, but feel free and register again, I also banned a bunch of the IP’s they were coming from, but after the first couple dozen, I realized each rogue user registration was from a different one.

Office Space kinda day

Today, I seriously felt like I was in Office Space…

Within a 45-minute time span, I was called three times, emailed four times, and even instant messaged twice, all by separate Tier 1 Desk Monkeys within the same office, all for the same problem. The utter lack of anything resembling coordination of any kind was as laughable as it was sadly typical. In fact, after the 2nd phone call with one Desk Monkey telling me that she had submitted an official outage notification, I was immediately IM’d by another one from the same team, asking me if I was going to submit an outage notification. I swear I could FEEL what it was like to go insane because I had to sit there and tell her that her co-worker, who she admitted worked three desks down from her, had JUST submitted the very outage notification she had just asked me to put in for this. And bear in mind, this was already after the local site had already called in to put a separate outage notification of their own… over an HOUR earlier, and which we were well aware of. Even Chicken Little got in on the action by asking us (again, after two outage notifications were already submitted), asking us if we had verified this in fact was a major service disruption, and if an outage notification was being put in…

And pray tell, what was so important that eight separate people felt the need to contact me within a 45-minute timeframe? A single network switch was down in a remote office because the (very old) UPS it was plugged into finally failed, and the three people in that office felt compelled to insist that the Desk Monkeys put it in as a high-priority issue, even though they had no dealings with patients whatsoever.

Say what now?

Today, I got a call from the Tier 1 Desk Monkeys that made me do a double take. The Desk Monkey had a user on the line that insisted whoever he called come into that facility to give him access to $serverRoom. I told him it wouldn’t be advisable for me to talk to $user for several reasons:

  • I lived 4 states away from this facility’s location, meaning if $user wanted ME to come in and give him access to $serverRoom, he had a long wait ahead of him (minimum 12-14 hours depending on traffic)
  • Even if I did live close by, I didn’t even have physical access to that facility’s server room at all, to say nothing about allowing some complete stranger into the room simply because they asked it of me
  • Why in the name of all that was good and right with the Universe didn’t the Tier 1 Desk Monkeys call… oh… I dunno… the local IT department for the facility?
  • And on that note, why did the Desk Monkey even bother putting in a ticket as the lowest priority possible, sending it to us, when their own protocols (which they’ve gone to great pains to remind us many times) dictate that they should only call us for high priority tickets or above

I told him there was nothing I could do for $user, and I’d have to call $localIT to give him the necessary access, if in fact he had the authority to be in there for the first place. When I called $localOnCall, he laughed a bit and agreed with me that the Desk Monkeys were instructed to call him personally for such a request, but they ignored the request and called my group instead.

Chicken Little

The other day, I mentioned that I recently have had the displeasure of beginning to deal with a Tier 1 Desk Monkey who seems to think every supposedly “high priority” ticket, especially those tickets my team is utterly unaware of, somehow requires him to IM me specifically, and (more recently) email or call my team to ask us if we’re doing something to fix it, and if we can validate that this really is an outage, as opposed to him doing these things himself. He’s done this so often, my team’s named him Chicken Little because he seems to think the sky is falling, when it’s almost always nothing of the sort.

Yesterday was no different. He emailed us about a ticket, asking us if we were aware of it, what we were doing to fix it, and whether we were going to declare this a site-wide outage. A cursory check of the ticket showed it was less than 5 minutes old, and was regarded a single switch being down in one section of the local facility, and not the entire site. A check of our monitoring system showed all the site’s switches/routers/servers were in the green, so we replied back to Chicken Little, informing him that not only was this ticket a mere 5 minutes old, but we were unaware of any issues, as our monitoring system showed everything was up and running for the site, nor had the local site even had time to acknowledge the ticket’s existence, let alone reach out for help. We even CC’d the local site to make sure, and they verified that the switch in question was rebooted to finish upgrading the firmware on it, and the users for that section were warned in advance, but a couple either forgot about it, or simply neglected to read the email, and promptly called in, claiming the entire network was down. The local IT folks verified everything was fine, and asked Chicken Little to close the ticket.

He complained that closing tickets was no longer his job, but apparently, reading though the ticket notes wasn’t a part of his job either, since on a number of prior tickets he sent to us, asking if we knew about it/asked us to declare it an outage, the site itself closed the ticket and fixed it prior to him emailing my team, but he apparently missed the fact the ticket was closed before emailing us and asking us questions about it. There’s a general consensus among my team that someone’s ordering him to do this in a vain effort to deflect the prior issues they had where the Tier 1 Desk Monkeys, by default, would declare every high priority ticket a site-wide outage, even if it was a single starfish claiming that their mouse not working constituted a “patient care issue” which a Tier 1 Desk Monkey would dutifully bump up to a higher priority. There’s also a grumbling among several team members in my group that there’s some ulterior motive/agenda for doing this, and there is something fishy in the state of Denmark.