Saturday, February 25, 2012

Why tech support is the most important job in IT

When I was young I made the impulsive, somewhat stupid decision to drop out of high school and go work for an ISP I had set my mind on getting a job at. As a kid in the early 90s I had a very isolated computing world, and my ISP was a large part of it. They hosted newsgroups, an IRC server linked to a major network, sent detailed, technical announcement messages, had a really advanced website with lots of graphs, and had really smart people who would answer the phone 24/7 to help me get my external USR modem connected via serial. I wanted to meet these wizards who were up at 2:00AM setting up ISDN lines; I kinda wanted to be one.

Thanks to the bohemian owners and leadership at the ISP (and knowing one person who worked there via an extremely small IRC channel), I got a full-time job there one week before I turned 17. In hind-sight, I wouldn't have hired me with whatever color my hair was that month, but there I was on day 1, sitting on the floor of the office and being taught how to crimp CAT5 cable, because they were moving their datacenter around and that's all I did for about 2 weeks.

Over the 2 years I spent there (and 1 additional year at another similar company), I took about a million phone calls, knew how to setup every email client in existence off the top of my head, could configure dial-up modems (and, later, Cisco 675 DSL modem/routers, and even later, PPPoE DSL) while playing nethack, and could run through basic network configurations and troubleshooting without even trying. I had front row seats to the ILEC, CLEC, and DSL broadband revolution that happened. They built a giant datacenter that I got to watch come together every day, and play a minimal role helping them set it up (mostly via physical labor). Towards the end of my arc there, I was getting on the network gear and bringing up new routed blocks for colocate customers. That ISP was the highest concentration of smart, kind people I have seen so far in my life, and it was the best job a nerdy teenager could ever hope for.

And, eventually, I hated it. 3 years of total immersion in open source nerd world and generally doing my job well left me way overqualified (in skill) to do tech support. Unfortunately, I didn't quite have the skills or, probably more importantly, the maturity to make the often difficult jump from customer service to internal support (aka system administration for me), and I got increasingly frustrated with being sent easy but relatively advanced work from admins (like editing zone files) while helping a customer setup Outlook on the phone.

So I made another impulsive decision -- I quit my job, said fuck IT, and went to college. I didn't even make it through high school so I never expected or really wanted to go to college, but I got talked into taking a class at the local community college by a friend and unexpectedly found it very enjoyable. My mantra during college was to a) not take any computer or engineering classes and b) learn stuff I didn't already know.

5 years later, I had my BA in English and felt like a much more developed person. I also didn't really care about computers anymore -- my home LAN used to be quite impressive and was now down to one desktop and one headless Linux box -- but I knew way more about how they worked than just about everyone I met in the philosophy and English circles at my large, public university. Although I did catch one of my English advisers using Pine one day and gave them props (and mentioned mutt).

I made my life as easy as I could using technology while in school, but this was pretty limited. On campus I rarely carried a laptop around and mostly used the lab or library computers to download stuff from my home apache server and print it. In hindsight, I would have loved to have had a Macbook Air and Dropbox, but they didn't exist yet and one wouldn't have been a very practical purchase.

During college I was fortunate to have my writing ego stroked slightly by doing several contract gigs for Pearson Education. They kept asking me to write instructional-type stuff on something I knew stuff about, and I kept delivering it. It was a pretty awesome part time gig during college.

Right out of college I knew getting a job in anything related to English-degree-land would be an uphill battle -- to say the least. I didn't even want to be a writer. In fact, I was sick of writing papers, and although everyone told me I should go to grad school, I never seriously considered it. I thoroughly enjoyed college but my favorite part was graduating and it being over.

I threw my resume at my editors at Pearson and offered to relocate but it wasn't going to happen. So, after a quick month long road trip to drive around the states and see some friends, family, and clear my mind, I resigned to the fact that I would have to dust off my computer skills to pay the bills.

Within a couple months I got a low level job doing QA for a small software company -- which I felt was a good mix of computing and other skills I had picked up, and they slowly realized I could do a lot more than QA so I ended up doing a little bit of everything. But after about a year they weren't doing great and had to 2/3-time me. Shortly after that, I lucked out and landed sort of my dream job at a somewhat dysfunctional company in desperate need of a nerd who was motivated to untangle their mess of Linux servers. By the time I started, they had already fired the last admin. It has been an uphill but productive battle getting their servers in order and out of the stone age, and I've come to really like the place and the people.

What's even better is there is enough technical talent around that the help desk duty is minimal and my desk phone very rarely rings. It took about 6 years and a detour through the world of literature, but I made the leap from support to admin.

For a short time I let the relief of not ever having to setup another fucking Outlook install again sink in. It was a huge blessing and I subconsciously told my brain to start removing that dated information -- time to free up that space for another literary criticism paper.

One evening I was sitting at home (probably watching TV) and my phone rang. It was one of my bosses and I knew he was out of town visiting a big client, so I answered. He tells me he's sitting in the client CEO's house and they need help with his email program -- it's Outlook, and it's not working correctly. I tell him I am way rusty on this kind of thing, but I will try. We aren't even sure what version it is but the phone is passed over. A few minutes later, I've got my laptop out and the correct Outlook screenshots and instructions up and am helping the CEO of a multimillion dollar company setup his personal email. It goes perfectly -- he is an easy person to give tech support to because he happily follows the instructions -- it takes about 10 minutes total, and he is very appreciative, as is my manager, and the CEO of my company, who is also there.

There was no parade when they got home but I immediately realized that, as much as I don't want to admit it, I had to have my IT knowledge include basic tech support. I immediately got on our internal wiki and started writing basic documentation on stuff we setup or need to troubleshoot commonly, created some virtual machines of various versions of Windows and various versions of common software I would probably care about some day, and started refreshing my knowledge of email clients, browser settings, general Windows 7 troubleshooting, OSX troubleshooting, and mobile device setup.

I viewed this process not as going backwards in career, but as reinforcing my fundamental computer knowledge. I thought about it in terms of cars (which I know nothing about): a mechanic that can take apart and rebuild a motor but couldn't check the oil would be fired. I felt like even though my system administration and network engineering skills were becoming more advanced every day, I should still have a solid fundamental computing knowledge. To me, that started in tech support, so that's what I built my foundation on. My tech support knowledge is far from perfect but it's solid enough that Google is only complimenting it (as opposed to guiding it) -- most of the time. And to be honest, it's more about communicating than knowledge.

One job of a system administrator is to do cool shit with beefy hardware, but at the end of the day you and your closest IT peers are going to be the ones most impressed by the fact you migrated from physical machines to an HA cluster of virtual servers -- there is now a lot more magic available and it seems to work okay too!

Another job of a system administrator is to just making the stupid system work. Some days this means using your computing hammer on the database servers at 2:00AM, but some days it means explaining the difference between POP3 and IMAP to a new hire when all of their email seems to have disappeared. These tech support skills are useless if you don't actually use them, so make yourself available (but not too available) and be friendly (but not too friendly).

Make the stupid system work and don't be a jerk in the process -- it's a balance. Until you get to know the person a little bit and they come to trust your computer skills a little bit, treat everyone who needs help like those grandparents who would call in with a skeptical demeanor, have a rocky start, but be so thankful when you finally got it all working.

Tell people what you're doing, explain it to them, don't treat them like idiots -- use that English degree (heh). Most of the time this is totally appropriate and welcomed, but sometimes it isn't. Maybe the next time someone POPs all their email off and begins to panic, one of the people you've explained it to before will take care of it for you and you'll never even hear it. You can teach people to fish without making them crab boat captains.

Although the vast majority of my job involves evoking the dark magics and pleasing the blood lords (like any Unix admin), I still help a lot of people with a lot of basic stuff, they are still thankful when it works, and I still crimp a lot of fucking cables.

Sunday, January 22, 2012

Brick and mortar stores still suck at the internet

It amazes me that prominent members of the brick and mortar retail industry are still completely clueless when it comes to how they need to be operating online. I remember about a decade ago I had tracked down a specific part for my vacuum via the sears.com website. When I drove 15 miles north to pick it up, they had no idea what I was talking about, couldn't do anything with the online item number I brought from the website, and had no vacuum part. First world problem -- I know.

Over the weekend I have been rearranging the living room and it turns out my 6' TOSLINK cable that pipes delicious audio from my desktop computer to the nice speakers isn't long enough anymore. We have a new furry member in the household that likes to eat wires so I have to do the new wiring carefully and it is somewhat inconvenient to change it later after it has all been bundled nicely and "locked in". With this in mind, I was ready to pay the 100%+ premium (compared to ordering from Monoprice or Amazon) and grab a cable at Fry's.

Turns out I suck at estimating lengths and the longest cable Fry's sells (15' for $30) didn't make it by about 3'. Doh. I checked Amazon Prime for 25' ones (preferably from their quality Amazon Basics brand) and they didn't have any obviously available, so I found what I needed in about 45 seconds on Monoprice for $5 + $5 shipping. I'll take the other one back to Fry's -- done deal, or so I thought.

This morning -- in a mix of curiosity, boredom, impulsiveness, and disappointment with the temporary, shitty wiring job I had to do using the too-short cable -- I decided to check Best Buy's website to see if they have what I need. Hey, with a family member who seems to survive on a diet of about 10-15% wires, it couldn't hurt to have 2 of these around.

I hit up http://www.bestbuy.com and searched for TOSLINK. Wow! A 20' Startech cable for $21 -- that isn't bad at all. I see a trip to Best Buy in my immediate future -- perhaps with an impulsive video game buy, or a replacement iPhone cable (stupid cat).

Best Buy doesn't have the best reputation for being super competent but hopefully they have enough clue to have their online to brick and mortar situation down pretty well.

Green Flag: "Check Stock In Stores Near Me" -- this should be standard at any online store with a significant brick and mortar presence.

Yellow Flag #1: the "BETA" icon next to the in-stock search -- it's 2012 and fairly talented US web developers are a dime a dozen. Find a shop or hire some coders and get the online stock checking perfected now. Fire anyone who tells you it's very difficult or impossible, because it isn't if you don't suck.

Yellow Flag #2: "Marketplace Item, Sold and Shipped by Buy.com" -- fuck this. Best Buy, don't be naive; don't waste your money trying to be Amazon; you will never win. Amazon built a brilliant shopping platform (and an even more brilliant computing platform) and you can't compete without bringing significantly new things to the table, which you aren't. I'm going to go out on a hyperbolic limb and guess Bezos has more innovation in his morning piss than your entire board of directors combined.

Anyway, back to the website and still feeling hopeful I will be making a trip to Best Buy today. Let me put in my zip code -- which is one of the densest neighborhoods on the west coast, with close to 4 million people in the surrounding area, and has at least 5 Best Buys if you go 10 miles in any direction -- I'm sure this relatively common 20' cable from this well known cable manufacturer is available at this store that should have its online shit together. "Go".



"There are not any stores within a 50 mile radius of 98122 with any of the displayed products in stock"

What. The. Fuck. That is one harsh informative message and one grand display of online retail incompetency. Entire floors of marketing, retail, and online sales people at Best Buy's headquarters should probably be fired over this.

If I wanted to order something online, I'm going to go to the place that I already pay $79 a year to for fast, free shipping and order it. They have almost everything I need if I can wait a whole 2-3 days (sometimes less). And if they don't have a sort of specialty item like a really long TOSLINK cable available via Prime, I'll go to a different online site that specializes in high quality cables of every type and size you could imagine at simply amazing prices that really highlight the insulting markup on cables found in most brick and mortar stores.

At no point during my online shopping experience does the thought of going to bestbuy.com and paying for shipping for something that is likely at least a little bit overpriced (and sometimes significantly overpriced, like cables) enter my head, nor will it ever at any point in the foreseeable future. I don't think I've ever ordered anything from bestbuy.com, but I've definitely made about a half dozen trips to the local stores in the last year and spent a few hundred bucks.

The only time bestbuy.com becomes relevant is if I'm being an impatient, impulsive consumer and I want to know if I can get my cable now -- possibly without even changing out of what I slept in or taking one look in a mirror. And it completely fails at this seemingly no-brainer service.

I guess I'm waiting until Wednesday or Thursday when my Monoprice cable shows up. Best Buy lost $20-70 worth of revenue today and won a badge they should be trying to avoid: sucking at being a brick and mortar store with a website. And they are far from alone.

Off the top of my head, one place that seems to be doing it right is Walmart. I honestly don't shop there very often (they don't have a large footprint in the Seattle area), but I recall their site-to-store connectivity being high quality. You've probably been taking lessons from Walmart for years -- here is another one.

Saturday, December 3, 2011

Storage Wars


In just 22 minutes this amazing cultural touchstone of a show manages to sum up so much of America. I may have watched every episode in existence (2 seasons) over the last week, including a 6 hour marathon on my Thursday off – during which I accomplished absolutely nothing else.

Obviously, as a result of this show, I have decided to quit my job, open a thrift store, and show up at the next storage locker auction with a truck and cash in hand. It actually looks pretty easy if you have a way to distribute the crap and aren't a complete moron.

The basic premise of this show, if you haven't seen it, is that somebody didn't pay their storage locker bills and the contents are going to be auctioned off to the highest bidder. They open the door up, give you 5 minutes to look (you can't go inside or touch anything), and then you bid, cash only. Afterwards you plunder this poor person's crap like a tomb robber and try to turn a profit like a good neoliberal capitalist. You're basically robbing poor people of all their stuff behind their back while it is all filmed and celebrated.

The show isn’t really about knowing what stuff is, or being able to spot value in piles of somewhere between boxes and crap; it's about being able to sum up – based on just a few visual clues – if this person has the type of life you'd want to buy at a discount rate, and if anyone else will be dumb enough to buy their dusty junk later.

Although there are commercial exceptions, most of the units are simply someone's entire house in a box. I’m going to guess that a person who packs and entire middle class household into a storage unit may be having sort of a shitty time in their life. I’m going to guess they really would prefer not to see their stuff auctioned off. I am wondering how awkward it would be to see your own storage unit rifled through and judged on TV. I’m wondering how many of these people are dead, which is probably one of the best case scenarios.

There are four main players doing the buying and on the show. I would say profit-turning as well, but one of the guys isn’t really in it for the cash.

The old retired guy – the antique spotter – is the easiest to understand. He doesn't want to resell everything at a profit, he is just looking for cool single items, and he sort of finds them. Over the course of 2 seasons I’d say he has found like 2-3 legitimately cool things – my personal favorite being the pool cue holder. However, I seriously do not believe the amount of money people will pay for antiques. Call me a cheap low-baller but I’d say the cash values on this show are inflated by at least double on average.

The young couple newbie team is my favorite cause the wife is that crafty mix of smart and bitchy and sweet and cute. The guy married way out of his league. They play-fight over everything and at first I hated her, then I hated them for being so cute, and now I like them the most by far. In one episode this woman found the most ridiculous plaster horse head that you hang on the wall and she struts out with it in an otherwise waste-of-money unit and I’m thinking this gaudy plastic crap is valued at $0. Turns out it's some French gold foil thing where in ye olden days you hung it outside of your shop if you dealt with horse meat and the dude gives her 2 grand cash for it on the spot. Who the hell gave these people so much cash?

Then there are two more dudes that are like alpha males who own businesses to distribute the wares and are in it for 100% profit and bring their kids for slave labor and general nepotism. The one who looks like a tough guy is kind of a whiny pussy (but I like his kid), and the one that looks like a pussy is a hardcore asshole and the one I respect the most. The latter is by far the smartest and has made some nice cream on completely obvious units full of commercial grade gear new in boxes that nobody else bid on for some reason.

There is also an auctioneer who is always with this blonde woman and he wears a big, gold pinky ring and asked a psychic (brought in to feel the energy of the unit, of course) what color underwear he was wearing (and she got it right) and he drives a nice car and is a little bit fabulous so obviously he is gay. Then this pretty, cheery blonde woman that I assumed was his minion starts calling him husband and I’m like whaaaaat. They seem like a classy couple and I have no idea how they put up with these yokels.

Anyway, it's my new favorite show on A&E. I have evolved from Intervention to Hoarders to Storage War$. There is a new season starting on Tuesday.

Sunday, July 31, 2011

Cisco ASA 5500 and iPhone/iPad VPN

This weekend I decided to setup VPN access to my home network for my iPhone and iPad. SSH just doesn't cut it all the time. I knew VPN was available natively in iOS but I was disappointed that nobody had written an app that supports OpenVPN -- my VPN software of choice. Jailbreaking (on iPhones at least) doesn't have a great hassle:reward ratio these days so I decided it was time to man up and turn my ASA into a VPN server that would be compatible with iOS.

First off, let me tell you about the general setup. The router is a Cisco ASA 5505 running ASA IOS 8.2(5) with a basic license. It is already doing NAT, dhcp, and some basic port mapping for my home network. The phone is an iPhone4 with the current Verizon iOS firmware (4.2.10). It is not jailbroken. And my iPad is running iOS 4.3.5 -- also not jailbroken.

Googling various combinations of "VPN" "iPhone" and "ASA" eventually landed me at this excellent Cisco article:

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/l2tp_ips.html

Although this looks a little scary at first, it is the best resource I found for understanding and setting up the basics. It also gives you a mostly working configuration example. In fact, there are only a handful of lines I change from its configuration example to "ensure ASA compatibility with a native VPN client on any operating system."

90% pasted straight from that Cisco link, here is what is required on your ASA:
ip local pool VPN_pool 172.16.29.1-172.16.29.10
group-policy VPN_policy internal
group-policy VPN_policy attributes
 dns-server value 8.8.8.8 8.8.4.4
 vpn-tunnel-protocol l2tp-ipsec
tunnel-group DefaultRAGroup general-attributes
 default-group-policy VPN_policy
 address-pool VPN_pool
tunnel-group DefaultRAGroup ipsec-attributes
 pre-shared-key *
tunnel-group DefaultRAGroup ppp-attributes
 no authentication pap
 authentication chap
 authentication ms-chap-v1
 authentication ms-chap-v2
crypto ipsec transform-set trans esp-3des esp-sha-hmac
crypto ipsec transform-set trans mode transport
crypto dynamic-map dyno 10 set transform-set trans
crypto map vpn 20 ipsec-isakmp dynamic dyno
crypto map vpn interface outside
crypto isakmp enable outside
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
crypto isakmp nat-traversal 20
username happyvpn password changeme mschap
access-list VPN_nonat extended permit ip 172.16.31.0 255.255.255.0 172.16.29.0 255.255.255.240
nat (inside) 0 access-list VPN_nonat
Let's go over the parts of this you shouldn't blindly paste from top to bottom:
  • 172.16.29.1-172.16.29.10 are arbitrary IPs that are assigned to VPN clients. Since a basic license can only handle 10 VPN clients, it is pointless for me to make my range larger than that. These IPs must be different from your normal internal IP range(s) -- it should be a totally new local subnet

  • The dns-server values should be changed to your preferred nameservers, possibly on your local network

  • DefaultRAGroup can not be edited. This is a special Cisco tunnel-group

  • pre-shared-key * is one of the two passwords needed to connect, the * represents what you want that password to be

  • At the previously mentioned cisco.com link, "crypto dynamic-map dyno 10 set transform-set set trans" has a typo in it and gives a fairly straightforward error. It is fixed in my version above

  • username happyvpn password changeme mschap is how you create an account on the ASA for a VPN user. There are other ways of authenticating but this is the easiest

  • access-list VPN_nonat extended permit ip 172.16.31.0 255.255.255.0 172.16.29.0 255.255.255.240 In this example, 172.16.31.0/24 is my normal home network, and 172.16.29.0/27 is the range I assign VPN IPs from (as set by the ip local pool VPN_pool ... command)
That is the full ASA VPN configuration needed.

The iPhone setup is very simple. This is under Settings->General->Network->VPN:


  • The Server is your public IP or hostname. I have been using a free account at DynDNS.com for years on my home cablemodem with a dynamic IP

  • The Account is the username your set on the ASA with username; the Password is its password

  • The Secret is the password you set with pre-shared-key

  • Send All Traffic does not work with this configuration because of the VPN_nonat rule. I didn't want to send all my traffic through my home VPN anyway
Finally, flip VPN from Off to On and you should get connected with a new VPN icon at the top:


You now have access to the private IPs behind your ASA like any happy VPN connection.

Doesn't work? Let me give you the phone number and IRC haunts of my buddy to helped me when I was stuck on that VPN_nonat thing... one sec... okay just kidding. I am by no means a Cisco guru and probably can't help you! This exact configuration works for me under the versions mentioned above as of today. I did find one document from Cisco on common VPN problems that includes fixes for both traditional IOS devices and PIX/ASA IOS:

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00807e0aca.shtml

That covers some solid troubleshooting. My next advice would be to setup logging on your ASA and see what the logs say. I used a simple app to ping for testing from the phone. Finally, some useful stats/troubleshooting commands on your ASA are:
show vpn-sessiondb
show crypto isakmp sa
show crypto ipsec sa
Good luck!