Showing posts with label system administration. Show all posts
Showing posts with label system administration. Show all posts

Friday, October 12, 2012

Clearing up the cloud

Congratulation Mercedes! I was watching TV the other day and one of your commercials where you said your cars are "cloud powered" or "cloud enabled" or something like that came on and I knew it had happened: the use of the word cloud has officially become a joke.

Overall, I realize marketing and salespeople mean no harm when they drop the c-word, and I realize that the vast majority of consumers aren't big enough nerds to have a nuanced idea of what cloud computing is, but I definitely think there is room for some general education on what the word means, and what systems administrators hear when you say it.

If the term cloud strikes you as an old one, it's because it's been around forever. You won't get 50 pages into any networking book from the 90s before you find your first network diagram that involves something that looks like a cloud. Historically, the cloud has been used in network diagrams to signify the public internet -- the part of the network where you no longer have control.

The first definition of cloud is simply the public internet. So if your company has a presence on the public internet, congratulations! You can put cloud on as many pieces of marketing you want to.

Obviously that is not a very high bar -- just about anyone is on the internet (you bet your ass my blog is cloud powered) -- so this definition of cloud has become more like #2 at this point.

The other definition is much more interesting, but requires a little bit of digging into how the internet works. I'll do my best to keep it simple.

Amazon Web Services is a shining example of what cloud computing really is. Note that I have never actually used it before, but that's part of why it's so good: I could be torn out of my world of physical servers and trips to datacenters and thrown into a world where everything is virtually hosted by the AWS cloud of servers that I will never see or physically interact with without really feeling that much pain. Amazon built a great system of servers to sell people stuff, but then they started to say to themselves, "What else can our cool network of servers do?" and they came up with AWS. Now I question what date the projection charts say AWS will be a bigger business than amazon.com the shopping site, if it hasn't happened already.

Traditionally, if I wanted to host a website somewhere, I'd have to do it myself using my own computer resources, or I'd pay a managed hosting provider to do it. Managed hosting is the right choice for a lot of people (and we can think of Google as the managed hosting provider of this blog), but you have to live within the confines of what the provider is willing to let you do (Google won't let me install Wordpress on blogspot for some reason), and what other people you share the servers with are doing. These drawbacks are often irrelevant (I could care less how many catblogs blogger hosts), but as you design more complicated web-based software, you really start to see the advantages of managing your own hosting.

It used to be that managing your own hosting in a business environment was a much more significant cost and time undertaking than it has the potential to be now, largely because of services like AWS that make it easy. You had to hire a systems person to coordinate it all, you had to find and lease some space at a datacenter to put your servers in, you had to buy a significant amount of server and networking hardware. The scale hugely varies, but nobody would call it cheap. I'd ballpark a full cabinet in a nice datacenter with a reasonable amount of bandwidth as bare minimum $1,000 per month, probably closer to $1,500, and definitely with 2-3 year contracts attached. That's just the rent, humans and hardware not included.

Another disadvantage to managing your own servers is that you have to worry about a lot of things that aren't necessarily top concerns at managed hosting places. For example, suddenly you are responsible for all the redundancy that must be built into your servers in expectation of hardware failing, you are responsible for keeping your backups running consistently and knowing how to use them, you are responsible for figuring out how to serve your clients traffic if your entire datacenter goes offline -- which definitely happens (though good luck getting a good datacenter sales guy to even acknowledge the possibility of it happening).

I'm not saying all managed hosting providers are competently on top of all these major issues (some certainly are), but at least they aren't entirely your responsibility.

This is where AWS comes in. Instead of having to deal with your own physical servers located in datacenters, what if Amazon told you they could sell you some of their unused capacity and you could treat it exactly as if it were your own computer? And you'd be charged based solely on usage. And it can scale as big as amazon.com which is infinitely for all intents and purposes. And it can handle catastrophic failures because it is big enough to be designed to. That is what AWS offers.

This is cloud computing: servers that scale immediately and pretty much infinitely, located in geographically distinct locations in case of major catastrophes, and have the ability to heal themselves within some generous limits.

It would be like if your Mercedes SL sports car suddenly turned into a G people-mover when you had more passengers to transport. Or if you got a flat tire, an identical Mercedes would appear within a few seconds and seamlessly move you and your passengers into the working vehicle. Now that is a cloud powered Mercedes!

Managed hosting is still the right choice for a lot of people. AWS is the right choice for a hugely increasing number of people. Hosting your own hardware is the right choice for another significant chunk of people. For now, the tech world is respectably split into these categories and none hold a significant monopoly, but I have a feeling that will change as AWS puts managed hosting out of business, and companies can't resist the appeal of doing things cheaper in AWS instead of managing it by themselves entirely.

No matter what you do, you are technically "cloud powered" to some extent if you are on the internet, but marketers and, more importantly, sales people need to understand when you drop the word cloud on a nerd who manages private clouds, or one that manages servers within a public cloud, you are using an extremely loaded term -- certainly not one that has any business being applied to cars.

Finally, I have used AWS as my sole example of a cloud computing provider in this post, but there are plenty of competitors, and the number is growing daily. I am still of the school that does everything in-house and I think I like it, but I do get jealous of my systems friends who provision very large servers with a few dozen keystrokes, while I have provision large servers by researching exactly what hardware I want, convincing my bosses it's a good idea to buy then, lugging them to the datacenter, getting them mounted and cabled properly, installing them, and networking them.

Saturday, February 25, 2012

Why tech support is the most important job in IT

When I was young I made the impulsive, somewhat stupid decision to drop out of high school and go work for an ISP I had set my mind on getting a job at. As a kid in the early 90s I had a very isolated computing world, and my ISP was a large part of it. They hosted newsgroups, an IRC server linked to a major network, sent detailed, technical announcement messages, had a really advanced website with lots of graphs, and had really smart people who would answer the phone 24/7 to help me get my external USR modem connected via serial. I wanted to meet these wizards who were up at 2:00AM setting up ISDN lines; I kinda wanted to be one.

Thanks to the bohemian owners and leadership at the ISP (and knowing one person who worked there via an extremely small IRC channel), I got a full-time job there one week before I turned 17. In hind-sight, I wouldn't have hired me with whatever color my hair was that month, but there I was on day 1, sitting on the floor of the office and being taught how to crimp CAT5 cable, because they were moving their datacenter around and that's all I did for about 2 weeks.

Over the 2 years I spent there (and 1 additional year at another similar company), I took about a million phone calls, knew how to setup every email client in existence off the top of my head, could configure dial-up modems (and, later, Cisco 675 DSL modem/routers, and even later, PPPoE DSL) while playing nethack, and could run through basic network configurations and troubleshooting without even trying. I had front row seats to the ILEC, CLEC, and DSL broadband revolution that happened. They built a giant datacenter that I got to watch come together every day, and play a minimal role helping them set it up (mostly via physical labor). Towards the end of my arc there, I was getting on the network gear and bringing up new routed blocks for colocate customers. That ISP was the highest concentration of smart, kind people I have seen so far in my life, and it was the best job a nerdy teenager could ever hope for.

And, eventually, I hated it. 3 years of total immersion in open source nerd world and generally doing my job well left me way overqualified (in skill) to do tech support. Unfortunately, I didn't quite have the skills or, probably more importantly, the maturity to make the often difficult jump from customer service to internal support (aka system administration for me), and I got increasingly frustrated with being sent easy but relatively advanced work from admins (like editing zone files) while helping a customer setup Outlook on the phone.

So I made another impulsive decision -- I quit my job, said fuck IT, and went to college. I didn't even make it through high school so I never expected or really wanted to go to college, but I got talked into taking a class at the local community college by a friend and unexpectedly found it very enjoyable. My mantra during college was to a) not take any computer or engineering classes and b) learn stuff I didn't already know.

5 years later, I had my BA in English and felt like a much more developed person. I also didn't really care about computers anymore -- my home LAN used to be quite impressive and was now down to one desktop and one headless Linux box -- but I knew way more about how they worked than just about everyone I met in the philosophy and English circles at my large, public university. Although I did catch one of my English advisers using Pine one day and gave them props (and mentioned mutt).

I made my life as easy as I could using technology while in school, but this was pretty limited. On campus I rarely carried a laptop around and mostly used the lab or library computers to download stuff from my home apache server and print it. In hindsight, I would have loved to have had a Macbook Air and Dropbox, but they didn't exist yet and one wouldn't have been a very practical purchase.

During college I was fortunate to have my writing ego stroked slightly by doing several contract gigs for Pearson Education. They kept asking me to write instructional-type stuff on something I knew stuff about, and I kept delivering it. It was a pretty awesome part time gig during college.

Right out of college I knew getting a job in anything related to English-degree-land would be an uphill battle -- to say the least. I didn't even want to be a writer. In fact, I was sick of writing papers, and although everyone told me I should go to grad school, I never seriously considered it. I thoroughly enjoyed college but my favorite part was graduating and it being over.

I threw my resume at my editors at Pearson and offered to relocate but it wasn't going to happen. So, after a quick month long road trip to drive around the states and see some friends, family, and clear my mind, I resigned to the fact that I would have to dust off my computer skills to pay the bills.

Within a couple months I got a low level job doing QA for a small software company -- which I felt was a good mix of computing and other skills I had picked up, and they slowly realized I could do a lot more than QA so I ended up doing a little bit of everything. But after about a year they weren't doing great and had to 2/3-time me. Shortly after that, I lucked out and landed sort of my dream job at a somewhat dysfunctional company in desperate need of a nerd who was motivated to untangle their mess of Linux servers. By the time I started, they had already fired the last admin. It has been an uphill but productive battle getting their servers in order and out of the stone age, and I've come to really like the place and the people.

What's even better is there is enough technical talent around that the help desk duty is minimal and my desk phone very rarely rings. It took about 6 years and a detour through the world of literature, but I made the leap from support to admin.

For a short time I let the relief of not ever having to setup another fucking Outlook install again sink in. It was a huge blessing and I subconsciously told my brain to start removing that dated information -- time to free up that space for another literary criticism paper.

One evening I was sitting at home (probably watching TV) and my phone rang. It was one of my bosses and I knew he was out of town visiting a big client, so I answered. He tells me he's sitting in the client CEO's house and they need help with his email program -- it's Outlook, and it's not working correctly. I tell him I am way rusty on this kind of thing, but I will try. We aren't even sure what version it is but the phone is passed over. A few minutes later, I've got my laptop out and the correct Outlook screenshots and instructions up and am helping the CEO of a multimillion dollar company setup his personal email. It goes perfectly -- he is an easy person to give tech support to because he happily follows the instructions -- it takes about 10 minutes total, and he is very appreciative, as is my manager, and the CEO of my company, who is also there.

There was no parade when they got home but I immediately realized that, as much as I don't want to admit it, I had to have my IT knowledge include basic tech support. I immediately got on our internal wiki and started writing basic documentation on stuff we setup or need to troubleshoot commonly, created some virtual machines of various versions of Windows and various versions of common software I would probably care about some day, and started refreshing my knowledge of email clients, browser settings, general Windows 7 troubleshooting, OSX troubleshooting, and mobile device setup.

I viewed this process not as going backwards in career, but as reinforcing my fundamental computer knowledge. I thought about it in terms of cars (which I know nothing about): a mechanic that can take apart and rebuild a motor but couldn't check the oil would be fired. I felt like even though my system administration and network engineering skills were becoming more advanced every day, I should still have a solid fundamental computing knowledge. To me, that started in tech support, so that's what I built my foundation on. My tech support knowledge is far from perfect but it's solid enough that Google is only complimenting it (as opposed to guiding it) -- most of the time. And to be honest, it's more about communicating than knowledge.

One job of a system administrator is to do cool shit with beefy hardware, but at the end of the day you and your closest IT peers are going to be the ones most impressed by the fact you migrated from physical machines to an HA cluster of virtual servers -- there is now a lot more magic available and it seems to work okay too!

Another job of a system administrator is to just making the stupid system work. Some days this means using your computing hammer on the database servers at 2:00AM, but some days it means explaining the difference between POP3 and IMAP to a new hire when all of their email seems to have disappeared. These tech support skills are useless if you don't actually use them, so make yourself available (but not too available) and be friendly (but not too friendly).

Make the stupid system work and don't be a jerk in the process -- it's a balance. Until you get to know the person a little bit and they come to trust your computer skills a little bit, treat everyone who needs help like those grandparents who would call in with a skeptical demeanor, have a rocky start, but be so thankful when you finally got it all working.

Tell people what you're doing, explain it to them, don't treat them like idiots -- use that English degree (heh). Most of the time this is totally appropriate and welcomed, but sometimes it isn't. Maybe the next time someone POPs all their email off and begins to panic, one of the people you've explained it to before will take care of it for you and you'll never even hear it. You can teach people to fish without making them crab boat captains.

Although the vast majority of my job involves evoking the dark magics and pleasing the blood lords (like any Unix admin), I still help a lot of people with a lot of basic stuff, they are still thankful when it works, and I still crimp a lot of fucking cables.