There be dragons! Entering a DevOps world

This post was written for the Linux Recruit blog, you can find their blog here

Systems Operations has been a very vocational task, it’s something that still
nowadays it is hard to learn at University and it came out of passion for hardware, networks, OSes and services working together like a technological orchestra. SysAdmins have passed tricks from the beginning, everyone created their own style and “bag of tricks” that could use to solve problems in a quick and efficient way.

DevOps has in a certain way optimised the methodology and approach to System Administration, due to the sheer volume of servers that a SysAdmin needs to manage. A SysAdmin would have been expected to manage anything from 50 to 100 servers in the past, where now that is considered a low baseline for what it is expected, something had to give and the traditional ways that I learned when I started in this profession did not cut it.

With DevOps this has changed substantially, today the work of SysAdmins has a lot more in common with Developers than what it used to, at the same time Developers have to dive more and more into the guts of the machine in order to be able to write code that scales, hence the alliance of both under the DevOps umbrella.

Right now a SysAdmin or a DevOps Engineer (still think that DevOps in a work title is wrong) needs to know a lot more not only about the way Systems work but also the way code integrates, converges, releases, tests, etc. On top of that the amount of tools available in the market has boomed in the last two years making the time to learn and adapt between new tools a lot shorter than it has ever been.

A bit of history

DevOps from the Ops side started with the Webops movement in companies like Twitter, Flickr or Google, people like John Allspaw, John Adams and Tom Limoncelli where using pure maths and known optimisation mechanisms
in other industries to start optimising the way we SysAdmins did work and started writing about them in the most reputable industry publications like Linux Magazine, Linux Journal and Usenix ;login;.

Everything started booming with the wonderful work of Mark Burgess creating CFEngine, truly a visionary of its time, the premise of Configuration Management and Systems Automation was something that was not considered as SysAdmins used their bag of tricks to keep maintaing systems in an efficient way without it as it required a level of abstraction that wasn’t stable enough to trust for production.

CFEngine became better and started gaining tractions, this meant that other competitors entered the market like Luke Kanies and Teyo Tyree founding PuppetLabs, Jessee Robbins and Adam Jacob did the same founding Opscode. CFEngine, Puppet and Chef started learning from one another and getting more and more efficient and robust, they started gaining big users like Facebook, Google (not for servers) and Etsy and even the most sceptic SysAdmins started playing with them.

Automation opened the door to better releases and destroyed the concept of servers being beautiful unique beings, a server could now be erased and brought back to life in little time, but as with any big power it carries a big responsability so mistakes in Automation would have bigger consequences than a manual deploy (even if they were less common). Thankfully the flexibility of being able to change server configuration on demand made testing a lot easier (and since Automation was code as well it enforced its tests), this was the seed for pure Continous Integration.

Things got very interesting with Amazon bringing servers on demand creating the first public Cloud available, this now meant that servers could be created and destroyed on demand, the need for big servers to carry a lot of traffic was not there anymore (as demonstrated by Google) so the commodisation of server hardware meant that the number of servers to manage would easily multiply.

Automation and the Cloud meant that SysAdmins as myself needed to manage hundreds of the servers with the same headspace and capabilities that we used to manage a lot less, tools and practices needed to evolve and WebOps was born out of this.

WebOps quickly evolved with the need to include developers into what is known as DevOps thanks to Patrick Debois creating DevOpsDays in Belgium, a place where Operations and Developers could talk about their common problems and reach goals together.

Nice story bro, how do I get into DevOps?

Getting into DevOps nowadays require a very steep learning curve, as I talked about in the book “Build Quality In” (buy it, it’s for a good cause) the steps to get into the DevOps mindset are usually as follows:

  1. Automation
  2. Provisioning
  3. Deployment
  4. Continuous Integration
  5. Continuous Delivery
  6. Cultural Change

Luckily for me I was able to learn each one of them as they came to the market and evolved, if you are starting now you will not be that lucky! I will try to recommend some book to help you get there.

Having a solid SysAdmin background is essential for this, I started with the book “The Unix Programming Environment” from Kernighan and Pike but that definitely will be outdated nowadays, you can start by finding a good Linux or Windows SysAdmin book that you feel comfortable with.

Getting into the DevOps mindset can be definitely fun, having a read at “The Phoenix Project” by Gene Kim and “Web Operations” by John Allspaw will get you in the right vibe.

Automation is the main gateway to DevOps no matter if you come from the Ops or Dev side, understanding the basics of Automation will help you embrace all the rest a lot quicker, there are some very good books out there, I specially recommend “Pro Puppet” by James Turnbull, there are also very good books on Chef like “Test-Driven Infrastructure with Chef” by Stephen Nelson-Smith.

If you want to get very deeploy into CI/CD the book “Continuous Delivery” by Jez Humble is the defacto standard. There are other books specifically about programming style like “The Pragmatic Programmer” from Andrew Hunt.

Being able to practice all this new learned information and learn from peers is as equally important, get involved with the community! You have great meetups in the UK like DevOps Exchange London, London DevOps, DevOps Manchester, DevOps Cardiff, Leeds DevOps, Infracoders London, London Continuous Delivery and London Web Performance. There are also tons of user groups around specific tools like Puppet, Chef, Docker, OpenStack, AWS.

Fix ldap_route `null’ in sendmail 8.14.4

I’ve got a sendmail setup with ldap_routing, it’s very convenient if you’ve got a distributed sendmail environment, in my case I’ve just got ldap_routing for mail hosts and not for addresses, so it’s expressed in the following form in sendmail.mc:

When upgrading my sendmail platform to the new Ubuntu 12.04 LTS (Precise Pangolin) I’ve found the following error:

This is due to a change in behaviour in ldap_routing.m4 in 8.14.4, it’ll try to automatically add  -T<TMPF> which breaks the special `null’ behaviour.

The way recommended to fix this is to replace ldap_routing.m4 with the version from 8.14.3 which is available here.

In my case (Ubuntu) I just had to replace the file located at /usr/share/sendmail/cf/feature/ldap_routing.m4, then process sendmail.mc again and everything went back to normal :)

MySQL upgrade to Ubuntu 12.04

Ubuntu 12.04 LTS (Precise Pangolin) has updated MySQL to version 5.5, the update is not as straight forward as in other releases so some caution must be always taken.

Updating from MySQL 5.x

This is a fairly easy case, if you have any extra config in /etc/mysql/conf.d there’s a high chance that the new package will actually uninstall your old packages without replacing them, be extremely careful with that, also check that all your parameters are in line with MySQL 5.5 syntax.

First of all once the upgrade to 12.04 is finished, check which packages for mysql-server are installed:

If you see all the 5.5 packages installed congratulations, your upgrade was flawless, but in any other case you’ll only see the mysql-server-5.1 package, so you’ll need to install manually the packages.

This should in all cases suffice to get MySQL server running again if there’s no errors in your my.cnf.

Updating from MySQL 4.x

In this case the binary structure changes slightly so you’ll need to dump all your data and upload it to a fresh new MySQL 5.5 instance, there’s not much way around this unfortunately and not following this can result in corrupt data.