The DevOps movement

I’m quite intrigued by the new devops movement that has been arising lately in the ops community.

A devops is basically a sysadmin with a deep knowledge of several languages and in touch with the code running over the platform he’s providing, normally related to new trends in systems administration as the ones used by Facebook, Google, Twitter, etc.

I can’t say I can’t feel identified with this movement since this is what I’ve been advocating for the last years, I’m quite happy to see that it has finally got enough intertia to develop into a full fledged movement that will take the sysadmin field into a new era, I’m completely giddy with excitement.

If you want to read more about what devops is about I think these are some of the most interesting blogs to follow:
http://www.planetdevops.net/
http://londondevops.org/
http://vuksan.com/blog/

Also there’s a couple Google Groups, devops-toolchain and agile-system-administration

Configuring samba server in Mac OS X Leopard

If you don’t have Mac OS X Leopard server you have a Samba implementation limited mostly to home directories and a lot of borking around, if you’re a typical Unix Admin as I am you’ll want to take things in your hands and add the shares you want yourself in the command line.

Leopard uses Samba 3 and its own authentication and locking methods connected to its auth layer and afp locking so a typical samba config file won’t work, it also has a dynamically modificable part which is configured via System Preferences.

This is not the smartest method neither prepared for faint hearted people, but it’ll work if you’re used to Linux.

If you had samba already working on Tiger the changes are only at locking, vfs and user auth, which is what enables all the new Leopard system to work properly.

This are the exact changes from Tiger to Leopard

You want to take a look at the realm SHA1 strings since they’re dependant on your installation, you can always check the new /etc/smb.conf in Mac OS X Leopard and then merge it up with your previous config, or replace the config as I did and just add this.

The Leopard samba configuration is brilliant, but at the same time limited to push you to buy the Server version, at the same time it’s interesting to play with the includes it adds too, but this at least will get your previous samba config out and running fine.

Optimize network throughput on your Apple TV

This is specially helpful if you are using your Apple TV wireless adapter instead of plugging it directly to the network with an Ethernet cable.

This small tweak assumes that you already have sshd access to your Apple TV, if not please check out AwkwardTV on how to do that.

I had some problems watching HD DivX files in my AppleTV, they were skipping frames and freezing in the most awkward moments, I have all of my files in a mac mini and shared on the network using AFP. The files are mounted in the Apple TV using the ATVFiles plugin and the aTV-ShareMounter plugin.

What needs to be added to give the network buffer enough buffer space on the network is tweak the kernel options at startup, that can be done as in Mac OS X editing the file /etc/sysctl.conf which in the Apple TV version of Mac OS X does not exist by default, you’ll have to create it, don’t worry I’ll put here two quick ways to do it.

How to edit /etc/sysctl.conf (if it does not exist)

First Option – The fast kamikaze strategy (will work for everybody, but be really careful!).

Second Option – The slow wise monk strategy (this will work in all cases)

Edit a sysctl.conf in your computer (just be sure that if you’re in a Posix system you’re not editing your local /etc/sysctl.conf).

Add these values to the file

Copy it to your Apple TV through scp and make sure it ends up in its correct position at /etc/sysctl.conf on your Apple TV

The changes you do will be available on the next reboot, I find it better to do it this way so you’re 100% sure that all the changes are loaded at the same time.

I hope this solves Leo Laporte streaming problems to his Apple TV ;)

Trunking between a Cisco Catalyst and a 3Com SuperStack

Trunking between this equipments is problematic at best, the meaning of trunk in the 3com is not the same as in the Catalyst, also the vlan methods are not the same either.

Trunk in the 3Com SuperStack is port aggregation between two 3Com devices, whether in the Cisco is really a downlink trunk, luckily both devices speak 802.1q so the trunk configuration shouldn’t be a big problem.

First of all we need to establish the trunk port between the Catalyst and the SuperStack, so we’ll start by defining the port in the Catalyst.

Some considerations on this config. It’s always recommended by Cisco and security-wise to use another vlan than vlan 1 for trunking, that’s what we’re doing here, also we’re restricting which vlans we will accept and retransmited to the 3Com switch.

There’s a huge implementation difference between the trunking trunking transmission between Cisco and 3Com, the 3Com switches tag all the vlans by default, but the Cisco switch won’t tag the trunk vlan, this is a really annoying factor that made me waste some hours!

The trick resides in adding all the vlans tagged into the port that you’re using as a trunk, you don’t really need to add the trunking vlan that you configured back on the cisco, it doesn’t work that way. So let’s add one by one all the vlans in the trunk port. In order to do that we need to use the bridge menu in the 3com switch.

Repeat this in the trunk port for each vlan you’re adding in the Cisco trunk side. When you have your trunk port configured properly (also be careful with duplex and speed configs) you just need to add the ports into the vlan untagged. So let’s say we want to add port 1 to the vlan 10.

As soon as that’s done the port will be talking head to head with all the other ports in vlan 10 also in the Cisco switch.

The difficult thing is making the 3Com switch accesible through an IP address, since the 3Com switch will only publish its public IP address though VLAN 1, this one is a though cookie.

IBM xServer 305 on CentOS 4 – Kernel incompatibilities

After a horrible week fighting against CentOS 4 and our firewalls deployment. I’ve found several incompatibilities (albeit already documented) with this machines and CentOS 4.

There were 2 different problems affecting this machines.

Keyboard not working after kernel boot, non responsive.

This happens even when installing the machine, I had to install this machine in graphical mode (it works when booting in graphical mode) but it didn’t work at all on a forced text installation.

This bug can be solved disconnecting the USB subsystem in the BIOS, after that the keyboard is operative again.

Machine freezes with a kernel panic after approx 24h. of operation, no logs or traces left.

This one was a hard bone to catch! Did a memory test, updates both BIOS and network cards firmware to the last version and did several extended checks on the hardware, there was not a single error.

After roaming around the RedHat and CentOS forums for a while looking for an answer, I saw a similar error in RedHat RHEL4 (the one CentOS is based on) about an xServer that had the same problem. It seems the problem is realted with the old version of ACPI this motherboard has, and it only happens with 2.6.9-42.x.EL versions of the kernel, just adding the noapic option in /etc/grub.conf to the kernel boot solved the issue.

Pushing the limits of Expose

After this interesting article in Digg I decided out of boredom to try this up myself, here’s the result.

expose_limits

After this I noticed that the concurrent file limit by default in Mac OS X is quite low! Only around 200 processes per user, which even if it’s more than enough for most users sometimes depending on what you do you need more ;)

Mac OS X respects the sysctl.conf kernel parameter file as most unix do, so it’s just a question of adding the following parameter there:

kern.maxproc=2048kern.maxprocperuid=512kern.sysv.shmmax=167772160kern.sysv.shmmin=1kern.sysv.shmmni=32kern.sysv.shmseg=8kern.sysv.shmall=65536

I noticed after raising the processes value that the shared memory values where low as well, so it’s a good option to change them all at the same time.

Of course this configuration is not really recommended for machines with less than 1Gb of RAM, so be careful!

Also you need to raise the limit on launchd, launchd is Apple replacement for initd and it manages the initial limits for users.

Just edit (or create /etc/launchd.conf) and add:

limit maxproc 1024 2048limit maxfiles 2048 4096

SpamAssassin : Tweaks for new Spam methods

There have been lately a huge increase in spam due mainly to botnets, spammers also have shifted their spam methods, using embedded images and obfuscating techniques to avoid OCRs.

This two factors together mean that I’ve got a lot more work maintaining my spamassassin installation :) and also that the standard config or some deviations on the scoring is not good enough, even with score tweaking I still got lots of Stocks and embedded gif spam, after some checking around I found some solutions in Rules Emporium. Also updating is a must so try always to keep up to date, right now I’m running Spam Assassin 3.1.7.
Finally after some tweaking and more tweaking I arrived to this config:

  • Auto White-list and Bayes using MySQL DB Engine
  • user_pref integrated into our user control panel
  • Razor2 integration
  • SPF Integration
  • Score tweaking
  • New rules added using Rules Emporium ImageInfo and Stock Rules

With this method the false positives have gone down and the stock and image spam is being stopped (finally!).

The Rules Emporium ImageInfo plugin consumes a lot less CPU than using an OCR plug-in and even if it’s based on broader rules it catches even the hardest embedded image spam, you can get the plugin here. Also the stock ruleset got rid of most of the stock spam that I was receiving, this spam is quite hard to guess indeed! You can get the ruleset here.
Here is the final tweaked local.cf config in SpamAssassin

Also it’s important to have this modules loaded in your v310.pre file:

The way to install the additional config and plugins should be as follows:

Copy the new .cf (configuration) files into the directory where SpamAssassin keeps the configuration in your install. In Red Hat machines this directory is /usr/share/spamassassin.

Copy the new .pm (modules) into the SpamAssassin PlugIn module which is by default /usr/lib/perl5/vendor_perl/5.8.3/Mail/SpamAssassin/Plugin/ (this directory of course, is for Perl version 5.8.3, change the version to the one you have installed).

Don’t forget to restart SpamAssassin after adding the new files!

It’s always a good idea to start spamd with -D after activating modules, since most of the times you’re missing a perl module which one of your modules have a dependency with.

This configuration is not really CPU hungry so it’s great for people who are running on a tight server budget.