I had my first encounters with OpenStack this week, attempting to setup two different distros of the popular AWS-in-your-datacentre stack. It was variously interesting, confusing, and bruising, and while it’s still early days there is one thing that’s already clearly separating the two distros I looked at – documentation.
So to start at the beginning, let’s talk about at the first distro I tried: StackOps.
A customer asked me to build a small, pilot OpenStack system running StackOps for them, and that’s when the infamous words “sure, how hard can it be?” crossed my mind. I had a look at the documentation at the start of the week, and immediately realised it was not going to be as straight forward as I’d imagined!
I spent the train ride over to the customer pouring over the documentation, trying to work through it all. It was not easy. I found myself bouncing all over the site searching for answers to questions raised as I read through the install procedures. During this I encountered blank pages, incomplete reference files, missing reference architectures… needless to say it wasn’t ideal! While it may not be fair to cherry pick a few particularly bad pages to link to, overall I found the structure to be rather disorienting and the content hard to follow.
Still, my colleague and I had the head node up and running well before lunch and then we tried for a compute node. After PXE booting, it simply shut down. No output, no messages, and nothing visible in the head node’s management tools. After videoing the PXE boot we spotted a kernel error which warned of BIOS problems. I reported this to StackOps and then spent the next few hours chasing down kernel errors and patches and BIOS updates, all to no avail.
So we called it a day and no sooner had the train home pulled away from the platform than an email from StackOps pops onto my phone – “have you activated your license?”. In all my bouncing around the documentation site looking for technical information, I’d completely overlooked the page on activating the license… As you can imagine I was rather annoyed with myself!
So when I got home I built an ESXi server out of my PC and had another go at setting up StackOps, and while waiting for the trial license to come through I took another crack at the documentation.
As the license didn’t come through, I had a go with the license-less StackOps Community Edition on Saturday morning. A day later and I’ve got it partly working.
The controller node seems to be stable, however the compute node refused to launch any instances. I started chasing down errors in its log files and during the course of trying to fix one of these rebooted. Its configuration doesn’t appear to have survived that reboot, as it now won’t even start the OpenStack ‘nova-compute’ service. A short while after this I decided StackOps had had enough of my time and moved on.
Clearly StackOps and I weren’t getting along very well and I was missing something key, so I switched tracks and tried Red Hat’s distro instead.
Red Hat OpenStack
The difference was clear straight away, Red Hat’s documentation was much, much more usable! It does have some issues, like pointing to non-existent arguments for the setup commands, however after just one pass through it I was close to a working system. After a couple of stalled attempts as I worked out the correct values for various variables in my test environment, it was up and running with a functioning instance launched.
So far I think Red Hat’s approach to the installation and setup of OpenStack is the better one.
Their “packstack” tool utilises Puppet to configure all the nodes and services, which I can see providing an avenue for integrating the management of OpenStack configuration items into existing data centre automation and configuration management tools. It does require more button pressing to setup each node than StackOps’ PXE booting approach, however I think this also gives it more flexibility and thus makes in more robust. And of course, most of that button pressing could be eliminated with tools like Cobbler, again integrating with data centre toolsets that are already in place.
StackOps’ ‘Smart Installer’ web app is pretty neat. It was nice to use and did include a lot of helpful commentary on the various variable that needed to be set, however overall I’m not really happy with it as a solution. While it does make the first time setup a little easier than packstack, it seems to be a world unto itself. I can’t see it integrating at all really into any existing toolsets for ongoing configuration management. And the fact that you have to enter all the details of your internal environment into StackOps’ website is bound to be a concern for any Information Security person who sees it.
Also of note is one big bug in the current Red Hat preview release. Initially, I couldn’t get any instances to launch due to a permissions issue on the compute node I’d setup. It turns out that this is a bug in libvirt which causes it not to re-label the virtual machine files correctly.
The incidence of this bug is apparently conditional on certain interactions with KVM control groups, and I spotted that the control group service was stopped by default. Starting it up with the following command resolved the issue and enabled me to launch OpenStack instances;
service cgconfig start
I haven’t played much with my OpenStacks yet, in large part because it’s very slow on my setup – turns out running qemu inside a virtual machine on desktop hardware is pretty slow! Next time I’ll have to take the time to implement this hardware virtualisation pass-through method.
However from what I have seen already, I’m beginning to better understand the private and hybrid cloud idea OpenStack represents…
So that’s all for now, and as always, thanks for reading :)