Puppet Camp London 2013

Today I attended Puppet Camp London at Mary Ward House. As with the previous London Puppet Camp in March, attendance was very good with something for everyone who’s using Puppet.

Here’s a quick link to the presentations and/or speaker profiles:

  • Puppet Enterprise Demo – Puppet Labs

Update 8/Dec/2013: there’s now a blog post on the Puppet Labs website about the event – Mind the Gap at Puppet Camp London so I’ve added in the video presentations in this article.


Fedora Kickstart Ultra Minimal KDE Installation

Issuing a “yum groups list” (or “yum grouplist” or “yum group list”) on Fedora 19 shows that “KDE Plasma Workspaces” is one of the available environment groups that can be chosen to get a KDE desktop up and running.  However, issuing a “yum grouplist hidden -v” shows this and other KDE groups are available:

KDE Plasma Workspaces (kde-desktop-environment)
KDE Applications (kde-apps)
KDE Educational applications (kde-education)
KDE Multimedia support (kde-media)
KDE Office (kde-office)
KDE Plasma Workspaces (kde-desktop)
KDE Software Development (kde-software-development)
KDE Telepathy (kde-telepathy)

So, if you want to have the full KDE stack on Fedora, it’s relatively easy to pick these groups and include them in your kickstart file.  However, I was after a relatively minimal KDE desktop setup, so I wanted to see what was installed by default and whether we could remove and groups from this selection.

If we take a look at the “KDE Plasma Workspaces” base installation via “yum groupinfo kde-desktop-environment” command we see that it will attempt to install the “+kde-desktop” group, along with groups such as printing, dial-up and multimedia. What if you don’t have a printer hooked up to your workstation, or require dial-up support?  Similarly looking at the “+kde-desktop” group via “yum groupinfo kde-desktop” we see that other groups will be added such as kdegames-minimal, kde-print-manager, kwrite.  I’m not suggesting that these will be unwanted by everyone, just that I don’t necessarily require all of these recommended defaults.

So, how can we create our own custom package selection list to get a minimal KDE desktop up and running?  To start with, we’ll need an X environment and some fonts.  I have no requirement to display Asian or Arabic fonts so I started with the basic X display and fonts and then removed the unwanted ones:

# Start with the fonts group, we then remove a number of fonts
# that we don't need
# Remove unwanted fonts
# Install X11 stack

We can now install KDE and some other useful utilities.  We begin with group “critical-path-kde” since this is an absolute minimum.  We then add the KDE parts we want:

# Start with the KDE absolute minimal
# Then add useful KDE and X Components

We’ll continue building up our Ultimate Fedora kickstart file in the next article, but for now we’ve got enough to a minimum KDE desktop installed by including the above in the %packages section.


Fedora Kickstart DNS Dependencies

In the previous post we added the updates and fedora repositories to our kickstart file. That should mean the packages are pulled down from the Internet if they can’t be found on the install media (those in the “url” statement).

When I first tried this on my home installation, I was still getting the errors about being unable to find some software packages. A look on the install consoles (ALT+F1,F2,F3, etc.) showed the repositories were being disabled for fedora and updates. Initially I couldn’t figure out why. Then it dawned on me. I was using a temporary DHCP server on a Centos 6 provisioning host, and my options were as follows:

option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16; # RFC4578
deny unknown-clients;
subnet netmask {
option routers;

Note that my DHCP server wasn’t returning any DNS servers as my client booted. As such, when the Fedora install started it had no concept of DNS servers to use.  So, I added the line below so that my Fedora install would use the Google public DNS servers (you could replace this with your own DNS servers or that of your ISP):

option domain-name-servers;

The installer was now able to use the fedora and updates repositories to pull down the required packages.

Lesson learnt: make sure that you supply a valid DNS server on your clients when using the kickstart file.


Fedora Kickstart Installation Sources

The previous post showed the kickstart file generated using a minimal installation on my Thinkpad W530.  It’s this base kickstart file which we’ll update and customise, in much the same way as we would do if we were working on the target machine.

I typically install the following packages via yum:


Apart from conky and simple-mtpfs all of those applications are fairly generic.  As such I was hoping that they would be available on the Fedora installation DVD.  So, I first updated the packages section of the kickstart file like this:


However, this kickstart file resulted in errors stating that the packages could not be found.

On an installed Fedora 19 system, I could see these packages came from either the “updates” repository or from a repository called “fedora”.

# yum list sysstat conky autofs simple-mtpfs
autofs.x86_64        1:5.0.7-28.fc19                @updates
conky.x86_64         1.9.0-4.20121101gitbfaa84.fc19 @fedora
simple-mtpfs.x86_64  0.1-6.fc19                     @fedora
sysstat.x86_64       10.1.5-1.fc19                  @fedora

I then came across the following links:

Anaconda/Kickstart – repo usage

Red Hat Bugzilla 979154 – Fedora 19 RC2 kickstart with “repo –name=fedora” crashes

Fedora 19 Common Bugs – Problems with Installation Source and Installation Destination spokes when installing from a partially complete kickstart

This first link states that “By default, anaconda has a configured set of repos taken from /etc/anaconda.repos.d plus a special Installation Repo in the case of a media install. The exact set of repos in this directory changes from release to release and cannot be listed here. There will likely always be a repo named “updates”.

I had another look on the DVD and sure enough those packages were not listed. So, what I actually needed to do was enable these extra repositories in the kickstart file.

Here’s what the updated sections of the kickstart file will look like:

# Use network installation
url --url=""
repo --name=fedora-kickstart --baseurl=
# Need fedora so we can pull down things like sysstat
repo --name=fedora
# Use this to get full updates
repo --name=updates

The updated kickstart file will cause the installer to use these extra repositories and use them when it gets to the %packages section of the kickstart file.  In this manner, it will also pull down updates to the O/S from the Internet using the “updates” repository.


Ultimate Fedora Kickstart

I recently decided to re-install Fedora19 on my Thinkpad W530. I thought it would be worthwhile documenting the steps and using a kickstart server (in this case running Centos 6) to be able to replicate the build in the future – for example when Fedora 20 is released – and for kickstarting other devices. Sure, it’s now possible to upgrade Fedora between releases using FedUp, but if all of your personal data is on a separate (backed up) partition then a clean, custom install will give you a fresh start and make sure no old configuration files or packages are left behind.  If you document your customisations via a kickstart file, it means the headaches of the re-install can be minimal.  In fact, everything you would do on the command line post-install can be done via a kickstart file.  Another advantage is that should a newer filesystem type come along you can simply reformat your O/S partition to this new type.

The next couple of posts will document some of the steps in creating the kickstart file and will cover:

Fedora Kickstart – Installation Sources
Fedora Kickstart – DNS Dependences
Fedora Kickstart – Ultra Minimal KDE Installation
Fedora Kickstart – Additional Repositories
Fedora Kickstart – Thinkpad W530 add-ons
Fedora Kickstart – Post-Installation Tasks

To begin this process, I first installed Fedora 19 by hand and chose minimal as the default installation. This gave me a /root/anaconda-ks.cfg kickstart file from which we can work.

It will look something like this:

# System authorization information
auth --enableshadow --passalgo=sha512
# Use network installation
url --url=""
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
# old format: keyboard us
# new format:
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_GB.UTF-8
# Network information
network --bootproto=dhcp --device=eth0 --noipv6 --activate
network --hostname=localhost.localdomain
# Root password
rootpw --iscrypted XXX
# System timezone
timezone Europe/London
user --groups=wheel --homedir=/home/user --name=user --password=XXX "User"
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
# Partition clearing information
clearpart --none --initlabel
# Disk partitioning information
part /boot --fstype="ext4" --onpart=sda6 --label=fedora19-boot
part / --fstype="ext4" --onpart=sda7 --label=fedora19-root
part swap --fstype="swap" --noformat --onpart=sda8
part /boot/efi --fstype="efi" --noformat --onpart=sda2 --fsoptions="umask=0077,shortname=winnt"

For clarity, the ‘url’ shown above would refer to an internal apache webserver from which the Fedora DVD is shared.

Whilst the final kickstart file won’t be for everyone, I’ve called it the ‘Ultimate Fedora Kickstart‘ because it’s the ultimate for my needs. No doubt, you’ll have your own version of this :)


Using flock to ensure only one instance of a script can run

Whilst browsing, I came across the following post from Randall Schwartz of Perl and FLOSS Weekly fame.

“flock” is one of those utilities I’ve not used very much, but if you want to create a script and ensure that only a single instance of it can run at any one time then this is a really neat utility. No lock or PID files to mess with, no “ps -ef | grep” type of scripting to incorporate.

if ! flock -n -x 200
echo "$$ cannot get flock"
exit 0
echo "$$ start"
sleep 10 # real work would be here
echo "$$ end"
) 200< $0

One to file away for future use :)


Puppet Camp 2013

Yesterday I attended Puppet Camp London 2013 at Somerset House. It was an interesting day with a lot of good talks and demonstrations.  In this article, I’ll attempt to link to all of the speakers and slides from the event and describe what I found interesting.  The day was sponsored by Red Hat and Quru.

The began with Dawn Foster, Community Manager at Puppet Labs, introducing Puppet Labs CEO Luke Kanies.

  • State of Puppet: Luke Kanies – Puppet Labs CEO

State of Puppet detailed the history behind the creation of puppet, how things started and where they are now. It was apparent from the slides that there has been a large growth in puppet deployments, community and modules over the last 12 months. I especially enjoyed the point that the ‘old’ ways of doing upgrades – eg taking down services for a migration on a Friday evening, performing the required steps, and then starting things up again on Monday – just don’t work in today’s environment. We’re used to having IT available at all times – we want to access Internet Banking when we want to. We expect access to news, blogs and entertainment 24 hours a day. And we’re more likely to be running services that are available internationally, so the traditional ‘maintenance window’ is no more. Another important fact was that when puppet was created, there wasn’t much cloud deployment. Nowadays, it’s everywhere and having a tool like puppet to manage these instances is very useful. We even have VM’s being created and destroyed dynamically for just a single HTTP request. With Puppet, we can basically keep everything in ‘sync’ using a standard programming syntax rather than custom scripts. Luke explained that Puppet Labs began with an Open Source product, and made money by providing consultancy services to set this up. Nowadays, they’re keeping some of the features hidden away their Enterprise products. There’s nothing wrong with this, I just hope that the Open Source version with features that might overlap with Enterprise, such as Puppet Dashboard, don’t fall by the wayside. Other items mentioned in the presentation include Puppet DB (which tracks the status and changes in your environment in a database) and plans for more configuration tools to push configurations to servers at specific times or under controlled conditions. There was also talk about the ability to add machine dependencies within Puppet, eg provision a database, but don’t start the webserver that talks to it until the database host has been fully provisioned. In terms of user base, Puppet has lots of clients including Barclays, FT and LSE in London, and Google, Cisco and HBO in the US. Plus many more. The size of deployments varies too, from managing just a few servers to managing tens of thousands.

The slides from the Luke’s talk can be found here: State of Puppet – London. Readers may also be interested in Chris Spence State Of Puppet slides featured on the Puppet Camp Barcelona Wrap Up blog post or the slides from the San Francisco Puppet Camp – State of Puppet – San Francisco 2013.

  • Building reusable modules: Jon Topper – Scale Factory

All of the talks were interesting, but this is the one where I can start to reap immediate rewards. Firstly, it provided good ways of writing puppet modules, and there are definitely good take-aways from this such as writing puppet modules that perform very small, discrete pieces of work. Dependencies between puppet classes is also a bad idea. RSPEC Puppet, puppet parser and Puppet Lint are great tools for checking your code, although it was pointed out that puppet-lint can be very, very picky, so use with appropriate settings that work for you.

You can find more about Scale Factory from their website, whilst the slides from Jon’s presentation can be found here – Building Reusable Puppet Modules.

Jon’s Twitter profile is jtopper.

  • Automated OS and Application deployment using Razor and Puppet: Jonas Rosland – EMC

The slides that Jonas presented can be found at Puppet Camp London 2013 Puppet And Razor Jonas Rosland.

Razor is a provisioning system that can be used quickly provision new servers – both physical and virtual. The key thing is that it’s event driven rather than user driven. In the demo, Jonas configured Razor to provision certain types of servers depending on certain conditions. The example used physical RAM to determine what type of Operating System should be installed when a server is PXE booted, but you can use it on any kind of variables that you get from factor. I’m not sure how this would work in remote sites where you don’t have a PXE server. The install of Razor looks very straightforward.

Other tools worth looking at are: The Foreman, Cobbler, vSphere Auto Deploy

Jonas has some useful links on his pureVirtual website: Puppet and Razor.

Jonas’s Twitter profile is virtualswede

  • De-centralise and Conquer: Masterless Puppet in a dynamic environment: Sam Bashton – Bashton Ltd.

The slides that Sam presented can be found at Decentralise And Conquer Masterless Puppet In A Dynamic Environment.

This was a really interesting presentation. Essentially, Sam was building a set of RPM’s which can then be deployed to the target servers via Pulp. Puppet then runs locally on the remote target, triggered from a postinstall command in the RPM package. There’s no central puppetmaster in this setup, so no single point of failure.

Sam’s Twitter profile is bashtoni

  • Building self-service on demand infrastructure with Puppet and VMware: Cody Herriges – Puppet Labs

Cody talked about the pros and cons about running your own infrastructure versus using hosted solutions such as Amazon. His slides can be found here – Building self-service on demand infrastructure with Puppet and VMware

  • Enterprise Cloud Management and Automation: John Hardy – Red Hat

John presented ManageIQ. This clever piece of software interrogates your SAN arrays and discovers the Virtual Machines that are installed there. It can then look into these machines to determine what’s running, what files are installed, record changes on these files and perform full inventory control. It can even prevent a VM from being powered on if it violates a policy, such as not being an approved O/S. ManageIQ is being used by UBS and other big organisations. Red Hat acquired ManageIQ in December 2012, so expect to see this rolled into Red Hat products soon. Hopefully, much of it will become open source too.

  • Puppet Demos: Chris Spence – Puppet Labs

There was no slideshow from Chris, it was a hands-on demo showing how Hiera can simplify puppet code, how configuration files (such as a load balancer) can be dynamically generated as servers are powered up and powered down, and he showed some useful Puppet 3.0 commands.

Chris has written some puppet modules which can be found on Puppet Forge and has some useful material on his blog.

Chris’s Twitter profile is tophlammiepie

  • Closing thoughts

Overall, it was a good set of talks and great to talk other puppet users to discover how they are using it. I’ll certainly be using Hiera for deployments and I’m going to start using tests for my modules. In terms of contact with the Puppet community, I’ll definitely make use of and puppet-users.

Finally, here’s a link to the official Puppet Camp London 2013 blog – Fun Times and Great Info at Puppet Camp London

Oh yes, and thanks for the post-camp drinks, T-Shirt and Hat! I look forward to Puppet Camp London 2014!

Red Hat Puppet

Red Hat Puppet


Display a future or past date in the bash shell

Here’s a quick and easy way to establish what the date will be in a specific number of days from today using the bash shell on Linux. Simply use the ‘-d’ option to the ‘date’ command.

Here’s the current timestamp:

-bash-3.2$ date
Thu Jan 17 15:04:28 GMT 2013

And this will be the date 60 days from now:

-bash-3.2$ date -d "now 60 days"
Mon Mar 18 15:04:31 GMT 2013

You can also use the same code to display dates from past. What was the date 94 days ago?

-bash-3.2$ date -d "now -94 days"
Mon Oct 15 15:07:35 GMT 2012

Thinkpad W530, Red Hat Enterprise Linux 6, Fedora and Windows 8 Multiboot

Now that we’ve successfully done a clean Windows 8 install on the W530 and got it dual booting with Fedora 17, it’s now time to add another distribution onto the laptop – Red Hat Enterprise Linux 6.

My first attempts to install RHEL 6.3 onto the W530 resulted in the graphics failing to load by the installer. This resulted in the screen displaying a strobing set of psychedelic colours. A few Red Hat Knowledge-base articles which might be relevant:

RHEL6 does not boot on Lenovo W520 Laptop with Discrete option selected to choose nVidia GPU

Blank screen during installation when using certain NVIDIA Quadro Graphics Adapters under Red Hat Enterprise Linux 6

Why won’t the Nvidia driver compile/install/load under Red Hat Enterprise Linux 6

I don’t recall exactly what settings I initially tried in the BIOS for display type which offers the following options:

  • “Integrated” – uses built-in Intel Integrated Graphics Controller”
  • “Discrete” – uses nVidia Graphics
  • “nVidia Optimus” – uses the built-in Intel Integrated Graphics Controller but allows the OS to use nVidia when needed (supported only with Windows 7 and Window 8)

Anyhow, I attempted to install with the following options on the kernel command line:

xdriver=vesa nomodeset

From what I can tell, that should have allowed the installer’s X Server to start successfully, but it did not. However, the installer helpfully told me that I could use a VNC client to perform the Red Hat install.

I told the installer to select /boot/efi as the EFI install partition, the one shared with Windows 8 and Fedora.

After the install, I was given the option to start Windows 8 or Red Hat Linux. The Fedora choice was no longer listed. Fortunately, there was a backup /boot/efi/EFI/redhat/grub.conf.rpmsave which contained the Fedora/Windows 8 option. It’s now just a case of merging the Red Hat and Fedora 8 grub files.

Here’s the result:

device (hd0,5) HD(2,96800,32000,ad8e8d71-db62-4c7a-8603-5bc6ce875d52)
 title Fedora (3.6.11-1.fc17.x86_64)
  root (hd0,5)
  kernel /vmlinuz-3.6.11-1.fc17.x86_64 rd.lvm=0 KEYTABLE=us SYSFONT=True rd.luks=0 root=UUID=a6f32b89-45ac-410f-8a1e-562b441304e3 ro LANG=en_US.UTF-8 rhgb quiet
  initrd /initramfs-3.6.11-1.fc17.x86_64.img
 title Windows 8
  set root=(hd0,gpt1)
  chainloader /EFI/Microsoft/Boot/bootmgfw.efi
 title Red Hat Enterprise Linux (2.6.32-279.el6.x86_64)
  root (hd0,8)
  kernel /vmlinuz-2.6.32-279.el6.x86_64 ro root=UUID=fe96af7b-07bd-451e-b4de-4eec673f4cca nomodeset rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet xdriver=vesa
  initrd /initramfs-2.6.32-279.el6.x86_64.img
 title Windows 8
  rootnoverify (hd0,3)
  chainloader +1
title Fedora
  rootnoverify (hd0,6)
  chainloader +1

Thinkpad W530 Windows 8 And Fedora Dual Boot

So after our Windows 8 clean install on Thinkpad W530 we had our W530 booted up and running Windows 8 in Secure Boot mode. I’ll do an in-depth write up about setting up a boot server to serve out Fedora, Red Hat Enterprise Linux and Centos in both UEFI and BIOS, 32-bit and 64-bit flavours later. But for now, let’s assume we have Fedora 17 64-bit, UEFI install server.

I normally hit F12 on the W530 to bring up the boot menu and select the network card as the bootable device. If you try to boot from the Fedora install media with Secure Boot enabled, you’ll get the error shown below: “Secure Boot: Image failed to verify with *ACCESS DENIED*.  Press any key to continue”.

Windows Secure Boot Access Denied Screenshot

So, first thing to do is to disable Secure Boot via the BIOS.

The disabling of Secure Boot shouldn’t be necessary once Fedora 18 is released, since it will contain a first-stage bootloader, shim, which holds a Fedora-specific public key. Shim will then validate against the Fedora-defined key which has been signed by Microsoft. More details here.

So, for now we move onwards with dual booting Windows 8 and Fedora 17 without Secure Boot. Firstly, it’s worth pointing out that for the Fedora install you don’t need change any BIOS settings relating to graphics options. I used “Nvidia Optimus” (runs with integrated graphics but with discrete graphics on demand” with “O/S Detection Enabled”. Follow the usual install process with the following notes: in my install I chose to create a 400MB filesystem for /boot (on ext3), 8GB for swap and 20GB for the / filesystem (ext4). I selected the second partition (/dev/sda2) as /boot/efi and was sure not to check “format”. I then chose to install GRUB into /boot/efi.

My partition table looks similar to this:
Number Start (sector) End (sector) Size Code Name
1      2048    616447 300.0 MiB 2700 Basic data partition
2    616448    821247 100.0 MiB EF00 EFI System Partition
3    821248   1083391 128.0 MiB 0C01 Microsoft reserved part
4   1083392 116426751  55.0 GiB 0700 Basic data partition
5 116426752 200312831  40.0 GiB 0700 Microsoft basic data
6 200312832 201132031 400.0 MiB 0700 Linux filesystem
7 201132032 243075071  20.0 GiB 0700 Linux filesystem
8 243075072 259852287   8.0 GiB 8200 Linux swap

Once the install has completed, the machine will boot and present you with a grub menu from which you can chose Windows 8 or Fedora.
The grub configuration can be seen Number in /efi/EFI/redhat /grub.conf and will probably look like this:

device (hd0,5) HD(2,96800,32000,ad8e8xxxx-db62-4c7a-8603-5xxxxxxx)
 title Fedora (3.6.8-2.fc17.x86_64)
  root (hd0,5)
  kernel /vmlinuz-3.6.8-2.fc17.x86_64 rd.lvm=0 KEYTABLE=us SYSFONT=True rd.luks=0 root=UUID=a6f32b89-45ac-410f-8a1e-562b441304e3 ro LANG=en_US.UTF-8 rhgb quiet
  initrd /initramfs-3.6.8-2.fc17.x86_64.img
 title Windows 8
  set root=(hd0,gpt1)
  chainloader /EFI/Microsoft/Boot/bootmgfw.efi

It’s worth pointing out that I tried the Fedora 18 beta install on the W530. Unfortunately, the installer is still a little bit unstable and crashed out on me when attempting to partition the drive. Fedora 17 is very usable and right now I don’t need any of the new Fedora 18 features, so no big loss.

One final thing I like to do is mount up the FAT32 Windows D drive under /Windows/D – this enables me to transfer files between Windows and Linux as needed.

The following line in /etc/fstab will do the job, after creating /Windows/D under Fedora:

/dev/sda5       /Windows/D      vfat    defaults 0 0

Now that the Fedora install is complete and we can easily switch between Windows 8 and Fedora 17, we can customise our Fedora install.  However, before we do that we’re going to install Red Hat Linux Enterprise and add that as a third boot option.