CentOS Dojo, Brussels 2016

On Friday I attended my first Centos Dojo in Brussels , Belgium held one day before FOSDEM. The timing fits in well for the Open Source community since you can attend this, FOSDEM and Configuration Management Camp as part of one trip. The official Centos events page will have links to the videos and slides, but I wanted to share these notes that I made from the day.

What is a dojo? The Centos Dojo website states that “The CentOS Dojo’s are a one day event, organised around the world, that bring together people from the CentOS Communities to talk about systems administration, best practises in linux centric activities and emerging technologies” and it certainly lived up to these expectations. The good: each presentation was about an hour long, the speakers knew their topics, many of the sessions were interactive allowing for questions and discussions.

The day began with Karanbir Singh introducing State of the CentOS Project. During the Q&A, a couple of interesting things were raised. Firstly, someone had issues with the graphical updater failing after an update to Centos 7.2. Karanbir suggested that many things were re-based (ie underwent a major upgrade) in RHEL / Centos 7.2 including many of the Gnome packages. It sounded as though these types of major update may become more commonplace in the future. On the one side people want reliability and stability, but at the same time they want newer functionality, so things like a graphical updater breaking may be an occasional casualty. The advice is raise bugs where ever possible.

There was another question around Centos providing an Extended Update Support (EUS). Essentially, this boiled down to asking Red Hat to make those EUS updates available as source RPMs. If they do that, then Centos could package these. There are 4 components at play in this discussion: Copyright, License, Contract and Intellectual property. Red Hat are one of the better companies for publishing Open Source code, but the EUS packages are not one of them and anyone using them has probably agreed in their contract not to redistribute them. As yourself this: do other Operating Systems such as SuSE and Ubuntu provide their sources in an easy-to-consume manner?

Why is a package in RHEL but not in Centos? Answer: check the Centos Release notes – they will state why a package cannot be built for Centos (for example, for branding reasons).

For the future of Centos? They want to enable future technologies. To facilitate this they’ve released Centos for 32-bit x86, ARM32 build as armv7, ARM64 built as aarch64, POWER8 Little Endian as ppc64le and POWER7 Big Endian as ppc64. They key thing about all of these different architectures is that the final release is familiar and consistent across the board. These provide a platform for Openstack to build on with guaranteed updates – and with groups such as the Cloud Special Interest Group working together things like the an update from Openstack or Centos shouldn’t break anything.

Karanbir was very enthusiastic about the Centos Continuous Integration (CI) platform. “The CentOS CI is a public resource that open source based projects can use for integration tests on bare metal hardware. The goal of the project is to be a resource for communities that build on top of CentOS in order to enable them to perform better automated testing.” In simple terms, you could put a docker file into the system, and you’d get a container out which would be fully tested.

On a final note, Karanbir also asked if anyone had contacts at Digital Ocean to help get the Centos team work with them to provide updates for Digital Ocean images.

Karanbir Singh Website
Karanbir Singh on Twitter

I then attended a number of other sessions. Here are the notes and links!

Relax-and-Recover simplifies Linux Disaster Recovery by Gratien D’haese

Relax-and-Recover (Rear) is now included in RHEL / Centos 7.2 so it’s now easy to install. After you’ve set things up, remember to continually: Rehearse, maintain, review. After all, what good is a backup plan if you’ve never used it and don’t know it works. Essentially you produce one image per server (physical or VM). The image will contain basic information such as disk partitioning details and the IP address to use. You could create a standard image if all your servers are the same, but remember that you’ll need to change your IP address in the kernel options at boot time. Also, if you’re integrated with TSM or another backup solution, the configuration file will need updating. So perhaps one image per server is good. Or maybe use a dedicated IP address which you always perform restores to. Once the server is recovered, you could then move it to an alternative address. You always need to rerun rear if partitions change. In terms of backup, it will not make a note of any SAN details by default. Rear has been removed from EPEL because it is now part of the core RHEL / Centos 7 base distribution.

Desktop security, keeping the keys to the castle safe by Michael Scherer.

Sometimes it’s necessary to step back and make sure you are doing all you can to keep your systems secure. As a sysadmin you may well have access to confidential information or trade secrets which makes you a rich target. Sure, there will always be low-skilled, automated attempts to login to your systems. But an advanced persistent threat (APT) is a network attack in which an unauthorized person gains access to a network and stays there undetected for a long period of time. How we can we protect ourselves and our companies?

Run a laptop: chose free, open source software beginning with the Operating System. Use supported, recent software. If the software has been written 10 years ago, that fine if it’s still being updated. But an obsolete package could leave you vulnerable. Newer distributions tend to have better cryptography and better security tools. Don’t use a random repository. Check the build system. Perform full disk encryption like Luks. Why not /home? Well if you’re running things like containers, these will probably write to areas outside of /home. And who knows, you may have other customisations which an attacker can use. Veracrypt and Truecrypt are good tools. Protect from thief. Prevent coldboot attack – steal RAM. Beware of the “Evil Maid Attack”, in which an attacker installs a bootkit on an unattended computer, replacing the legitimate boot loader with one under his or her control. Typically the malware loader persists through the transition to protected mode when the kernel has loaded, and is thus able to subvert the kernel. Use Secureboot to ensure trust. Perhaps use TPM? Anti evilmaid can be complicated to setup. What about Anti Evil Made 2? TPM OTP. May be good? Not easy to use. FireWire DMA. Inception. Boot loader on a USB stick. LUKs. Self encrypted stick. USB is a bus so able to sniff traffic. Do not take random gifts. Filesystem bugs. USB stack bugs. USB guards. Hardware security can be depressing! Consider QubesO. See also Installing and Using Anti Evil Maid (AEM) with Qubes OS

Use strong password. Take human factors in account. Use a Password Manger but use a good password. Keep no data on laptop. Disable what you don’t need. Do not listen on network. Different users. Use VM or Vagrant. Containers maybe. But not secure. Virus scanners can be dangerous – a bad file can cause them to crash (they’re likely to be running as root!). Beware of IPv6 and shodan. Manager of NSA. Phishing. No open random attachment. Open in VM. Selinux-sandbox. Firejail, etc. No Docker. Selinux on desktop. MCS policy. Contained user. XDG-apps. Firefox. Remove flash. Block java. Enable when needed. Block multimedia content. Disable webrtc and network access. Httpseverywhere. No script. Cert patrol. Some people remove all CA. Rowhammer.js . Privacy? Mass surveillance. Exploit in adverts. Ad block. Cookiemonster. Local attack. Use screensaver with password. Lock on idle. Use TMOUT to protect tty. Never leave root shell open. Sudo credential authorisation require password again. Disable ptrace or use selinux boolean to disable it. Use SSH keys rather than passwords for login. Do, however, put a password on the key. Use an SSH agent. However, don’t use agent forwarding – a remote admin can take advantage. Use one key per device (laptop1, laptop2, etc) and change key on a regular basis. Automate the key changing. Store they key on smartcard. Yubikey. Audit all the time and store the audit on different server. Make it hard/slow to delete log files (eg issue a command today to delete the logs, but the actual deletion will occur many days later). Machine learning to track sysadmins. Can be a little creepy but it can track unusual behaviour. Used at Facebook and Google. Learns that a sysadmin that has unusually logged into multiple servers and flags this. Use Aide/tripwire on servers. However, not that this can be poor for Fedora due to frequent updates. Use ostree. Logwatch on laptop. For example, an application may be crashing regularly as a result of an exploit someone is trying to take advantage of. After crashing the application 10 times they may get root!

Automated Infrastructure Testing with Oh-My-Vagrant and the CentOS CI by James Shubin.

This was a very entertaining, lively and informative presentation! It included fire! James began by giving a quick docker overview in that docker is just a process on the computer and vagrant gives you a way to spin up virtual machines really easily. Vagrant allows you to quickly setup things like a puppetmaster or ansible server so that you can test out code before it hits production. Oh-My-Vagrant makes it easier and faster to deploy a cluster of virtual machines and containers than with Vagrant alone. With a simple yaml file you can setup a suite of servers. James also introduced a whole suite of tools which can save a lot of time. He also demonstrated hooks into subscription-manager, allowing a container to start, register with Red Hat, perform the necessary tests, and then unregister when the container is destroyed (via vdestroy). In addition, he showed that some images may have limited disk size, so his RHEL script will grow the filesystem (xfs grow) when they are created. See Oh-My-Vagrant examples on github. He is working on a new config management tool called mgmt which he’s presenting next week.

James Shubin blog
James Subin on Twitter
James Shubin on Github

Path from Software Collections to Containers for OpenShift by Honza Horak.

This presentation was aimed more at developers than administrators, but it gave some very useful advice for creating containers. For example, if you want to keep the container size down, always run a “yum clean all” as part of the dockerfile. If installing software, use “yum –setopt=tsflags=nodocs” so as not to include documentation and remember not to run your docker instances as root. Honza also referred to the Nulecule standard way of defining multi container applications configuration.

Getting started with kubernetes by Sebastien Goasguen.

Sebastien was the author of the Docker Cookbook. Sebastien went through a brief history of orchestration technologies including https://en.wikipedia.org/wiki/HTCondor. Essentially, Linux containers have arrived and docker makes them easy to develop. Google gave us cgroups which allows us to isolate processes from one another. What we’re now seeing is a battle for the Orchestrator role: Mesos, Rancher, HTCondor, Kubernetes. Essentially all orchestrator’s work roughly the same way – there is an Admin node, a datastore for persistence, an agent running on the node. Yes, you can now use Kubernetes to run batch jobs just like 25 years ago! However, you can also do a lot more. In choosing an orchestrator, why use Kubernetes? Google got a lot of experience and learned many lessons from Borg. These lessons make Kubernetes very valuable to the Open Source community. If manager fails, pods keep on running. Technically master can restart and recover if data store was stored externally. The main primitives in Kubernetes are pods, replication controller and services. Atomic is pre-packaged with a dockers engine, Kubernetes systemd units, flannel systemd units which makes it a very good choice of foundation for starting out on the Kubernetes journey.

Atomic Developer Bundle – Containerized development made easy by Navid Shaikh & Brian Exelbierd.

The ADB is a vagrant stack which allows developers to work in a familiar environment. For example, they gave an example of “Command line Carl” who is a developer who primarily uses the command line for this work. In this scenario, the pre-built vagrant instance starts up and gives a whole suite of tools ready for development. The docker daemon runs in the instance and the whole things protected by TLS. Similarly, if a developer uses an IDE such as Eclipe, they can just run “vagrant adbinfo” and use a Docker plugin for eclipse. Why does the ADB chose to work with Centos? They like it because they can get valuable feedback from the community and they can make rich use of the software the other special interest groups product. They hold regular meetings and in the future they want to add in more support for additional hypervisors and orchestrators.

Link: Atomic Developer Bundle (ADB) on github

That’s about it! It was a great event that I’ll hopefully return to. I highly recommend it!

Puppet Camp London 2015

Today I attended my third Puppet Camp – Puppet Camp London Spring 2015 – at The Mermaid Conference Center in Blackfriars, London. The event was really informative and I thought it was worth posting up some notes and links about the day.

The auditorium was really nice with plenty of room, very clear sound and a nice clear cinema-sized screen.  Some smaller tech events I’ve been to recently weren’t as good as this, so well done to the organisers and many thanks to sponsors Solidfire, Pagerduty and Speerhead.

Here’s a list of the talks:

Puppet Keynote by Gareth Rushgrove of Puppetlabs. Twitter: @garethr. Gareth gave a great overview of Puppet and Configuration Management (ideal for some members of the audience who were looking at Puppet for the first time) and then spoke about the new features in Puppet 4.0. The key take away was how the newer, best practice methods of managing servers has really evolved over the last 10-20 years. He pointed to the 2014 DevOps Report in which servers and infrastructure are now more reliable than ever, but teams also need to be more agile (more releases, more often) in order to beat their competition.  A quick show of hands in the audience showed how configuration management is used by both traditional developers and traditional sysadmins in equal measure.


Why Puppet? Why now? by David Mytton, Server Density. Twitter: @davidmytton An interesting talk from the point of view of someone who started building a business using a small number of hand-crafted servers and quickly realised the importance of using Configuration Management to scale out the infrastructure in a consistent way.

Helping Data Teams with Puppet by Sergii Khomenko, STYLIGHT. Twitter: @lc0d3r

Autosigning Certificates with Time-based One Time Passwords by David Ellis, TIM Group. (David Ellis, TIM Group, Developer Blog)

Puppet and Your Metadata by Marc Cluet, Ukon Cherry. Twitter: @lynxman. Marc was a really good presenter and gave a great overview of the different types of metadata that is available: Structural Metadata like IP Addresses, Architecture (things that are typically set and cannot be changed) and Descriptive Metadata like $puppetver, $apachever (things that you typically want to set on a host).  Marc made two great references which I’ll look into for storing sensitive information: hiera_gpg and hiera_eyaml (hiera_eyaml on github). When automating metadata there are 4 sources to look for: Provisioning, Puppet, Monitoring, Services. Marc touched on Consul for data discovery and there was more on that in a later talk. Another great point he made was to make sure that any custom facts you create in puppet must be returned in a timely manner. Consider it might take 20 seconds to generate a dynamic fact.   Scale that out over hundreds of servers and hundreds of puppet runs and you can see that is wasteful. For any custom facts that may take time to generate (say, running an SQL query to set some custom stats) run them as a cron job and simply store the data in a text file in /etc/facter/facts.d/

Slides for Marc’s talk can be found here: Puppet and Your Metadata

Puppet Demo by Steven Thwaites, Puppet Labs

Puppet Contained by Owen Ben Davies, Big Sofa.  Personal Homepage: Owen Ben Davies UK. Slides: Puppet Contained Ben covered how developers can use Vagrant and Docker together in order to set up and test applications that are representative of production environments.  One point around Docker images was discussed at the end of the presentation which was when to use Puppet – during the creation of a container, or during the runtime of a container?  As containers are small and made up of small deltas, there is an argument to say you should use puppet on the build side for generating images, and then use Service Discovery with Puppet when you the images are being run.  Do the work once up front.

The last presentation was meant to be Puppet Performance Profiling (Intermediate) – R.I.Pienaar but unfortunately RI couldn’t make the conference.

Instead, Gareth Rushgrove gave a talk and demo titled Service Discovery and Configuration Management – Two Speeds of Configuration.  The slides for Gareth’s talk can be found at here: Service Discovery and Configuration Management.  The idea here is that we can use Service Discovery with puppet to dynamically configure our infrastructure.  A number of different service discovery tools exist already such as etcd, Consul and Zookeeper.  They’re all use in well known projects (etcd by CoreOS, Cloud Foundry and Kubernetes; Hadoop used by Zookeeper; SocketPlane and Cloud Foundry use Consul).  In Gareth’s example, he demonstrated Consul with puppet, triggering a puppet run when changes were made to the running services.  The puppet run creates a dynamic configuration file (in this case NGINX) pointing at the active applications which were discovered by Consul.  So, stopping an application on one node triggers an immediate puppet run on a server which relies on it.  Within in a few seconds web requests are redirected only to applications running on different servers.  The code for the module can be found at lynxman/hiera_consul on github.

Overall, this was a day well spent with plenty of ideas to make better use of puppet.

Oh, and thanks to Pagerduty for the hat, Puppetlabs for the T Shirt and Stickers, and Solidfire for the socks!

Puppet Camp London 2015

Puppet Camp London 2013

Today I attended Puppet Camp London at Mary Ward House. As with the previous London Puppet Camp in March, attendance was very good with something for everyone who’s using Puppet.

Here’s a quick link to the presentations and/or speaker profiles:

  • Puppet Enterprise Demo – Puppet Labs

Update 8/Dec/2013: there’s now a blog post on the Puppet Labs website about the event – Mind the Gap at Puppet Camp London so I’ve added in the video presentations in this article.

Fedora Kickstart Ultra Minimal KDE Installation

Issuing a “yum groups list” (or “yum grouplist” or “yum group list”) on Fedora 19 shows that “KDE Plasma Workspaces” is one of the available environment groups that can be chosen to get a KDE desktop up and running.  However, issuing a “yum grouplist hidden -v” shows this and other KDE groups are available:

KDE Plasma Workspaces (kde-desktop-environment)
KDE Applications (kde-apps)
KDE Educational applications (kde-education)
KDE Multimedia support (kde-media)
KDE Office (kde-office)
KDE Plasma Workspaces (kde-desktop)
KDE Software Development (kde-software-development)
KDE Telepathy (kde-telepathy)

So, if you want to have the full KDE stack on Fedora, it’s relatively easy to pick these groups and include them in your kickstart file.  However, I was after a relatively minimal KDE desktop setup, so I wanted to see what was installed by default and whether we could remove and groups from this selection.

If we take a look at the “KDE Plasma Workspaces” base installation via “yum groupinfo kde-desktop-environment” command we see that it will attempt to install the “+kde-desktop” group, along with groups such as printing, dial-up and multimedia. What if you don’t have a printer hooked up to your workstation, or require dial-up support?  Similarly looking at the “+kde-desktop” group via “yum groupinfo kde-desktop” we see that other groups will be added such as kdegames-minimal, kde-print-manager, kwrite.  I’m not suggesting that these will be unwanted by everyone, just that I don’t necessarily require all of these recommended defaults.

So, how can we create our own custom package selection list to get a minimal KDE desktop up and running?  To start with, we’ll need an X environment and some fonts.  I have no requirement to display Asian or Arabic fonts so I started with the basic X display and fonts and then removed the unwanted ones:

# Start with the fonts group, we then remove a number of fonts
# that we don't need
# Remove unwanted fonts
# Install X11 stack

We can now install KDE and some other useful utilities.  We begin with group “critical-path-kde” since this is an absolute minimum.  We then add the KDE parts we want:

# Start with the KDE absolute minimal
# Then add useful KDE and X Components

We’ll continue building up our Ultimate Fedora kickstart file in the next article, but for now we’ve got enough to a minimum KDE desktop installed by including the above in the %packages section.

Fedora Kickstart DNS Dependencies

In the previous post we added the updates and fedora repositories to our kickstart file. That should mean the packages are pulled down from the Internet if they can’t be found on the install media (those in the “url” statement).

When I first tried this on my home installation, I was still getting the errors about being unable to find some software packages. A look on the install consoles (ALT+F1,F2,F3, etc.) showed the repositories were being disabled for fedora and updates. Initially I couldn’t figure out why. Then it dawned on me. I was using a temporary DHCP server on a Centos 6 provisioning host, and my options were as follows:

option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16; # RFC4578
deny unknown-clients;
subnet netmask {
option routers;

Note that my DHCP server wasn’t returning any DNS servers as my client booted. As such, when the Fedora install started it had no concept of DNS servers to use.  So, I added the line below so that my Fedora install would use the Google public DNS servers (you could replace this with your own DNS servers or that of your ISP):

option domain-name-servers;

The installer was now able to use the fedora and updates repositories to pull down the required packages.

Lesson learnt: make sure that you supply a valid DNS server on your clients when using the kickstart file.