On Friday I attended my first Centos Dojo in Brussels , Belgium held one day before FOSDEM. The timing fits in well for the Open Source community since you can attend this, FOSDEM and Configuration Management Camp as part of one trip. The official Centos events page will have links to the videos and slides, but I wanted to share these notes that I made from the day.
What is a dojo? The Centos Dojo website states that “The CentOS Dojo’s are a one day event, organised around the world, that bring together people from the CentOS Communities to talk about systems administration, best practises in linux centric activities and emerging technologies” and it certainly lived up to these expectations. The good: each presentation was about an hour long, the speakers knew their topics, many of the sessions were interactive allowing for questions and discussions.
The day began with Karanbir Singh introducing State of the CentOS Project. During the Q&A, a couple of interesting things were raised. Firstly, someone had issues with the graphical updater failing after an update to Centos 7.2. Karanbir suggested that many things were re-based (ie underwent a major upgrade) in RHEL / Centos 7.2 including many of the Gnome packages. It sounded as though these types of major update may become more commonplace in the future. On the one side people want reliability and stability, but at the same time they want newer functionality, so things like a graphical updater breaking may be an occasional casualty. The advice is raise bugs where ever possible.
@kbsingh opening up the @centos dojo at IBM pic.twitter.com/ZlitE7fVdL
— Fabian Arrotin (@Arrfab) January 29, 2016
There was another question around Centos providing an Extended Update Support (EUS). Essentially, this boiled down to asking Red Hat to make those EUS updates available as source RPMs. If they do that, then Centos could package these. There are 4 components at play in this discussion: Copyright, License, Contract and Intellectual property. Red Hat are one of the better companies for publishing Open Source code, but the EUS packages are not one of them and anyone using them has probably agreed in their contract not to redistribute them. As yourself this: do other Operating Systems such as SuSE and Ubuntu provide their sources in an easy-to-consume manner?
Why is a package in RHEL but not in Centos? Answer: check the Centos Release notes – they will state why a package cannot be built for Centos (for example, for branding reasons).
For the future of Centos? They want to enable future technologies. To facilitate this they’ve released Centos for 32-bit x86, ARM32 build as armv7, ARM64 built as aarch64, POWER8 Little Endian as ppc64le and POWER7 Big Endian as ppc64. They key thing about all of these different architectures is that the final release is familiar and consistent across the board. These provide a platform for Openstack to build on with guaranteed updates – and with groups such as the Cloud Special Interest Group working together things like the an update from Openstack or Centos shouldn’t break anything.
Karanbir was very enthusiastic about the Centos Continuous Integration (CI) platform. “The CentOS CI is a public resource that open source based projects can use for integration tests on bare metal hardware. The goal of the project is to be a resource for communities that build on top of CentOS in order to enable them to perform better automated testing.” In simple terms, you could put a docker file into the system, and you’d get a container out which would be fully tested.
On a final note, Karanbir also asked if anyone had contacts at Digital Ocean to help get the Centos team work with them to provide updates for Digital Ocean images.
Karanbir Singh Website
Karanbir Singh on Twitter
I then attended a number of other sessions. Here are the notes and links!
Relax-and-Recover simplifies Linux Disaster Recovery by Gratien D’haese
Relax and recover at the @CentOS Dojo. #centos pic.twitter.com/ZMl2GK3UKf
— Unix (@UNIXSA) January 29, 2016
Relax-and-Recover (Rear) is now included in RHEL / Centos 7.2 so it’s now easy to install. After you’ve set things up, remember to continually: Rehearse, maintain, review. After all, what good is a backup plan if you’ve never used it and don’t know it works. Essentially you produce one image per server (physical or VM). The image will contain basic information such as disk partitioning details and the IP address to use. You could create a standard image if all your servers are the same, but remember that you’ll need to change your IP address in the kernel options at boot time. Also, if you’re integrated with TSM or another backup solution, the configuration file will need updating. So perhaps one image per server is good. Or maybe use a dedicated IP address which you always perform restores to. Once the server is recovered, you could then move it to an alternative address. You always need to rerun rear if partitions change. In terms of backup, it will not make a note of any SAN details by default. Rear has been removed from EPEL because it is now part of the core RHEL / Centos 7 base distribution.
Desktop security, keeping the keys to the castle safe by Michael Scherer.
Protecting the keys and keeping data safe in IT. Michael walking through challenges and potential solutions. pic.twitter.com/i21ajiNerC
— Karanbir Singh (@CentOS) January 29, 2016
Sometimes it’s necessary to step back and make sure you are doing all you can to keep your systems secure. As a sysadmin you may well have access to confidential information or trade secrets which makes you a rich target. Sure, there will always be low-skilled, automated attempts to login to your systems. But an advanced persistent threat (APT) is a network attack in which an unauthorized person gains access to a network and stays there undetected for a long period of time. How we can we protect ourselves and our companies?
Run a laptop: chose free, open source software beginning with the Operating System. Use supported, recent software. If the software has been written 10 years ago, that fine if it’s still being updated. But an obsolete package could leave you vulnerable. Newer distributions tend to have better cryptography and better security tools. Don’t use a random repository. Check the build system. Perform full disk encryption like Luks. Why not /home? Well if you’re running things like containers, these will probably write to areas outside of /home. And who knows, you may have other customisations which an attacker can use. Veracrypt and Truecrypt are good tools. Protect from thief. Prevent coldboot attack – steal RAM. Beware of the “Evil Maid Attack”, in which an attacker installs a bootkit on an unattended computer, replacing the legitimate boot loader with one under his or her control. Typically the malware loader persists through the transition to protected mode when the kernel has loaded, and is thus able to subvert the kernel. Use Secureboot to ensure trust. Perhaps use TPM? Anti evilmaid can be complicated to setup. What about Anti Evil Made 2? TPM OTP. May be good? Not easy to use. FireWire DMA. Inception. Boot loader on a USB stick. LUKs. Self encrypted stick. USB is a bus so able to sniff traffic. Do not take random gifts. Filesystem bugs. USB stack bugs. USB guards. Hardware security can be depressing! Consider QubesO. See also Installing and Using Anti Evil Maid (AEM) with Qubes OS
Use strong password. Take human factors in account. Use a Password Manger but use a good password. Keep no data on laptop. Disable what you don’t need. Do not listen on network. Different users. Use VM or Vagrant. Containers maybe. But not secure. Virus scanners can be dangerous – a bad file can cause them to crash (they’re likely to be running as root!). Beware of IPv6 and shodan. Manager of NSA. Phishing. No open random attachment. Open in VM. Selinux-sandbox. Firejail, etc. No Docker. Selinux on desktop. MCS policy. Contained user. XDG-apps. Firefox. Remove flash. Block java. Enable when needed. Block multimedia content. Disable webrtc and network access. Httpseverywhere. No script. Cert patrol. Some people remove all CA. Rowhammer.js . Privacy? Mass surveillance. Exploit in adverts. Ad block. Cookiemonster. Local attack. Use screensaver with password. Lock on idle. Use TMOUT to protect tty. Never leave root shell open. Sudo credential authorisation require password again. Disable ptrace or use selinux boolean to disable it. Use SSH keys rather than passwords for login. Do, however, put a password on the key. Use an SSH agent. However, don’t use agent forwarding – a remote admin can take advantage. Use one key per device (laptop1, laptop2, etc) and change key on a regular basis. Automate the key changing. Store they key on smartcard. Yubikey. Audit all the time and store the audit on different server. Make it hard/slow to delete log files (eg issue a command today to delete the logs, but the actual deletion will occur many days later). Machine learning to track sysadmins. Can be a little creepy but it can track unusual behaviour. Used at Facebook and Google. Learns that a sysadmin that has unusually logged into multiple servers and flags this. Use Aide/tripwire on servers. However, not that this can be poor for Fedora due to frequent updates. Use ostree. Logwatch on laptop. For example, an application may be crashing regularly as a result of an exploit someone is trying to take advantage of. After crashing the application 10 times they may get root!
Automated Infrastructure Testing with Oh-My-Vagrant and the CentOS CI by James Shubin.
Really great talk on Oh-My-Vagrant by @purpleidea at the @CentOS Dojo! Thanks @redhatopen #centos pic.twitter.com/krCInpcMLx
— Unix (@UNIXSA) January 29, 2016
This was a very entertaining, lively and informative presentation! It included fire! James began by giving a quick docker overview in that docker is just a process on the computer and vagrant gives you a way to spin up virtual machines really easily. Vagrant allows you to quickly setup things like a puppetmaster or ansible server so that you can test out code before it hits production. Oh-My-Vagrant makes it easier and faster to deploy a cluster of virtual machines and containers than with Vagrant alone. With a simple yaml file you can setup a suite of servers. James also introduced a whole suite of tools which can save a lot of time. He also demonstrated hooks into subscription-manager, allowing a container to start, register with Red Hat, perform the necessary tests, and then unregister when the container is destroyed (via vdestroy). In addition, he showed that some images may have limited disk size, so his RHEL script will grow the filesystem (xfs grow) when they are created. See Oh-My-Vagrant examples on github. He is working on a new config management tool called mgmt which he’s presenting next week.
James Shubin blog
James Subin on Twitter
James Shubin on Github
Path from Software Collections to Containers for OpenShift by Honza Horak.
App definition Nulecule example for #postgresql @hhonza pic.twitter.com/4EamcRgapF
— Karanbir Singh (@CentOS) January 29, 2016
This presentation was aimed more at developers than administrators, but it gave some very useful advice for creating containers. For example, if you want to keep the container size down, always run a “yum clean all” as part of the dockerfile. If installing software, use “yum –setopt=tsflags=nodocs” so as not to include documentation and remember not to run your docker instances as root. Honza also referred to the Nulecule standard way of defining multi container applications configuration.
Getting started with kubernetes by Sebastien Goasguen.
The road to Kubernetes with @sebgoa at the #CentOSDojo pic.twitter.com/SNEq647Wlw
— Karanbir Singh (@CentOS) January 29, 2016
Sebastien was the author of the Docker Cookbook. Sebastien went through a brief history of orchestration technologies including https://en.wikipedia.org/wiki/HTCondor. Essentially, Linux containers have arrived and docker makes them easy to develop. Google gave us cgroups which allows us to isolate processes from one another. What we’re now seeing is a battle for the Orchestrator role: Mesos, Rancher, HTCondor, Kubernetes. Essentially all orchestrator’s work roughly the same way – there is an Admin node, a datastore for persistence, an agent running on the node. Yes, you can now use Kubernetes to run batch jobs just like 25 years ago! However, you can also do a lot more. In choosing an orchestrator, why use Kubernetes? Google got a lot of experience and learned many lessons from Borg. These lessons make Kubernetes very valuable to the Open Source community. If manager fails, pods keep on running. Technically master can restart and recover if data store was stored externally. The main primitives in Kubernetes are pods, replication controller and services. Atomic is pre-packaged with a dockers engine, Kubernetes systemd units, flannel systemd units which makes it a very good choice of foundation for starting out on the Kubernetes journey.
Atomic Developer Bundle – Containerized development made easy by Navid Shaikh & Brian Exelbierd.
@bexelbie and @SwordPhilic talking abt the @ProjectAtomic Atomic Developer Bundle @ the Brussels @CentOS #centosdojo pic.twitter.com/xxtArSQZ7Y
— Dusty Mabe (@dustymabe) January 29, 2016
The ADB is a vagrant stack which allows developers to work in a familiar environment. For example, they gave an example of “Command line Carl” who is a developer who primarily uses the command line for this work. In this scenario, the pre-built vagrant instance starts up and gives a whole suite of tools ready for development. The docker daemon runs in the instance and the whole things protected by TLS. Similarly, if a developer uses an IDE such as Eclipe, they can just run “vagrant adbinfo” and use a Docker plugin for eclipse. Why does the ADB chose to work with Centos? They like it because they can get valuable feedback from the community and they can make rich use of the software the other special interest groups product. They hold regular meetings and in the future they want to add in more support for additional hypervisors and orchestrators.
Link: Atomic Developer Bundle (ADB) on github
That’s about it! It was a great event that I’ll hopefully return to. I highly recommend it!