I have a rocky relationship with Systemd. On the one hand I love how powerful and extensive it is. On the other hand, I hate how cumbersome and clunky it can sometimes feel. There are a TON of moving components and it is very confusing to use if you have no experience with it. The aim of this post is NOT to debate relative merits of Systemd but instead to take users through a few basic examples of how to accomplish tasks with Systemd and get familiar with how to manage systems with this framework.
My background is primarily with Debian/Ubuntu so moving over to this init system has been a learning curve. The problem I had when I first made the transition is that there aren’t a lot of great resources currently for making the change.
Most of my knowledge has been pieced together from various blog posts and sites, the best of which is over at the Arch wiki. As Systemd continues to mature it is becoming increasingly easier to find good guides and resources but the Arch wiki is still the defacto, go to place to find resources about how to use Systemd.
Another thing that has helped is forcing myself to use CoreOS, which, for better or worse has forced me to learn how to use Systemd. It seems that most modern Linux distro’s are moving towards Systemd, so it’s probably worth it to at least begin experimenting with it at this point. Ubuntu 15.04 as well as Debian 8 have both made the jump, along with other RedHat based distro’s, with more to follow.
While the learning curve can be a little steep to start with, you can learn about 80% of the things that Systemd can do with about 20% of the effort. Getting the basics down takes a little bit of effort but will more than be enough to manage a system. Before starting, as a disclaimer, I do not claim to be an expert but I have learned the hard way how things work and sometimes don’t work with Systemd.
Basics of Systemd
So now that we have a little bit of background stuff out of the way, we can look at some of the meat and potatoes of Systemd. The first thing to do is get an idea of what services are running on your system.
systemctl
This will give you a listing of everything running on your system. Usually there are a very large number of units listed. This isn’t really important but say for example, there is a failed unit. You can filter systemctl to only spit out failed units.
systemctl --failed
That’s pretty cool. If, for examle there is a failed unit file you can drill down in to that unit specifically.
systemctl status <unit>
This will give you process and service information for the unit file and the last 10 lines of logs for the unit as well, which can be really handy for troubleshooting.
One of the easier ways to work with unit files is to use the edit subcommand. This method removes much of the complexity of having to know exactly where all of the unit files live on the system.
systemctl edit --full <unit>
You can manage and interact with Systemd and the unit files in a straight forward way. We will examine a few examples.
systemctl start <unit> - start a stopped service
systemctl stop <unit> - stop a running service
systemctl restart <unit> - restart a service
systemctl disable <unit> - stop unit from loading on boot
systemctl enable <unit> - load unit on boot
If you make a change to a service or unit file (we will go over this later) you need to reload the Systemd daemon for it to pick up the changes to the file.
sytemctl daemon-reload
This is super important (on CoreOS at least) when troubleshooting because if you don’t run the reload your process will not update and will not change its behavior.
There are many many more tips and tricks for working with systemctl but this should give users a basic grasp on interacting with components of the system. Again, for lots more details I advise that you check out the wiki.
Journalctl
If you are coming from older Debian/Ubuntu distros you will need to get familiar with using journalctl to view and manipulate your log files.
One of the first changes that people notice right away is that there aren’t any beloved log files /var/log in Systemd any more. Well, there are still logs, but it isn’t what most folks are used to.
The files haven’t gone away, they are simply disguised as a binary file and are accessible via journald. To check how much space your logs are taking up on disk you can use the following command.
journalctl --disk-usage
Journalctl is a powerful tool for doing just about everything you can think of with log files. Because journald writes logs as a binary file you need this tool to interact with them. That’s okay though because journalctl has pretty much all of the features needed to look through logs with its built in flags. For example, if you want to tail a log file, you can use the “-f” flag to follow “-u” to specify the systemd unit and the systemd unit name to follow it as illustrated below (this is a CoreOS system so your results may vary on a different OS).
journalctl -f -u motdgen
To follow all the entries that journalctl is picking up (pretty much just tailing all the logs as an equivalent) you can just use this.
journalctl -f
Sometimes you only want to view the last X number of entries in a log file. To look at older entries you can use the “-n” flag.
journalctl -n 100 -u motdgen
You can filter logs to only show messages since the last boot.
journalctl -b
There are many more options and the filtering capabilities of journalctl are very powerful. Additionally, journalctl offers JSON outputs, filtering by log priority, granular filtering by time with –unit and –since flags, filtering on different boots, and more.
I suggest exploring on your own a little bit to see how far the capabilities can go. Here is a link to all of the various flags that journalctl has as well as a good writeup to get a good idea of some of the more advanced capabilities of journalctl.
Drop-in Units
Writing unit files is somewhat of an art form. I feel that it can quickly become such a complicated topic that I won’t discuss it in full detail here.
By default, Systemd unit files are written to /usr/lib/systemd/system and custom defined unit files are written to /etc/systemd/system. Systemd looks at both locations for which files to load up. Because of this, you can extend base units that live in the /usr/lib path by augmenting them in /etc/systemd.
This has a few implications. First, it makes management of unit files a little bit easier to work with. It also makes extending units a little bit easier. You don’t need to worry about manipulting the system level units, you just add functionality on top of them in /etc/systemd.
Drop-in units are a handy feature if you need to extend a basic unit. To make the drop-in work, it must follow the format of /etc/systemd/system/unit.d/override.conf, where unit.d is the full name of the service that you are extending.
The following example extends the CoreOS baked in etcd2 service
/etc/systemd/system/etcd2.service.d/30-configuration.conf
[Service]
# General settings
Environment=ETCD_DEBUG=true
Replacing Cron
Well not exactly. Systemd doesn’t use cron jobs so if you want to get something to run on a schedule you will need to use a timer unit and associate a service with it.
[Unit]
Description=Log file cleaner (runs daily at midnight)
[Timer]
OnCalendar=daily
Notice that there is a log-cleaner.timer as well as a log-cleaner.service (below). The easiest way I have found is to create a service/timer pair, which ensures that the timer runs the service based on its schedule.
[Unit]
Description=Log file cleaner
[Service]
ExecStart=/usr/bin/bash -c "truncate -s 1m /data/*.log"
The only thing that the service is doing is running a shell command which will get run based on the timer directive in the timer unit file.
The timer directive in the timer unit is pretty flexible. For example, you can tell the timer to run based on the calendar either by day, week or month or alternately you can have the unit run services based on the system time by seconds, minutes or hours.
One of the down sides of timers is that there is not an easy way to email results. With cron this is easily configurable. Likewise, timers don’t have a random delay to run jobs like cron has. So, timers are definitely no replacement for cron but have a lot of usable functionality, it is just confusing to learn at first, especially if you are used to cron.
Here is more involved tutorial for working with timers. Also, the Arch wiki once again is great and has a dedicated section for timers.
Conclusion
This introduction is really just the beginning. There are so many more pieces to the Systemd puzzle, I have just covered the basics, in fact I have deliberately chosen not to cover many of the topics and concepts because they can convolute the learning process and slow down the knowledge. I chose to focus instead on the most basic ideas that are used most frequently in day to day administration. Feel free to go do your own research and play around with the more advanced topics.
Systemd aims to solve a lot of problems but brings about a lot of complexity as well to accomplish this task. In my opinion it is definitely worth investing the time and energy to learn this system and service manager though, pretty much all the Linux distro’s are moving towards it in the future and you would be doing yourself a disservice by not learning it.
Other areas of Systemd that aren’t covered in this post but are worth researching and playing with are:
- Targets (analogous to runlevels)
- Managing NTP with timedateclt
- Network management with networkd
- Hostname management with hostnamectl
- Username management with loginctl
One of the easiest ways to get started with systemd is by grabbing a Vagrant box that leverages Systemd. I like the CoreOS Vagrant box but there are many other options for getting started. One of the best tools for learning is by doing, so take some of these commands and resources and go start playing around with Systemd. Feel free if you have any extra additions or need help getting started with Systemd.
Read More
