Init systems and service management
| Linux-related notes
Shell, admin, and both:
|
| 📃 These are primarily notes, intended to be a collection of useful fragments, that will probably never be complete in any sense. |
tl;dr, commands
SysV start/stop
# historically: /etc/init.d/apache2 start
The service script, when present, likely wraps at least sysv, so these days you can often also do the following and it'll also work on systems with newer service managers:
service apache2 start
SysV enable/disable
Note: SysV and its various imitations may vary a little in the details.
SysV init basically runs entries in specific directories - which are typically symlinks.
- from directories under /etc/rc.d/ representing runlevels
- to the scripts in a place like /etc/init.d
- with names start with S10service and K40service, where S and K means start and kill, and the number is the priority within this runlevel
As such, going to a specific runlevel runs a bunch of starts and stops.
As it's work to alter /etc/rc.d/ contents manually, so the contents of these directories are
sometimes tweaked via convenience tools such as
- update-rc.d (CLI)
- chkconfig (CLI)
- rcconf (CLI curses)
- sysv-rc-conf (CLI curses)
- bum (GUI)
- jobs-admin (GUI)
- sometimes computed from something else
upstart start/stop
initctl interacts with upstart's init daemon.
initctl start apache2 initctl stop apache2
There are specific helpers scripts for less typing:
One-time start/stop:
start apache2 stop apache2 restart apache2 status apache2
(The service script, when present, likely wraps upstart)
upstart enable/disable
Via basic configuration:
- a service definition placed in /etc/init will be used
- its startup requirements can be be configured
- e.g. #commenting the start on line will effectively disable it.
(It's arguably a little cleaner to disable existing services via overrides. That is, you can create a /etc/init/<service>.override and having it contain manual, because (man page:) "If an override file is present, the stanzas it contains take precedence over those equivalently named stanzas in the corresponding configuration file [...]". This effectively allows local exceptions without touching the service definitions.)
systemd start/stop
systemctl talks to the systemd daemon.
There are some scripts around it.
systemctl start apache2
systemctl status apache2.service
# .service is optional
The service script, when present, likely wraps systemd, as does RHEL/Centos's chkconfig - see below)
systemd enable/disable
If you have just edited it you need to do a
systemctl daemon-reload
first, otherwise you can skip that.
Enabling services basically means "putting it in a target (unit)" (read up for details), so
systemctl enable apache2
systemctl disable apache2
systemctl typically alters /etc/systemd/system/multi-user.target.wants/, which is similar to what historically was runlevel 3 [1].
Overview of all services:
# active services systemctl list-units --type=service # _all_ services systemctl list-units --type=service --all
Start/stop in all (in theory)
The service command was initially meant to run sysv scripts in a predictable environment rather than your shell's (verify).
Then people used it to wrap other things, also during transitions between systems,
so if present it's likely to control
sysv, upstart (initctl commands), systemd (systemctl commands), and openrc, whichever are applicable.
While you can't count on it always being there, it's convenient when it is.
service apache2 start service apache2 stop service apache2 restart service apache2 status
To enable/disable services you still need the system-specific commands.
Also, chkconfig
RHEL/Centos also has chkconfig, which seems to unify systemd (via systemctl), sysv, and xinetd. It has historical origins, like service.
chkconfig --list
# add to startup. This will also turn them on for, by default, runlevels 2 through 5
chkconfig --add iptables
# remove
chkconfig --del ip6tables
# turn on/off at said runlevels
chkconfig mariadb on
# turn on/off at a specific runlevel
chkconfig ip6tables --level 2 off
# check whether configured for startup: add only service name
chkconfig network && echo "Network service is configured for startup"
# check whether configured for startup in specific runlevel: name and --level
chkconfig network --level 2 && echo "Network service is configured for startup in level 2"
init/service systems in more detail
SysV-style init
Runlevels and init scripts in imitation of how SYSV did it (and of little else from SYSV)
- scripts in /etc/init.d/ are from used to start and control services
- many controlling a single process.
- These scripts can be run directly (e.g. /etc/init.d/apache start).
- ...but usually the basic idea is that sysv starts a particular service in a runlevel (mostly made to deal with order of services at boot time)
Most of these scripts stay short by using provisions from /sbin/runscript via a hashbang, which handles most of the boilerplate and means you only need to flesh out out a few major functions.
It's also not unusual to see init scripts themselves use start-stop-daemon to start their target executable, e.g. for its pidfile-handling convenience.
From the perspective of some later, fancier alternatives, things that sysvinit lacks include:
- dependencies - e.g. starting networking before a networked daemon can only be handled through ranking services and starting them in series
- can't start things lazily (only when needed, also including dependencies)
- can't start unrelated things in parallel
SysV init scripts
upstart
See Upstart notes
systemd
See Systemd notes
OpenRC
Like SysV, but cleaner and easier to handle. (verify)
https://wiki.gentoo.org/wiki/Project:OpenRC
https://en.wikipedia.org/wiki/OpenRC
Epoch
http://universe2.us/epoch.html
https://github.com/Subsentient/epoch
finit
http://troglobit.com/projects/finit/
SMF (solaris)
http://www.oracle.com/technetwork/articles/servers-storage-admin/intro-smf-basics-s11-1729181.html
launchd (osx)
https://en.wikipedia.org/wiki/Launchd
Some supporting concepts and utilities
runlevels
Runlevels came from sysv but were adopted
There are usually six levels, and they usually represent specific states of the system.
Over time, they have most usually meant something like:
- 0 shutdown/halt
- S/s/1 single-user, possibly all the same, possibly all mildly different, possibly only 1 exists
- (s is n't a runlevel internally; it tells init "go to 1 but also do this specific thing")
- used as a "everyone keep out while I fix low-level stuff" mode
- 2 multi-user mode, no networking (or same as 3)
- 3 multi-user mode
- 4 not used (or same as 3)
- 5 multi-user mode graphical interface (or same as 3)
- 6 reboot
More recently it's usually simpler, something like
- 0 shutdown
- 1 single-user
- 3 graphical multi-user mode
- 6 reboot
- and everything not named aliased to 3
And since that's only two actual states, one of which is rare, modern init systems have moved away from runlevels as such (also because the sysv way of administering them is a bit involved)
...but may still imitate them. For example, in systemd you may find:
- runlevel0.target is a link to poweroff.target
- runlevel1.target is a link to rescue.target
- runlevel2.target is a link to multi-user.target
- runlevel3.target is a link to multi-user.target
- runlevel4.target is a link to multi-user.target
- runlevel5.target is a link to graphical.target
- runlevel6.target is a link to reboot.target
and e.g. booting to one via kernel parameter would be
systemd.unit=multi-user.target
where classically it was sticking the number of the runlevel on the end
See also:
pidfiles
Pidfiles are used as an on-filesystem reminder of which process (by its PID) is which instance of that program, and are most typically used in service management.
It's simply a text file that stores the PID (sometimes followed by a newline, sometimes not), not unusually stored in /var/run, and not unusually with the .pid extension.
In effect, pidfiles are (filesystem-based) advisory locking, though they allow more than locking - you can fairly easily:
- resolve a service to the process that currently represents it
- check whether that service is (still) running
- send signals to the the right process (reload, kill, etc.)
- allow multiple instantiations of a program/service (without ambiguity)
If a program writes its own pidfile, you often want to be able to tell it where to write it. Aside from avoiding hardcoded assumptions, it also means that you can move all the responsibility for this to whatever does the service management. It's also basically necessary if you want to allow multiple instantiations.
It's also bit rickety (and, in some more modern service management, not really necessary).
This all relies on the assumption that the process a pidfile points at is actually the one that we started.
The most significant possible problem in working with pidfiles is that of a stale pidfile: if a pidfile was not removed when the service process quit (or crashed, as there's often nothing watching for that case, or whether the system didn't shut down cleanly), then the pidfile may not refer to a present process, or it may point to a completely unrelated process (and there usually isn't a simple way for process management to check the latter).
Stale pidfiles caused by crashes can be avoided by adding a guarding process, such as start-stop-daemon. Regardless of the way the service quit (cleanly and removed the pidfile, or not), the wrapping program can remove the pidfile as necessary.
It's generally handiest to have the service process itself create the pidfile when it starts and remove it when it quits. A guardian program is primarily useful as a fallback for only removal. You don't probably don't want the guarding process to create the pidfile. While it knows the PID of the process it started, that PID isn't always the one you want - consider forking daemons, apps being started though helper/wrapper scripts, and such.
To use pidfiles more robustly, consider the following:
- having the service process itself create the pidfile, for the reason just mentioned
- Use a guardian process, and note that you'll need to tell both it and the underlying app the pidfile path
- having the service process itself remove the pidfile
- at program exit, including signaled kills (also makes management around it easier)
- preferably have it do so via an at-program-exit hook so that it will may also happen in controlled crashes
- if the pidfile a program should write to exists, either complain and quit, or check it as well as we can, and remove it when applicable (when it doesn't point to a process, or we are sure it cannot be another instance of ourselves) before starting. This avoids non-startable daemons caused by stale pidfiles.
- you can do this either in a possible manager around the service, (and/)or in the program itself. A program may be better at telling whether whether another copy of itself is running. A manager around it can often only give a warning like "Process might still be running. Check manually, remove /var/run/my.pid if it has stopped, and try again."
start-stop-daemon
start-stop-daemon is a utility which can, among other things, keep track of a pidfile (created by what it starts, or created by itself), change effective user, daemonize, hand in environment variables, chroot, etc. It's a handy tool when starting services.
You can specify process(es) via --exec (full path), --pidfile, --user, and/or --name. The most specific/expressive is probably the pidfile. (Note that you can use start-stop-daemon to stop things we have not started ourselves)
Notes:
- --start means that unless it already exists, the process is started (using --exec or --startas).
- --stop means that if a process exists, a signal (see also --signal) will be sent to it.
- return code depends on case and on use of --oknodo. Generally: 1 on error, 0 on success or when --oknodo is used.
- if the executable you specify
- fairly immediately terminates (forks off / starts something else in the background). (note that -m, telling start-stop-daemon to create a pidfile based on the process it runs, doesn't make sense in this case. Because of this, you usually want a --pidfile argument to the process.)
- doesn't terminate, you can use -b to have start-stop-daemon itself be the thing that forks off, and it will be the parent of the real process. (You can use -m in this case).
Daemons that fork also mean that --exec won't match them, so some init features won't necessarily work, and unless you use one of the other features (e.g. security) you may forego start-stop-daemon.
See also:
tcpwrappers
'TCP wrappers' refers to a setup where a daemon (e.g. inetd, xinetd) acts as the network listener, and controls when to launch other network daemons. Basically, when it gets a connection, it runs a daemon.
One main reason was ease on admin: it lets you centralizes logging, and the logic to accept/reject on a per-host/net basis (which is the tcpd part(verify)).
(Another is that if accesses to services are rare, their resource is essentially zero while they're not running)
That said, some services were always better off running their own logic.
Also, logging is often easy enough anyway, and iptables often does fine for access control (and more centralized yet, really),
...so tcpwrappers is not very common anymore.
configuration files
hosts.allow, hosts.deny
Read by tcpd, so primarily really applies when using tcpwrappers, but there seem to be some daemons that choose to adhere to these files if they exist(verify).
See also
man 5 hosts_access man 5 hosts_options
unsorted
http://www.atnf.csiro.au/people/rgooch/linux/boot-scripts/ http://www.novell.com/documentation/suse91/suselinux-adminguide/html/ch13s04.html http://www.novell.com/coolsolutions/feature/15380.html
linux
http://www.atnf.csiro.au/people/rgooch/linux/boot-scripts/
gentoo
Uses a customized system.
/etc/runlevels, which is usually updated via rc-update, mentions init scripts, with symlinks that point to items in /etc/init.d/. (note the runlevels are defined in /etc/inittab)
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4