Init systems and service management

From Helpful
Jump to navigation Jump to search
Linux-related notes
Linux user notes

Shell, admin, and both:

Shell - command line and bash notes · shell login - profiles and scripts ·· find and xargs and parallel · screen and tmux ·· Shell and process nitty gritty ·· Isolating shell environments ·· Shell flow control notes


Linux admin - disk and filesystem · Linux networking · Init systems and service management (upstart notes, systemd notes) · users and permissions · Debugging · security enhanced linux · PAM notes · health and statistics · Machine Check Events · kernel modules · YP notes · unsorted and muck


Logging and graphing - Logging · RRDtool and munin notes
Network admin - Firewalling and other packet stuff ·


Remote desktops



These are primarily notes
It won't be complete in any sense.
It exists to contain fragments of useful information.

tl;dr, commands

SysV start/stop

# historically:
/etc/init.d/apache2 start


The service script, when present, likely wraps at least sysv, so these days you can often also do the following and it'll also work on systems with newer service managers:

service apache2 start

SysV enable/disable

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Note: SysV and its various imitations may vary a little in the details.

SysV init basically runs entries in specific directories - which are typically symlinks.

  • from directories under /etc/rc.d/ representing runlevels
  • to the scripts in a place like /etc/init.d
  • with names start with S10service and K40service, where S and K means start and kill, and the number is the priority within this runlevel

As such, going to a specific runlevel runs a bunch of starts and stops.


As it's work to alter /etc/rc.d/ contents manually, so the contents of these directories are sometimes tweaked via convenience tools such as

  • update-rc.d (CLI)
  • chkconfig (CLI)
  • rcconf (CLI curses)
  • sysv-rc-conf (CLI curses)
  • bum (GUI)
  • jobs-admin (GUI)
  • sometimes computed from something else

upstart start/stop

initctl interacts with upstart's init daemon.

initctl start apache2
initctl stop apache2


There are specific helpers scripts for less typing:

One-time start/stop:

start apache2
stop apache2
restart apache2
status apache2


(The service script, when present, likely wraps upstart)

upstart enable/disable

Via basic configuration:

  • a service definition placed in /etc/init will be used
  • its startup requirements can be be configured
e.g. #commenting the start on line will effectively disable it.


(It's arguably a little cleaner to disable existing services via overrides. That is, you can create a /etc/init/<service>.override and having it contain manual, because (man page:) "If an override file is present, the stanzas it contains take precedence over those equivalently named stanzas in the corresponding configuration file [...]". This effectively allows local exceptions without touching the service definitions.)

systemd start/stop

systemctl talks to the systemd daemon.

There are some scripts around it.

systemctl start apache2
systemctl status apache2.service
# .service is optional


The service script, when present, likely wraps systemd, as does RHEL/Centos's chkconfig - see below)

systemd enable/disable

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

If you have just edited it you need to do a

systemctl daemon-reload

first, otherwise you can skip that.


Enabling services basically means "putting it in a target (unit)" (read up for details), so

systemctl enable apache2
systemctl disable apache2

systemctl typically alters /etc/systemd/system/multi-user.target.wants/, which is similar to what historically was runlevel 3 [1].


Overview of all services:

# active services
systemctl list-units --type=service
# _all_ services
systemctl list-units --type=service --all

Start/stop in all (in theory)

The service command was initially meant to run sysv scripts in a predictable environment rather than your shell's (verify).


Then people used it to wrap other things, also during transitions between systems, so if present it's likely to control sysv, upstart (initctl commands), systemd (systemctl commands), and openrc, whichever are applicable.

While you can't count on it always being there, it's convenient when it is.

service apache2 start
service apache2 stop
service apache2 restart
service apache2 status


To enable/disable services you still need the system-specific commands.




Also, chkconfig

RHEL/Centos also has chkconfig, which seems to unify systemd (via systemctl), sysv, and xinetd. It has historical origins, like service.

chkconfig --list             
 
# add to startup. This will also turn them on for, by default, runlevels 2 through 5
chkconfig --add iptables
# remove
chkconfig --del ip6tables
# turn on/off at said runlevels
chkconfig mariadb on
# turn on/off at a specific runlevel
chkconfig ip6tables --level 2 off
 
# check whether configured for startup: add only service name
chkconfig network && echo "Network service is configured for startup"
# check whether configured for startup in specific runlevel: name and --level
chkconfig network --level 2 && echo "Network service is configured for startup in level 2"


init/service systems in more detail

SysV-style init

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Runlevels and init scripts in imitation of how SYSV did it (and of little else from SYSV)

  • scripts in /etc/init.d/ are from used to start and control services
many controlling a single process.
  • These scripts can be run directly (e.g. /etc/init.d/apache start).
  • ...but usually the basic idea is that sysv starts a particular service in a runlevel (mostly made to deal with order of services at boot time)


Most of these scripts stay short by using provisions from /sbin/runscript via a hashbang, which handles most of the boilerplate and means you only need to flesh out out a few major functions. It's also not unusual to see init scripts themselves use start-stop-daemon to start their target executable, e.g. for its pidfile-handling convenience.


From the perspective of some later, fancier alternatives, things that sysvinit lacks include:

dependencies - e.g. starting networking before a networked daemon can only be handled through ranking services and starting them in series
can't start things lazily (only when needed, also including dependencies)
can't start unrelated things in parallel


SysV init scripts

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)



upstart

See Upstart notes

systemd

See Systemd notes

OpenRC

Like SysV, but cleaner and easier to handle. (verify)

https://wiki.gentoo.org/wiki/Project:OpenRC

https://en.wikipedia.org/wiki/OpenRC


Epoch

http://universe2.us/epoch.html

https://github.com/Subsentient/epoch


finit

http://troglobit.com/projects/finit/

SMF (solaris)

http://www.oracle.com/technetwork/articles/servers-storage-admin/intro-smf-basics-s11-1729181.html

launchd (osx)

https://en.wikipedia.org/wiki/Launchd

Some supporting concepts and utilities

runlevels

Runlevels came from sysv but were adopted

There are usually six levels, and they usually represent specific states of the system.

Over time, they have most usually meant something like:

  • 0 shutdown/halt
  • S/s/1 single-user, possibly all the same, possibly all mildly different, possibly only 1 exists
(s is n't a runlevel internally; it tells init "go to 1 but also do this specific thing")
used as a "everyone keep out while I fix low-level stuff" mode
  • 2 multi-user mode, no networking (or same as 3)
  • 3 multi-user mode
  • 4 not used (or same as 3)
  • 5 multi-user mode graphical interface (or same as 3)
  • 6 reboot


More recently it's usually simpler, something like

  • 0 shutdown
  • 1 single-user
  • 3 graphical multi-user mode
  • 6 reboot
and everything not named aliased to 3


And since that's only two actual states, one of which is rare, modern init systems have moved away from runlevels as such (also because the sysv way of administering them is a bit involved)


...but may still imitate them. For example, in systemd you may find:

runlevel0.target is a link to poweroff.target
runlevel1.target is a link to rescue.target
runlevel2.target is a link to multi-user.target
runlevel3.target is a link to multi-user.target
runlevel4.target is a link to multi-user.target
runlevel5.target is a link to graphical.target
runlevel6.target is a link to reboot.target

and e.g. booting to one via kernel parameter would be

systemd.unit=multi-user.target

where classically it was sticking the number of the runlevel on the end


See also:

pidfiles

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Pidfiles are used as an on-filesystem reminder of which process (by its PID) is which instance of that program, and are most typically used in service management.

It's simply a text file that stores the PID (sometimes followed by a newline, sometimes not), not unusually stored in /var/run, and not unusually with the .pid extension.

In effect, pidfiles are (filesystem-based) advisory locking, though they allow more than locking - you can fairly easily:

  • resolve a service to the process that currently represents it
  • check whether that service is (still) running
  • send signals to the the right process (reload, kill, etc.)
  • allow multiple instantiations of a program/service (without ambiguity)


If a program writes its own pidfile, you often want to be able to tell it where to write it. Aside from avoiding hardcoded assumptions, it also means that you can move all the responsibility for this to whatever does the service management. It's also basically necessary if you want to allow multiple instantiations.



It's also bit rickety (and, in some more modern service management, not really necessary).

This all relies on the assumption that the process a pidfile points at is actually the one that we started.

The most significant possible problem in working with pidfiles is that of a stale pidfile: if a pidfile was not removed when the service process quit (or crashed, as there's often nothing watching for that case, or whether the system didn't shut down cleanly), then the pidfile may not refer to a present process, or it may point to a completely unrelated process (and there usually isn't a simple way for process management to check the latter).

Stale pidfiles caused by crashes can be avoided by adding a guarding process, such as start-stop-daemon. Regardless of the way the service quit (cleanly and removed the pidfile, or not), the wrapping program can remove the pidfile as necessary.

It's generally handiest to have the service process itself create the pidfile when it starts and remove it when it quits. A guardian program is primarily useful as a fallback for only removal. You don't probably don't want the guarding process to create the pidfile. While it knows the PID of the process it started, that PID isn't always the one you want - consider forking daemons, apps being started though helper/wrapper scripts, and such.


To use pidfiles more robustly, consider the following:

  • having the service process itself create the pidfile, for the reason just mentioned
  • Use a guardian process, and note that you'll need to tell both it and the underlying app the pidfile path
  • having the service process itself remove the pidfile
    • at program exit, including signaled kills (also makes management around it easier)
    • preferably have it do so via an at-program-exit hook so that it will may also happen in controlled crashes
  • if the pidfile a program should write to exists, either complain and quit, or check it as well as we can, and remove it when applicable (when it doesn't point to a process, or we are sure it cannot be another instance of ourselves) before starting. This avoids non-startable daemons caused by stale pidfiles.
    • you can do this either in a possible manager around the service, (and/)or in the program itself. A program may be better at telling whether whether another copy of itself is running. A manager around it can often only give a warning like "Process might still be running. Check manually, remove /var/run/my.pid if it has stopped, and try again."


start-stop-daemon

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

start-stop-daemon is a utility which can, among other things, keep track of a pidfile (created by what it starts, or created by itself), change effective user, daemonize, hand in environment variables, chroot, etc. It's a handy tool when starting services.

You can specify process(es) via --exec (full path), --pidfile, --user, and/or --name. The most specific/expressive is probably the pidfile. (Note that you can use start-stop-daemon to stop things we have not started ourselves)


Notes:

  • --start means that unless it already exists, the process is started (using --exec or --startas).
  • --stop means that if a process exists, a signal (see also --signal) will be sent to it.
  • return code depends on case and on use of --oknodo. Generally: 1 on error, 0 on success or when --oknodo is used.


  • if the executable you specify
    • fairly immediately terminates (forks off / starts something else in the background). (note that -m, telling start-stop-daemon to create a pidfile based on the process it runs, doesn't make sense in this case. Because of this, you usually want a --pidfile argument to the process.)
    • doesn't terminate, you can use -b to have start-stop-daemon itself be the thing that forks off, and it will be the parent of the real process. (You can use -m in this case).

Daemons that fork also mean that --exec won't match them, so some init features won't necessarily work, and unless you use one of the other features (e.g. security) you may forego start-stop-daemon.


See also:

tcpwrappers

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

'TCP wrappers' refers to a setup where a daemon (e.g. inetd, xinetd) acts as the network listener, and controls when to launch other network daemons. Basically, when it gets a connection, it runs a daemon.

One main reason was ease on admin: it lets you centralizes logging, and the logic to accept/reject on a per-host/net basis (which is the tcpd part(verify)).

(Another is that if accesses to services are rare, their resource is essentially zero while they're not running)


That said, some services were always better off running their own logic.

Also, logging is often easy enough anyway, and iptables often does fine for access control (and more centralized yet, really),

...so tcpwrappers is not very common anymore.


configuration files

hosts.allow, hosts.deny

Read by tcpd, so primarily really applies when using tcpwrappers, but there seem to be some daemons that choose to adhere to these files if they exist(verify).


See also

man 5 hosts_access
man 5 hosts_options

unsorted

http://www.atnf.csiro.au/people/rgooch/linux/boot-scripts/ http://www.novell.com/documentation/suse91/suselinux-adminguide/html/ch13s04.html http://www.novell.com/coolsolutions/feature/15380.html



linux

http://www.atnf.csiro.au/people/rgooch/linux/boot-scripts/

gentoo

Uses a customized system.

/etc/runlevels, which is usually updated via rc-update, mentions init scripts, with symlinks that point to items in /etc/init.d/. (note the runlevels are defined in /etc/inittab)

http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4