Pipes

From Helpful
Jump to: navigation, search
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Seems to apply to pipes and to sockets(verify)


For context, 'pipes are one-directional FIFOs, an OS-backed way of inter-process communication, and usually refers to *nix (there is a similar concept on Windows, but semantics differ).

It refers to either

  • named pipes, which can be referred to via the filesystem


Note there may be pipes involved where you expect none. Subprocesses are often tied together with pipes, which includes wrapper/launcher scripts (e.g. firefox on *nix is a shell script that does some testing and launches firefox-bin, the actual binary).


Broken pipe

...or "signal 13", which is SIGPIPE and the way this is handled by the OS.


A program sending on a pipe assumes the receiving program keep accepting data until the sender is done.


If the receiver close()s it (or the OS closed it because the receiver process exited or crashed) before the sender is done, and the sender write()s more data, that next write() will trigger this signal.

That means the sending side will receive SIGPIPE. Unless you handle that signal the write operation will get EPIPE, which typically means an error like write failed: broken pipe and the sender process stopping.



Examples:

  • Ctrl-C in a piped construction
because it usually kills the last part of that pipe (verify)


  • ssh
    shell logins
specific shells are basically pipes over the SSH connection, so will often fail this way
often means the connection was dropped (if the cause is an overzealously cleanly modem, look at keepalive, see Ssh#Dropped_idle_connections)


  • pipes that intentionally stop
consider that the use of head, tail in pipes may be perfectly functional, in that you're basically saying "I don't want more input" via signals that amount to "shut up and die". You're just seeing the OS noticing this has happened. (often you could
>/dev/null
this away, but it's still sort of ugly)
Example:
find . -type f -print0 | xargs -0 ls -ltr | head -n 10
this is essentially lazy evaluation
It seems a bunch of command line either handle SIGPIPE or check for close more often (verify)
...but e.g. xargs does not[1] so usually does what that intended to and complains.



  • pipes that unintentionally stop
Consider "list lines from files under /etc matching foo" done like:
find /etc -type f | xargs grep foo | (read f && echo "MATCH: " $f)
read
is a bash builtin that reads a single line[2]. Because it's a single command in a subshell, that subprocess reads one line and stops, which closes the pipe created for it, and causes the grep process to write into a closed pipe and stop. You'll get xargs reporting this, i.e.: xargs: grep: terminated by signal 13
(The intent was probably a for loop in that subshell, but it's akward to work that way)
In this particular case, the best fix would probably be to handle xargs's output as a stream, e.g.
find /etc -type f | xargs grep foo | sed 's/^/MATCH: /'