View source for Python usage notes/pty and pexpect notes

{{Pythonstuff}}

=pty=


When running barely-interactive or non-interactive commands, [[subprocess]] does everything you want, and is probably much easier to deal with.


But '''when a program wants to be interactive''', you want something like the <tt>pty</tt> module, to interact with the program via this a pseudoterminal. 

pty mostly just wraps existing C functions, adds some convenience, though is not always friendly or well documented.

Before you write a custom pty-based class around your specific command -- which can make sense in itself -- you may want to check what <tt>[http://www.noah.org/wiki/pexpect#Description_of_Pexpect pexpect]</tt> can do for you.

{{comment|(pexpect wraps <tt>pty</tt> with some pure-python convenience functions, geared to easier pattern-based interactions. Notes on <tt>pexpect</tt> are below those on <tt>pty</tt>)}}


====communication====

<tt>pty.fork()</tt> gives you a file descriptor object that is tied to the child's <tt>stdin</tt> that you can <tt>os.read()</tt> on, and stdout which you can <tt>os.write()</tt> on.

You ''may'' want to poll the file descriptor, to avoid blocking reads.
See e.g. [http://stackoverflow.com/questions/6422016/python-how-to-peek-into-a-pty-object-to-avoid-blocking/6490489#6490489]}} -- and e.g. read <tt>pty.spawn()</tt>'s implementation, as it [http://docs.python.org/library/select.html select()]s on the fd and only calls the callback when there is something to do.


====the functions====

* '''pty.fork()''' - creates a child which sees the calling process as its pseudoterminal master
** returns a pair (child_pid, fd)
*** in the master process, ''child_pid'' is the subprocess's PID, and ''fd'' is connected to the child's controlling terminal
**** you can use os.read() and os.write() on the fd. That's stdout and stdin; stderr is not handled{{verify}}
*** in the child process,  ''child_pid'' is 0, and ''fd'' isn't valid
** You'll probably use one of the os.exec* functions in the child
** basically consists of <tt>pty.openpty()</tt> plus <tt>os.fork()</tt>, and fiddling with the standard streams


* '''pty.spawn(argv_string[, master_read[, stdin_read]])'''
** returns nothing
** spawns child process, sets its controlling terminal to be us <!-- (probably a combination of pty.fork() and os.exec* ? )-->
** Not useful for interaction, but nice when you're looking ''just'' to fool a program into thinking it's in a pty
** master_read and stdin_read arguments are functions that take a file descriptor and return (by default) 1KB of data from it. These arguments seem to exist to let you intercept the data and do something more with the data (e.g. [http://docs.python.org/py3k/library/pty.html this example] logs the output to a file)


* '''pty.openpty()''' - creates new pty (like [http://docs.python.org/library/os.html#os.openpty os.openpty], but is slightly more portable than it (){{verify}})
** returns a 2-tuple: the file descriptors for master and slave end{{verify}}

<!--
def reading_callback(fd, nbytes):
   return fd.read(nbytes)

pty.spawn(my_argv, reading_callback, my_std_read_callback)
-->


===Examples===

<code lang="python">
#!/usr/bin/python
# I have this in a script named  logbatch
import pty,sys,os

if len(sys.argv)<2:
    print "Need a command to run"
    sys.exit(-1)

logfilename = '%s.log'%os.path.basename(sys.argv[1]) # executable name + '.log'
if os.path.exists(logfilename):
    print "\nRefusing to overwrite existing log %r\n\nRename or remove it and try again\n"%logfilename
    sys.exit(-1)

logfile = open(logfilename, 'wb')

def read(fd):
    data = os.read(fd, 1024)
    logfile.write(data)
    logfile.flush() # probably
    return data

sys.stdout.write('Writing log to %r\n'%logfilename)
sys.stdout.flush()

logfile.write('Log file for command: %r\n---------------------\n'%' '.join(sys.argv[1:]))
pty.spawn(sys.argv[1:], read)

sys.stdout.write('Done with logging %r\n'%cmd)
</code>

If I run {{inlinecode|logbatch ls -l}}, this script itself runs {{inlinecode|ls -l}} and write its stdin+stdout to <tt>ls.log</tt>

{{comment|(It was made for automatically generated batch scripts, and to avoid typing something like <tt>scriptname_1 >& scriptname_1.log &</tt>. And no, {{inlinecode|script}} [http://linux.die.net/man/1/script] didn't do what I wanted).}}

===See also (pty)===

* http://docs.python.org/py3k/library/pty.html
* http://sqizit.bartletts.id.au/2011/02/14/pseudo-terminals-in-python/
* http://stackoverflow.com/questions/4022600/python-pty-fork-how-does-it-work
<!--
* man openpty
-->

=pexpect=

<tt>pexpect</tt> lets you deal somewhat more cleverly with things spawned via <tt>pty.spawn</tt>.

{{comment|(It also has a module that eases remote interaction via SSH)}}


===Quick intro===

<code lang="python">
import pexpect

# The simplest intraction is a "wait until you see this, then send this" thing:
p = pexpect.spawn('ftp ftp.example.com')
p.expect ('[Nn]ame')
p.sendline ('anonymous')
p.expect ('[Pp]assword')
p.sendline ('noah@example.com')
# etc.
# But when responses are conditional or otherwise flexible, this stops being sensible.


# If you hand in a list, the return value is the index to which of the patterns matched.
# This also makes it easier to deal with errors, timeouts 
#   and varying code paths - you could build a rule system, or state machine
p = pexpect.spawn('ftp ftp.example.com')

while True:
  i = p.expect([pexpect.EOF, pexpect.TIMEOUT, '[Nn]ame', '[Pp]assword','[#\$]', 'refused'])
  if   i==0:
    print 'program finished'
    sys.exit(0)
  elif i==1:
    print 'program did not respond (fast enough)'
    sys.exit(2)
  elif i==2: 
    print 'sending username'
    p.sendline('foo')
  elif i==3:
    print 'sending password'
    p.sendline('pw12')
  elif i==4:
    print 'looks like a prompt'
    p.sendline('put myfile')
  elif i==5:
    print 'connection refused'
    sys.exit(1)
# not a working example, but you get the idea
</code>


'''Timeout''':
* defaults to 30 seconds. 
* You can change that default
* you can specify a timeout on each expect() call

* By default, timeout is raised as an exception. 
: If you added <tt>pexpect.TIMEOUT</tt> to the list of patterns, you get the according index instead of an exception.

===<tt>spawn()</tt>===
When you have arguments, you can put it in the single command string, or hand them in as a list of strings, so e.g.
<code lang="python">
pexpect.spawn('ssh -X user@example.com')
pexpect.spawn('ssh',['-X','user@example.com'])
</code>

Notes:
* This is '''different''' from the things <tt>[[#the_functions|pty.spawn()]]</tt> can take '''(!)'''

* You could use full paths, or (in the first style) use <tt>/bin/env</tt>, but pexpect tries to imitate the behaviour of {{inlinecode|which}} anyway, so ideally will Do What You Want.

* spawn mostly just forks and execs, so if you want to use shell features in your command, your command should probably be something like  <tt>'bash -c "ls > test "'</tt> (and more escaping fun)

* On the environment:
** The process inherits from the spawning python process - i.e. <tt>os.environ</tt>. The simplest way to make a specific environment is to alter that
** If you use the <tt>env</tt> parameter on spawn, it sends that ''instead'' of os.environ. The cleanest way to hand in a tailored environment is probably to make a copy of <tt>os.environ</tt> and alter, then hand it in for the specific <tt>spawn(env=thatcopy)</tt> call.

===<tt>expect()</tt>===
<code lang="python">
# The arguments to expect:
p.expect(pattern, timeout=-1, searchwindowsize=None)
</code>


'''pattern''' can be:
* a string type - will be compiled as regular expressions
* an already-compiled regexp object, used as-is
* <tt>pexpect.EOF</tt> - return this, rather than raise an exception on EOF
* <tt>pexpect.TIMEOUT</tt> - return this, rather than raise an exception on timeout
* a '''list''' of any of these
** return value will be the index of the matching pattern. If nothing matches, you will get a timeout as an exception
::: note that if it was in the list, you get it as an index (what happens if nothing matches and timeout==None?{{verify}})


A '''timeout''' value of -1 means fallback to the class default, which seems to be 30 (seconds).
You could raise or lower it, or block indefinitely by using <tt>timeout=None</tt>.


'''searchwindowsize''': how far back to search within the incoming data buffer. 
Default is to search the whole thing (since the last match), which can be unnecessarily slow if, say, you want to match the 'done' message after a large bulk of debug output)




'''Returns''':
* If you didn't hand in a list: 0
* If you hand in a list: the index into the ''first'' applicable match (so if one of your patterns is a longer version of another, you probably want it first)


After <tt>expect()</tt> returns, you can inspect the contents of the latest bit of input buffer, probably one or more of:
* '''<tt>p.before</tt>''' - the text before the match
* '''<tt>p.match</tt>'''  - the matched text
* '''<tt>p.after</tt>'''  - the text after the match



Instead of using this general-purpose <tt>expect()</tt>, you can use some of the underlying functions:
* <tt>expect_exact()</tt> - skips regexp stuff, so can be faster when just doing simple substring tests
* <tt>expect_list()</tt>  - when your list contains only compiled regexps (or <tt>EOF</tt> or <tt>TIMEOUT</tt>), you can skip some of the overhead and go straight to the matching

===Reading data without expect()===
{{stub}}

Instead of using expect(), you can do the work yourself, after one of:
* read() 

* read_nonblocking() -  Can do (per-character?) timeout  (may raise eof and timeout exceptions)

* readline() - reads a single line

* readlines() - reads until EOF, returns lines.

===Sending in data===
{{stub}}

For line based programs, you only need to remember sendline().

The list of sending functions:
* sendline(s) - sends a string/bytes, plus os.linesep
* send(s) - send a string/bytes
* sendeof() - send EOF character
* sendcontrol(''char'') - send a control character
* sendintr() - sends SIGINT
* write() - like send, but doesn't return value. Here to make the object usable as a generic python file object {{comment|(see also read, close)}}
* writelines(seq) - is basically <tt>for s in seq: write(s)</tt>


===<tt>interact()</tt>===

(Aside from a few potentially finicky details), this just hands over control completely:

Keystrokes are sent to the child, the child's stdout and stderr are read and printed on our own stdout and stderr (and logfile, if set).


You can hand along an escape character to break out of this interactive mode with - by default 0x1D, {{keyhold|Ctrl}}{{key|]}} {{comment|(a telnet convention, apparently)}}.


It looks like <tt>interact()</tt> expects the user to break out before the child process closes - see the section below.

<!--
If it is interactive only on errors, you could have the timeout catch it. 
<tt>expect([pexpect.EOF,pexpect.TIMEOUT])</tt> to handle it more explicitly (and you could consider shortening the timeout, if sensible).
-->

===Termination of the subprocess===

...handling this cleanly takes a little thought.

It seems that in essentially all cases '''you are expected to close()''' the connection to the subprocess yourself.
{{comment|(multiple close()s are accepted so you don't have to do a lot of codepath checking)}}


====Waiting for EOF====
Many programs will, at some point, '''want no further input before they eventually decide to exit themselves'''.


If you know this is true of a specific program, the simplest solution is probably:
* expect(pexpect.EOF, timeout=None) '''to wait indefinitely for the child to exit'''
* close()


Waiting until it's really done
: is also one of the easiest sure ways to know the exit status is readable.
: avoids early child termination if the parent python process stops before the child
: may also avoid the subprocess being write-blocked on a pipe ({{verify}}I don't know yet)

====Not waiting indefinitely====

Not explicitly waiting indefinitely means a timeout applies, probably the default 30 seconds.

This often isn't enough for a "okay moving on now" thing, yet is preferable when you want to report how long the process has been at it,
terminate() it when you know it's pointless or when a program ''would'' run forver but you want to stop it, etc. 


====Details you may care about====

* {{inlinecode|isalive()}} - tests whether the child process is still there

* {{inlinecode|wait()}} - blocking wait for child to exit. Note that a child is considered alive until all its output is read, so wait() is ''only'' useful if you know it won't.
: In most cases, <tt>expect(pexpect.EOF, timeout=None)</tt> is what you'ld want instead of wait() {{verify}}

* {{inlinecode|close()}} - closes connection with child. If the child process had not closed its streams, this will look like a broken pipe to it (if and when it tries to use them), which often means it doesn't exit cleanly.
: if you don't, the handle will stay open. If you run a lot of subprocesses via pexpect, you will run out.

* {{inlinecode|kill(''signum'')}} - send signal.
: can also be done via close's optional force argument

* {{inlinecode|terminate()}} - tries [[Signal handling|SIGHUP]]/[[Signal handling|SIGINT]], and can be asked to move onto [[Signal handling|SIGKILL]]


Once a process is done, ''either'' '''<tt>self.exitstatus</tt>''' or '''<tt>self.signalstatus</tt>''' will have a value, the other None.

Which of the two it is mostly depends on whether the process was left to terminate by itself (usually with  <tt>expect(pexpect.EOF)</tt>) or was terminated (by your code, or by something else).


====termination and interact()====
{{stub}}

It looks like <tt>interact()</tt> expects the user to break out before the child process closes.

In most examples, the user seems expected to know this.


If the child terminates while still under <tt>interact()</tt> control, you'll probably find it doesn't handle the <tt>EOF</tt>, and you'll get a stack trace ending like:
   File "/usr/lib/python2.4/site-packages/pexpect.py", line 1510, in __interact_read
     return os.read(fd, 1000)
 OSError: [Errno 5] Input/output error


You ''could'' decide to catch and ignore the <tt>OSError</tt> - I expect that if it comes from interact(), it's typically  this.

===logging, echo===
<!--

when ''proc''.logfile isn't None, it is expected to be an open file object that it can write() to.

{{comment|<nowiki>(Keep in mind that when you both echoing responses relevant to you, and set logfile=sys.stdout you will see such output twice)</nowiki>}}


: '''On echoing:'''

Look at:
* <tt>getecho()</tt>
* <tt>setecho()</tt>
* <tt>waitnoecho()</tt> - blocks (with timeout) until terminal echo is disabled. Useful to wait for password prompts.

When an application asks for a password, it typically does so, then asks the system to disable echo for its stdin.
If that isn't handled before pexpect sends the string, it will still be echoed. It helps to delay that send, or to check whether echo is disabled.

pexpect always waits for 0.05 sec (delaybeforesend) for this reason.

-->

===Unsorted===

: '''On matching end of line'''

Since expect() matches substrings on possibly-buffered output, matching against $ won't work (will always match, on the end of the smallish window it's looking at)

So you probably want to match {{inlinecode|[\r\n]}} instead {{verify}}


<!--
=pexpect.pxssh=
An extension of pexpect that specializes in SSH.
Pretty decent for doing things on trusted hosts.

prompt() expects a prompt - and you may want to tweak its timeout
-->