Quasi - Yet Another Python Shell

Hello. This is the SourceForge web page for Quasi, another Python shell, but with an interesting structure. Well, I think it's interesting (that's why I work on it), but then I would, wouldn't I?
You can get from here to the SourceForge Project Page where the downloads are, or read/join the extremely-low-traffic mailing list.

A Sort Of Overview

So what's so darn interesting about all this then? Well, the key point is that Quasi supports "pluggable contexts". A context here means that it's a different way of interpreting the commands you type. I sense a need for examples, so here goes:

>>> for x in `os ls *.py`:
...     print x

The default context (the text in this colour) is, as you might expect, Python. The "backtick" quotes (the opening and closing ` characters)introduce text that's in a different context (text in blue); in this example, the os context. The interpretation of the text depends on the context. Simple, eh? And not particularly original.

However, it gets clever in that Quasi has several different contexts (and more can be added). For example:

>>> #Here's a SQL example
>>> data = `sql select * from myTable`
>>> #Now data is a list of rows...
>>> data
[('Some', 'All'), ('Here', 'There'), ('A', 'B')]
>>> #And each is a clever object...
>>> row = data[0]
>>> row.keys()
('ColumnA', 'ColumnB')
>>> row.values()
['Some', 'All']
>>> row.ColumnA
'Some'

Quasi works out what context a particular backticked part of a line is for by examining the first few characters (essentially, either the first word or the first character). For instance, to invoke the os context, which executes commands and returns the output back to Python, start a context with os, as in `os ls`.

You can get a list of the contexts installed in Quasi by typing help. Amongst the help text that's printed is a list. Here's what it lists in version 0.84:

Context keywords and the contexts they invoke.
A line beginning with one of these keywords will automatically invoke the given context.

cd,pwd,ls,dir,ostype,pushd,popd
  Context for builtin commands.

os (more help available; type help os_context)
  Operating system commands that return output as a list of strings.

sql,& (more help available; type help sql_context)
  SQL context for MySQL.

!,shell (more help available; type help shell_context)
  Operating system commands that don't return output.  Use for invoking programs that need input from stdin.

A neat trick is that if the full line is intended to be a context, you don't need to type the backticks. So:

`!vi quasi.py`

!vi quasi.py

Incidentally, that's an example of invoking the shell context to run a program (i.e., vi) that needs to take over the console for input. And it's also an example of how to edit quasi from inside Quasi, which is, frankly, going over the top.

Most of the contexts also add commands to the set available. The help text above shows how you'd list them; for example, in the SQL context, you'd type help sql_context. And you'd get this (taken from version 0.84):

This context has the following commands (that follow a context keyword):
commit: Commit cursor operations.
connect: connect host='host', user='user', passwd='passwd', db='db'
        Connect to a MySQL server.  The args are
        in Python keyword-value format.
connection: Return the actual connection, for those as wants it.
cursor: Generate and return a cursor for the current connection.
describe: describe [<table>]
        Describe the given table, or, if there's no table given,
        describe the last SQL statement (list the fields returned).
encoding: Set the encoding type from/into which
        Unicode string literals are munged.
        By default it's 'latin1', which matches the
        MySQL default.  This is a global setting.
rollback: Rollback cursor operations.

There's also a general description of each context. Incidentally, if you don't have MySQLdb installed, then you'll still see the SQL context listed, but it won't work for you (you'll get exceptions thrown). This may change in future versions, especially as more contexts are added.

Variable Substitution

All the contexts support variable sustitution. You can drop the value of a Python variable right into a command. Here's an example:

>>> a='*.py'
>>> for x in `os ls $a`:
...     print x

The $a drops the value into the other context. There are two ways to write a variable substitution; like this: $a, where the a is a Python identifier, and like this: $(a), where the parentheses can contain any valid Python expression.

Smart Substitution

The first method (no parentheses) is called "smart substitution". Quasi (or, more exactly, the context-specific code doing the variable substitution) takes a good guess at the right way to substitute the variable's value into the context. For example, if you have a list of strings, how should they be inserted into an OS command? On Unix the best answer is probably as whitespace-separated strings, but that also implies that any values containing spaces need to be quoted. On Windows, the default is to use commas to separate. In the SQL context, the substitution code goes as far as scanning the SQL command to deduce what the variable is being used as. Time for more examples, this time using a neat Quasi trick: the explain command. Preceding a context with explain tells Quasi not to actually execute the command but just to show what it would look like after variable substitution.

>>> cd quasitest
'c:\\Documents and Settings\\ben\\My Documents\\ben\\python\\quasitest'
>>> files = `os ls`
>>> #files is a list of the filenames, and one contains a space.
>>> files
['a.b', 'a.c', 'my File.txt', 'myFile.txt', 'test.txt']
>>> #smart substitution puts the filename with a space into quotes
>>> explain `os hackwith $files`
'hackwith a.b a.c "my File.txt" myFile.txt test.txt'

A couple of points; yes, I'm running under Windows. Mostly under cygwin, if you care. But note that the cd command returns the new path (which is why Python prints it with all the backslashes escaped). Quasi has built-in ls (or dir), cd, pushd, popd and pwd commands.

>>> # x is a sequence of strings
>>> x
('One', 'Two')
>>> # let's try some SQL and see how smart subs work
>>> explain `sql select $x from myTable`
"select 'One','Two' from myTable"
>>> explain `sql insert into myTable ($x) values ($x)`
"insert into myTable ('One','Two') values ('One','Two')"
>>> # now, let's try a dict
>>> x = {'ColumnA' : 'One', 'ColumnB' : 2, 'ColumnC' : 32.5 }
>>> # Dicts (and the objects that a SELECT returns), have column names
>>> # and values.  The smart sub uses whichever is appropriate.
>>> explain `sql select $x from myTable`
"select 'ColumnA','ColumnC','ColumnB' from myTable"
>>> explain `sql insert into myTable ($x) values ($x)`
"insert into myTable ('ColumnA','ColumnC','ColumnB') values ('One',32.5,2)"
>>> explain `sql insert into myTable set $x`
"insert into myTable set ColumnA='One',ColumnC=32.5,ColumnB=2"

Note that a dict doesn't return the keys in any particular order, hence the A, C, B ordering. SQL select commands return the data in smart objects that do preserve the order. The smart substitution decides to use the column names, values or both, depending on the SQL syntax that precedes it.

The SQL context also has a neat trick for SELECT clauses. Use the word SMARTSELECT instead of SELECT, and the results are handled slightly differently. If there's only one row returned, it comes back as an object, not a list of length one, so you can do:

>>> employee = `sql smartselect * from Employee where EmployeeId=123`
>>> #There's only one record, so smartselect just gives us the Bag object with the column values.
>>> print employee.Name, employee.Department
Ben Last Dept. Of Getting Things Done

Smart selection also works differently for rows that only contain one value, so:

>>> employees = `sql smartselect count(*) from Employee`
>>> #employees will just be the actual count (a long)
>>> print employees
12L

I like smart selection :)

Expression Substitution

Of course, there are drawbacks to any system that thinks it knows what you want, and the Zen of Python says that explicit is better than implicit. Thus the second form of variable substitution; expression substitution. You can put any Python expression you like in the parentheses in the $(a) form. More examples:

>>> # files is still that list of strings
>>> files
['a.b', 'a.c', 'my File.txt', 'myFile.txt', 'test.txt']
>>> explain `os fgrep something $(string.join(files,' '))`
'd:\\cygwin\\bin/fgrep something a.b a.c my File.txt myFile.txt test.txt'
>>> # Hmm, what about the file with a space in the name?  Use the built-in
>>> # quoted() method.
>>> explain `os fgrep something $(quoted(files))`
"d:\\cygwin\\bin/fgrep something 'a.b' 'a.c' 'my File.txt' 'myFile.txt' 'test.txt'"
>>> # Or we can filter out just the text files...
>>> explain `os fgrep something $([quoted(x) for x in files if x.endswith('.txt')])`
"d:\\cygwin\\bin/fgrep something 'my File.txt' 'myFile.txt' 'test.txt'"

Builtin Stuff

The smart-eyed reader will have noticed the use of the built-in quoted() method in the examples above. There are a number of these that are available in the interpreter namespace (the source is, as you'd expect, in quasi.py):

• quoted() returns any string surrounded by either single or double quotes (single-quotes as default). If passed a list, it returns a list of the strings, each quoted(), joined with whitespace. Currently it uses Python's repr() as a quick-and-dirty way to do this.
• dquoted does the same, but uses double-quotes and will escape any double-quotes already in a string.
• commas() joins the elements of a list with commas; this is mostly for Windows work.
• modules() returns a list of the currently imported modules in the interpreter namespace; just to make one's working life a little easier.
• Bag() is a useful class. Bags can behave like dicts (in that they have a keys() method, and you can access values using the x[y] syntax) or like objects (in that you can access values using the x.y syntax). You must always assign values using the former (x[y]), but after that you can reference values either way. A Bag will also remember the order in which values were assigned; this is used in the SQL context, for example, so that a Bag's keys or values are always returned in the order that a SELECT obtained them.

There are also a number of built-in shell commands. Unlike nearly everything else that gets typed into Quasi, these execute in the namespace of the shell, not the interpreter. More on that another time, but it means that any variables your Python code may have defined are not accessible to built-in shell commands. This is why none of them need any real arguments:

• exit: exit Quasi.
• help: show lots of help on various sorts of command.
• credits: show the names of the people who have contributed to Quasi.
• license: show the Quasi (BSD-style) license.
• history: show the command history. Supports slice notation (1:20), plus history searching. For example, history sq will find all command lines that begin with "sq".
• recall: recall a previous command or set of them. This is very useful when you've been writing indented code and want to recall a set of previous lines. Same syntax as history - specify a single line or a slice-like set of them. Recall has a limitation; it requires a functional readline module available, to insert the recalled command into the input buffer for editing. If that can't be done, the recalled line(s) are printed to allow copying and pasting. Also, if more than one line is recalled, there's no way to edit them, so they're executed.

There are also a set of built-in commands which execute in the interpreter namespace and can, therefore, do variable-substitution trickery:

• cd: change directory. You can do cd $x to change to the directory whose path is in variable x.
• pwd: return the current working directory. You can do x=`pwd` to return the path of the current working directory into x, or just type pwd to see it.
• pushd, popd: Oh come on... they work like the bash equivalents, ok? Both return the directory where they end up, as a string, so x=`pushd subDirectory` or here=`popd` work.

Windows users should Beware The Backslash Problem. If you want to set variable x to c:\windows, you need to type x='c:\\windows'. It's Python. You can use raw strings to help: x=r"c:\windows", but beware - even a raw string cannot end in a slash, so r"c:\" won't work. It's a familiar problem to anyone who does path work under Python on Windows, and if there's a neat solution to it, please tell me...

Miscellaneous

Quasi attempts to import and use readline support where it's available, via import readline. On Windows there are a number of readline modules, and some of them are deficient in some way; either they lack features or they override basic key settings. Quasi attempts to detect which one you have installed and set it up sensibly. Where possible, it overrides any default association for the tab key - at least one readline module uses it for command completion, but my view is that when you're mostly typing Python, you need a tab key.

If you're doing any Python work on Windows, I strongly recommend that you install Gary Bishop's readline. You'll also need to install the ctypes module for it to run. Quasi supports extended functionality with this installed.

Where Next?

There is, obviously, more to it all than this, but that'll get you started for now, until I expand this page to be all-encompassing. Until then, enjoy!