PyMake 0.1 - Samuel Bayer (sam@mitre.org)
----------

PyMake is intended to be a substitute for UNIX "make", and perhaps
other system construction tools on other platforms, if there is
sufficient interest and I can get some help (see the section on "The
make context" below for some hopeful pointers). It is modeled after
the functionality of GNU "make", but uses scripts written in Python.
This first release of PyMake consists of an executable, pymake.py,
this README file, and a directory of examples of corresponding GNU
makefiles and PyMakefiles.

If more than a couple people ask for this, I'll post it on starship.
For bug reports, etc., I can be reached at sam@mitre.org.

The command
-----------

pymake.py [-f makefile] [-vv] [-silent] [-n] [target]

All arguments are optional. makefile defaults to "PyMakefile". -silent
and -vv modify the normal verbose output, either turning it off or
enabling additional output. -n blocks the execution of rule bodies,
essentially printing out a plan for construction of the target;
because the execution of bodies is blocked, recursion into
subdirectories will not happen in this mode. target defaults to
"all"; unlike "make", pymake.py does not use the first target it
finds.

The format of PyMakefiles
-------------------------

A PyMakefile is a Python file. The only special aspect of a PyMakefile
is a series of calls to the function MFRule(), which supports special
string processing which mimics certain properties of make. A simple
example is the best illustration:

CC = "gcc -G"

CFLAGS = "-I/usr/include/X11"

LIBS = "-L/usr/lib/X11 -lX11"

XFILES = ["first.c", "second.c", "third.c"]

FULLCC = "%(PURIFY) %(CC) %(PROFILE)"

MFRule("all", XFILES,
       "%(FULLCC) %(CFLAGS) %(XFILES) -o myapp %(LIBS)")

The construction %(<token>) is a special directive meant to evoke both
the ${<token>} of make and the %(<token>)s directive of Python string
substitution. The <token> may be any sequence of [A-Za-z0-9_] (that
is, a legal Python token). If the token is bound to a string or list
of strings in the PyMakefile, that value is used; string values are
used without modification, and lists of strings are joined with
string.join(). If the token has no such binding, the empty string is
used as its value. Recursive substitution is performed on the strings
until all directives are eliminated; circular references are blocked.

The MFRule call
---------------

MFRule(target, dependency_list, body [, restriction_list])

Calls to MFRule() introduce rules, which are searched in reverse order
of introduction (that is, the later rule takes precedence). All
strings in the target, dependency_list, body and restriction_list are
expanded as described immediately above. 

The target is a string which is matched against the input target.
After expansion, it is treated as a regular expression. It uses shell
conventions rather than GNU regexp conventions for characters such as
"*", which matches any sequence of characters in the shell but does
nothing by itself in regexp. The result of matching this target
creates a number of additional bindings for the dependency_list and
body: "FILE" is bound to the full filename, "BASE" is bound to the
basename, and "EXT" is bound to the extension (so BASE + "." + EXT ==
FILE if EXT is nonempty, and BASE == FILE otherwise). In addition, if
the target contains any symbolic names for regexp groupings, these are
also made available. So a target like

"foo.\(<MIDEXT>*\).*"

will match a file like "foo.c.orig" and enable the following
additional expansions in the dependency_list and body:

%(FILE):	"foo.c.orig"
%(BASE):	"foo.c"
%(EXT):		"orig"
%(MIDEXT):	"c"

The dependency_list is a list of strings which correspond to filenames
or other rule targets. They are expanded just like the target (except
for having these additional bindings associated with them). However,
the result is NOT treated as a regular expression, since it
potentially must be treated as an input target for the purpose of
recursive matching against rules.

The body can be a string, a list of strings, or a function. In the
case of strings or lists of strings, the strings are expanded
identically to the dependency list, and the result is treated as a
sequence of system calls. This case is almost identical to UNIX make.
The function case allows the user to provide Python flow of control in
the rule body; the function takes a single argument which provides the
make context, and all actions should be treated as calls to methods of
this context (see the section "The make context" below).

The restriction_list is a list of strings which are processed like the
target. They are intended as a sequence of filters on the input. If
one filter matches (and they can be regular expressions, etc.), the
input is matched against the target and processing continues normally.
So if I want to write a rule which compiles a certain list of C files,
I can write:

CC = "gcc"

XFILES = ["first.c", "second.c", "third.c"]
OFILES = ["first.o", "second.o", "third.o"]

MFRule("*", XFILES,
       "%(CC) %(BASE).c -c -o %(FILE)",
       OFILES)

If the .o file is in OFILES, matching proceeds to the target (which
always matches, since it is "*") and the body is evaluated under the
normal circumstances. Restriction lists are also useful as a
substitute for makefile dependency rules; see the section "Obvious
differences from make" below.

Flow of control
---------------

I have attempted to adhere to the flow of control for make. When a
rule is matched, the body is evaluated if:

(1) there are no dependencies, or

(2) the input target does not exist in the file system, or

(3) at least one dependency corresponds to a rule whose body
must be evaluated, or

(4) at least one dependency corresponds to a file in the file system
which has been modified more recently than the input target

All dependencies are checked, even in the case of (2), unless an error
is encountered which prevents further execution (such as a dependency
neither existing in the file system nor matching a rule). Circular
dependencies are caught and signalled. Bodies are executed in the
order in which they're encountered, so an element later in a
dependency list may depend implicitly on elements before it (assuming
execution isn't blocked with the -n flag). 

The make context
----------------

If the user wants program control over the bodies of rules, or
(perhaps) if the user wants to attempt some cross-platform system
construction (I don't know enough about compiling things under MacOS
or Windows to know if that's even an interesting possibility), the
body of a rule can be a function of one argument, which is a makefile
context. IT IS VERY IMPORTANT FOR EVERY CONSEQUENTIAL CALL IN THESE
FUNCTIONS TO BE INVOKED AS A METHOD OF THE CONTEXT OBJECT. This is
because execution is sometimes disabled, but we still want to print
out what might have happened. The context object has some useful
flow-of-control methods, as well as a Execv() method which takes care
of the vast majority of Python calls which have no special context
object method. 

Here are the methods of the context object:

SystemCall(str): the string is expanded identically to rule string
bodies and executed via a call to os.system(). 

Execv(*args): the first element is a Python callable, and all other
arguments are the arguments to the callable. If the arguments are
strings, they are expanded identically to rule string bodies.

Expand(str): the str may contain %(<token>) directives.  Raises
KeyError if the element is not found in the make context. The returned
value is fully expanded.

Lookup(str): the str is a key in the environment of the PyMakefile.
These are the actual values, not the postprocessed ones consulted by
Expand(). The Lookup() method is necessary because the function body
is not evaluated in the context of the MFRule() call. The PyMakefiles
are loaded using execfile() and their environment is captured, but is
not preserved as an evaluation context. So for the example above:

  Lookup("OFILES") -> ["first.o", "second.o", "third.o"]
  Expand("%(OFILES)") -> "first.o second.o third.o"

InDirectory(dir, *tuples): In the context of the directory "dir",
Execv() is called on each tuple in "tuples".

Make(target): invoke system construction, using the flag values from
the current invocation of pymake.py (except the -f flag). 

Here's how "make clean" might be implemented in the function and
string strategies:

SUBDIRS = ["sub1", "sub2"]

PROGRAMS = ["program1", "program2"]

# Function.

def program_clean(context):
    context.SystemCall("rm -f %(PROGRAMS)")
    for subdir in context.Lookup("SUBDIRS"):
	context.InDirectory(subdir, (context.Make, "clean"))

MFRule("clean", [], program_clean)

# String

def clean_strings():
    comm = []
    for element in SUBDIRS:
        comm.append("cd %s; %(MAKE) clean" % element)
    return comm

MFRule("clean", [], 
       ["rm -f %(PROGRAMS)"] + clean_strings())

These different approaches can be mixed and matched in any makefile
(but not in the body of a single rule).

Obvious differences from make
-----------------------------

There are no implicit rules, no dependency rules, and so far only one
implicit %(<token>) binding; the pymake.py call and its arguments
(except for the -f flag) are bound to MAKE for use in recursion, as
shown in the previous section. Dependency rules in UNIX make are
simply rules where the body is provided elsewhere; these are not
necessary where there are no implicit rules and there is a simple way
of providing a body:

CC = "gcc"
CFLAGS = "-I."

DEFAULTCOMPILE = "%(CC) %(CFLAGS) %(BASE).c -c -o %(BASE).o"

MFRule("first.o", ["first.c"], DEFAULTCOMPILE)
MFRule("second.o", ["second.c"], DEFAULTCOMPILE)
MFRule("third.o", ["third.c"], DEFAULTCOMPILE)

It's also possible to use restriction lists to handle this sort of
thing:

CC = "gcc"
CFLAGS = "-I."

XFILES = ["first.c", "second.c", "third.c"]
OFILES = ["first.o", "second.o", "third.o"]

MFRule("*", XFILES,
       "%(CC) %(CFLAGS) %(BASE).c -c -o %(BASE).o",
       OFILES)

Examples
--------

PyMake is distributed with an example directory hierarchy, where each
directory contains a UNIX Makefile and the corresponding PyMakefile.
The curious can compare the two files in each directory to see how
idioms correspond. The strength of Python as a scripting language
really comes through here: instead of some arcane ad hoc language
which is a weird mix of UNIX shell and other things, we have Python.
All hail Guido. 

Known bugs
----------

The value of system calls is not checked; so failures will not
terminate pymake.py appropriately. 

The shell used by the system calls is not appropriately controlled
yet. 

The -n flag should keep track of what WOULD have been built and use it
in subsequent processing, instead of consulting the file system as the
arbiter. It's possible that the right thing to do is construct an
execution plan before executing. 
