What is it?

Source 2 Binary (S2B) is a software build utility with that aims to be easier to use and more flexible than the traditional 'make' environment.

Motivation

Simply put, software build environments using 'make' and its supporting utilities are more complicated than they need to be.

The combination of makefiles, shell scripts, standard Unix utilities, m4 Autoconf macros, and Perl scripts currently used to provide portable software builds is an effective but complex solution. For larger projects, forcing the amount of required flexibility out of these tools is a difficult task. Attempting to support multiple platforms and widely varying system configurations in addition to user preferences can result in build environments that are closer to full-fledged programs than they are simple files used to drive software compilation..

S2B is an attempt to simplify the software build process. Whether or not it will succeed in this arena has yet to be determined. At this point, the project consists of a general idea and a proof-of-concept prototype. The current goal is to generate some discussion and see if enough people think this project is worth pursuing. So if you love it, hate it, or just have an opinion; let's hear about it in the Open Forum

Overview

At the fundamental level, S2B is similar to almost every other project in this vein. It consists of a library of Python modules that provide the basic functionality found in Autoconf and Make (dependency tracking, library availability checks, etc) and do the normal OO thing of platform/tool abstraction. These modules are then used by project-specific scripts to drive the build. Pretty standard. The interesting part comes from some tricks Python allows us to play with the scripts used to drive the build.

The idea behind these tricks (for the lack of a better term), is pretty simple. We instruct the Python interpreter to load the project-specific script and give us the resulting module object. We then do some inspection and processing of the module contents. This scheme allows us to break away from the normal programming style and present the user with a simplified interface. By declaring a few variables and adhering to a few conventions, the user can implicitly describe what they want to happen. In effect, we can use a normal Python module almost like customized S2B scripting language.

One of the reasons 'make' is so popular is the ability to quickly and concisely describe targets, dependencies, and operations. Attempting to do the same with a programming language is much more difficult and probably the reason 'make' wasn't replaced by Perl scripts a long time ago. The module processing scheme employed by S2B allows us to support an interface similar to that of traditional makefiles while still allowing full access to the underlying language. For general use, users will be able to use a simple makefile-like interface. When complicated situations arise that make this interface awkward to use, they can switch over to normal Python code.

How it Works

As previously mentioned, the fundamental structure of S2B is pretty standard and uninteresting. Consequently, it will be largely ignored in this discussion. Since the simplified interface is what makes this project stand out, it will receive the most attention.

The Overview mentioned that the key to the simplified interface lies in inspecting and processing the contents of Python modules. For those unfamiliar with the capability Python provides in this area, the following list describes the key features related to this project.

• We can get a list of all variables defined in the module (Both the names of the variables and their contents)
• Run Time Type Inspection can be used to determine the exact type of every module element
• We can gain references to functions defined in the module and use those references to call the functions
• The contents of Class objects and Object instances can be examined in the same way

We will use the following example to describe how everything fits together.

#!/usr/bin/python

import s2b_lib

from s2b_lib.targets import Target

test = Target()

test.sources = """ main source_file1 source_file2 """

s2b_lib.run()

The last line causes the s2b library to inspect the contents of this module. The inspection will reveal a single object of type Target with the member variable 'sources'. If the user is building the test target, the sources variable will be examined to determine its type. The type could be, as in this example, a list of source files (the extensions will be automatically determined by scanning the directory), a normal Python function that returns a list of source files, a regular expression object like regex.compile("server_*.c") to be applied to the directory contents, or any number of things that could be used to gain a list of source files.

Since the variable name used for the Target object is test and no specific output type was specified in the Target() constructor, the library will assume that the result is to be an executable instead of an object file (for which the user could either have named the variable test_o or constructed the object with Test( targets.OBJ )). By default, the name of the variable is what will be used for the name of the output. Knowing the list of sources, the name of the output, and what the output type is; the library will check the to see if each one of the source files is up to date (i.e. has a matching .o file that is current), compile those that are not, and link them together to result in the executable "test" binary.

To specify extra dependencies, we could add the line test.depends = "aTargetName aSourceName aBooleanFunctionName" By examining the names in the this string after the module has been imported, the s2b library would be able to determine that the test target depends on another Target object, a source file, and a regular Python function (which, of course, would be called during the dependency check).

Library dependencies can be delt with in a similar manner. Consider the the case in which our example included the line test.libraries = "X11 pthread QT3". Obviously, this would indicate that the files in this target will need access to the header and '.so'/'.a' files associated with these libraries during the compile/link phase.

By loading all of the project-specific modules and collecting all the <target_name>.libraries = "..." information, we can create a complete list of library dependencies for the project. This list can then be used to accomplish the same sort of check done by the 'configure' scripts. A side advantage of this scheme is that we do not need to place all of our external dependencies in a single, top-level location.

Rather than using m4 to define the library checks, as is done with Autoconf, we can use a more flexible and consistent (if more verbose) method. Since the rest of S2B is written in Python, we might as well do this in Python as well. Library-specific issues are managed by library-specific modules in the s2b_lib.libs package. By following a few simple naming conventions, support for a new library can be added by simply dropping a new module into this package.

The following code is an example of a module for X11.

from s2b_lib.sys_conf import lib_info

from s2b_lib.tools    import test_compile


include_dir   = "/usr/X11R6/include"

libraries_dir = "/usr/X11R6/lib"



def detect_configuration():

  if test_compile.compile_and_link("x11_test.c", include_dir, libraries_dir):

    return lib_info.LibInfo(..., include_dir, libraries_dir, ...)

  else:

    return lib_info.library_unavailable()

Look pretty self-explanatory? Good. That's the what we're shooting for. One of the nice things about doing this check in a Python function is that we can put some extra intelligence in. We could, for example, look in a few common locations such as /opt or /usr/local if the initial check fails. We could look in different places depending on the underlying OS.

One thing about this code that might not be obvious, is that the name and location of the include and library paths allow us to implement an easy scheme for overriding the default paths via command-line switches. Should the user add the switch "--lib_x11_include=<path>" the s2b library would know to overwrite the include_dir variable in the module s2b_lib.libs.x11.

Similar to "configure", we can cache the results of this detection. Our good ol' friend 'pickle' makes caching the LibInfo objects platform-independently trivial.

Another thing about S2B that should be mentioned is the method for handling the source tree. By defining the variables SUB_DIRS = "src lib etc" and EXCLUDE_DIRS = "test doc whatever" the directories visited by the S2B build can be controlled. By default, S2B will visit every directory in the subtree. These two module-level variables modify that behavior.

The final thing to note about S2B in its current state is that the simplified interface we provide is not mandatory. The S2B library could be used like any other Python library for application development. If someone wanted to develop a cross-platform, distributed build system; S2B could be used to do it. It is likely that this capability would also prove beneficial to larger programs that need more logic than our simplified interface can easily provide. The majority would be probably be able to use the easier method but for those spots where it makes more sense to write some real code, you can go ahead and do it..

Miscellaneous Advantages, Disadvantages, and Ideas

• 'make' has ungodly inertia. It might be hard to get people to take any other method seriously.
• Requires Python to be installed on the build system.
• Some may complain that they don't want to have to learn Python to compile their software. Two points seem to defeat this argument. First, unless they're doing something complicated, they'll primarily be declaring variables, not writing code. Second, the S2B system + Python should FAR easier to learn and use than the existing system. That's the whole point..
• The code comprising the s2b library is quite small. It should be reasonable to package the complete library in the distribution tarball and thereby avoid the requirement of having s2b installed on the build system.
• The library-specific modules and test-compile files will likely constitute the majority of the bytes of the s2b library. Before distribution, the s2b library could be told to strip itself of unneeded files.
• By developing python equivalents or wrappers for most of the common unix tools (find, copy, move, etc) most projects should be able to avoid any direct use of the Unix utilities. This should increase consistency across platforms by hiding differences in command-line switches and such.
• Should straightforward to implement automatic installs by scanning the project-specific modules for EXE and DLL targets and user-defined variables containing the names configuration files and the like.
• If Python tools are used in place of the Unix tools, we will be able to track every file copied, deleted, created, or moved. This information would be beneficial to package managers like RPM.
• Parallel builds should be straight-forward to implement via Python threads.
• Everything can be done in Python. Processing a file with Python may not be as concise or as "easy" as with sed+awk but the consistency will benefit future maintainers. Also, you won't need to mix Makefiles, m4 macros, shell scripts, and Unix utilities to get the job done. It's all in one, easy to read syntax.