Welcome to Duncan White's Practical Software Development (PSD) Pages.

I'm Duncan White, an experienced and professional programmer, and have been programming for well over 30 years, mainly in C and Perl, although I know many other languages. In that time, despite my best intentions:-), I just can't help learning a thing or two about the practical matters of designing, programming, testing, debugging, running projects etc. Back in 2007, I thought I'd start writing an occasional series of articles, book reviews, more general thoughts etc, all focussing on software development without all the guff.

See all my Practical Software Development (PSD) Pages

picture of D. White

Automatic Code Generators: build tools to help you program at a range of small...medium scales.

As programmers, we are used to building large-scale software that (hopefully) performs useful tasks, correctly, and at decent speed. We often use large-scale tools to reduce the amount of code that we must write by hand. For example, tools like Yacc (aka Bison) and Lex (aka Flex), or my own Datadec tool (described in Article 8 of this series) are large tools, that took substantial amounts of time to design and build.

They each solve one problem - letting you write some simple high-level representation of that problem (Yacc: parser grammars in BNF, Lex: lexical analyser rules in regexes, Datadec: recursive data types), and then each tool automatically generates valid C modules from your high-level representation, that are guaranteed to work and which can be linked straight into the rest of your code.

Each of these tools is what the Pragmatic Programmer book calls Code Generators, and the input to such tools is what Kernighan and Pike in The Practice of Programming call Little Languages. Using the Pragmatic Programmer terminology of Code Generators, they exhort us to write Code Generators - programs that write programs. They emphasize that this sounds more complex than it is, that a simple Code Generator can be written in a few minutes and still save hours of work.

So Yacc, Lex and Datadec are (large scale) Code Generators, that repay the time and effort taken to design and build them by enabling programmers to learn these tools once, and then reuse these tools aggressively on many different projects throughout their careers. But when it comes to building such tools, opportunities to build such large-scale tools don't come along very often in a programmer's career - just as well, perhaps, or we'd spend all our time building tools rather than building the software that we're paid to produce!

But what about much smaller Code Generators? There are many occasions when opportunities to build and use small Code Generators come up. This article is going to explore some examples of small to medium Code Generators, and how to design and build them. Here we can use my guiding principle of Ruthless automation to help ourselves: If you find yourself doing something boring and repetitive while programming, consider writing a quick tool to make your own life easier! Don't suffer, write a tool!

The core principle underlying most Code Generators is spotting patterns and then exploiting them.

Case 1: the Perlstub Code Generator

In my Article 4: Perlstub: Building a Tool, Extending your Editor, I described building one small code generator that helps me program in Perl, and how I embedded it in my favourite editor, There, the pattern I spotted was that I laid out every Perl function (subroutine) in a common format, with some duplication - violating the Pragmatic Programmer's Don't Repeat Yourself (DRY) principle. Specifically I would write a function like this:

    # my @nodups = remove_duplicates( @list );
    #     return @nodups, a list in the same order 
    #     as the input @list without the duplicates.
    sub remove_duplicates (@)
        my( @list ) = @_;
        # STUB: body goes here.
It occurred to me that a simple tool could produce all the above - including the entire subroutine stub - from the following "example call description":
    my @nodups = remove_duplicates( @list );
    return @nodups, a list in the same order 
    as the input @list without the duplicates.
I exploited that pattern - and made my life easier - by building a code generator to construct the Perl function declaration from the example call description. Building the code generator took only a modest amount of time, and I then worked out how to invoke this code generator from my favourite editor, passing it some text. Now this code generator saves me time every single time I write a new Perl function, I write the example call, and press a single key, and the function stub is automatically written for me. It's paid back my investment a hundredfold - probably more.

Read the whole of article 4 for more details of how I did that.

Case 2: Ad-hoc Code Generators: Pattern Instances

Case 3: Autogenerating Stack Tests via a Code Generator and Little Language