PSD: Duncan White's Practical Software Development Pages

Welcome to Duncan White's Practical Software Development (PSD) Pages.

I'm Duncan White, an experienced and professional programmer, and have been programming for well over 30 years, mainly in C and Perl, although I know many other languages. In that time, despite my best intentions:-), I just can't help learning a thing or two about the practical matters of designing, programming, testing, debugging, running projects etc. Back in 2007, I thought I'd start writing an occasional series of articles, book reviews, more general thoughts etc, all focussing on software development without all the guff.

See all my Practical Software Development (PSD) Pages

Perlstub: Building a Tool; Extending your Editor
Programmers spend a lot of time in their favourite editor. Personally, I'm a long time vi user to expert level, but that's my choice - your milage may vary. Whatever editor you use, you should follow the Pragmatic Programmer's sage advice:
Tip 22: learn how to use a single editor well - the editor should be an extension of your own hand, configurable, extensible and programmable.
(You can read my review of the Pragmatic Programmer here).

No editor/IDE, no matter how complex and helpful it is, is going to be perfect for you - because every programmer is different. As the Pragmatic Programmer's tip 22 above says, you should be able to extend your chosen editor to perform custom editing tasks, easily and simply, to fix things that irritate you, or slow you down, boosting your own productivity and happiness. If your editor can't support this - choose a better editor!

The minimum that an editor needs to be called extensible (by me!) is some form of macro which allows you to bundle a sequence of editor operations together, plus key binding to allow you to bind your macro to a key sequence of your choice, and the ability for your macro (or the editor commands it invokes) to pipe a block of text out to an external program, replacing the block with the output of that external program.

You might think such editor extensions have to be large and complex beasts. and be put off building them yourself. In this article, I want to convince you that they're quick and easy, by means of an example:
A few years ago, while programming a reasonably large Perl system, I noticed that all my Perl subroutines have a common comment layout, specifically a comment - written first - defining the interface via a "typical call example", followed by any number of free text comments, all indented and describing what the subroutine does (referring to the example call's arguments), followed by the subroutine declaration itself, as in this example:
    #
    # my @nodups = remove_duplicates( @list );
    #     return @nodups, a list in the same order 
    #     as the input @list without the duplicates.
    #
    sub remove_duplicates (@)
    {
        my( @list ) = @_;
        # STUB: body goes here.
    }
I observed that this violates the Pragmatic Programmer's Don't Repeat Yourself (DRY) principle by spreading the knowledge that remove_duplicates takes an array into three places, and the knowledge that the array's local name is @list into two of those places. Then it occurred to me that a simple tool could produce all the above - including the entire subroutine stub - from the following input:
    my @nodups = remove_duplicates( @list );
    {
    return @nodups, a list in the same order 
    as the input @list without the duplicates.
    }
So, in about half an hour, I wrote a simple Little Language Parser (a la Programming Pearls) or Code Generator Tool (a la the Pragmatic Programmer) called perlstub (click here for perlstub.tgz file) and linked it into my favourite editor (vi) as a macro - all described in the README file in perlstub.tgz, and throughout this webpage. Having set this up, whenever I'm happily editing Perl code in vi and decide that I want a new function, I do three things:

name the function.
decide how I want to call the function

write a comment describing what it will do.
I assemble that info into the above simple format, position the cursor over the typical use line and press a key that invokes the macro, runs perlstub with the current paragraph as input, and then replaces that block with the perlstub output - the complete commented stub shown above.
perlstub in More Detail
Ok, having explained what perlstub does, let's look at how I built it. First off, I wrote perlstub in Perl itself, as Perl is the most productive text processing language I know (your milage may vary, that's ok). In general, there's no requirement that the tool is written in the same language as you're wanting to work with in the editor. In some editors, you might need to write some/all of the tool in a built-in editor macro language (such as emacs' powerful built-in Lisp interpreter). Have fun with that:-)
How did I write perlstub? I started with a specification (added to the code as a set of comments) describing the input, the task to perform and the output:
    # perlstub: ok, I've got bored typing all those perl function stubs
    #	    this is a code generator (see The Pragmatic Programmer)
    #	    that reads a function stub of the form:
    #		typical call example
    #		{
    #		comment lines
    #		}
    #	    for instance:
    #		my( $a, $b ) = wibble( x, y, @z )
    #		{
    #		do some wibbling with $x, $y and @z
    #		returning a pair ($a,$b)
    #		}
    #	    and then writes a complete perl subroutine stub, complete with
    #	    standard comment format, prototype and the breaking of the
    #	    param array into variables named precisely as in the example
    #	    call, with '$' prefixes added if they're missing..
    #
    #	    Given the above example input, the output would be:
    #	        #
    #	        # my( $a, $b ) = wibble( x, y, @z );
    #	        #     do some wibbling with $x, $y and @z
    #	        #     returning a pair ($a,$b)
    #	        #
    #	        sub wibble ($$@)
    #	        {
    #	              my( $x, $y, @z ) = @_;
    #		      # STUB: write me
    #           }
    #
    #	(C) Duncan C. White, May 2007
    #
I also extracted the above example, both the input and the expected output, and then placed it in separate files called eg1 and eg1.out, ready for testing (when we have anything to test). Then I added two more examples eg2 and eg3 (along with their expected outputs) to show some variations.
Next, I considered the basic input parsing problem, specifically with respect to the examples:

The first line is the example call, which we'll need to do some more parsing on to extract the function name and the parameters.

The second line must be a '{'.

then every line (up to a closing '}') is a line of comments to be indented.

This suggested the following Perl code, developed in several incremental tussock hops along the way:
    my $func = <>;		# read first line
    chomp $func;

    $func =~ s/[^\)]*$//;       # lose anything after the close bracket

    print "#\n";
    print "# $func;\n";

    while( <> )                 # consume all the rest of the input.
    {
	print "#\t$_" unless /^[{}]$/;
    }
    print "#\n";

    $func =~ s/\s+//g;          # lose all whitespace
    $func =~ s/^.*=//;          # lose any 'thing =' prefix..

    # what's left should look like: wibble(x,y,@z)
    $func =~ /^(\w+)$([^)]*)$$/;

    my $funcname = $1;          # extract the parts
    my $params   = $2;
    print "debug: func $func, name $funcname, params $params\n";
Running the above on eg1 produced the following output:
    #
    # my( $a, $b ) = wibble( x, y, @z );
    #       do some wibbling with $x, $y and @z
    #       returning a pair ($a,$b)
    #
    debug: func wibble(x,y,@z), name wibble, params x,y,@z
This shows that rough and ready parsing is dead simple. Note that I didn't insist that the second line of input be '{' and the last line be '}', I made the Perl code ignore any '{' or '}' lines, and treat every other line (beyond the first) as a comment line! Similarly, I simply assumed that the function call line would be valid, so each regex search-and-replace and pattern match would always work.

In general: a tool doesn't have to be perfect to be useful!
Next, I added code to split the parameter string on ',', iterated over the parameters adding a '$' sigil to any parameter without a sigil, building an array of parameter names and a Perl prototype declaration at the same time. Then finally I joined the parameters up with commas again:
    my @params;
    my $protostr = "";
    foreach (split( /,/, $params ))        # foreach comma separated parameter
    {
	    s/^/\$/ unless /^[\$@%]/;      # prepend $ unless a sigil present
	    push @params, $_;              # build up an array of parameters
	    /^(.)/;                        # extract the parameter's sigil
	    $protostr .= $1;               # add it to the prototype string
    }

    my $paramstr = join( ", ", @params ); # join parameters up again
    print "debug: paramstr=$paramstr, protostr=$protostr\n";
No doubt other solutions exist (after all: There's More Than One Way to Do It) that don't involve splitting and joining the parameters. That's not a problem, I wrote the code the way I liked; feel free to write it whatever way you prefer!
This enhanced program ran as before, but added the new debug message, showing that each parameter with no sigil has now got a '$' prefixed, and the prototype string is the correct sequence of sigils:
    #
    # my( $a, $b ) = wibble( x, y, @z );
    #       do some wibbling with $x, $y and @z
    #       returning a pair ($a,$b)
    #
    debug: func wibble(x,y,@z), name wibble, params x,y,@z
    debug: paramstr=$x, $y, @z, protostr=$$@
Now, we have all the pieces in hand, printing out the Perl subroutine declaration is as simple as:
    print "sub $funcname ($protostr)\n";
    print "{\n";
    print "\tmy( $paramstr ) = \@_;\n" if $paramstr;
    print "\t# STUB: write me...\n";
    print "}\n\n\n";
(Note the neat way we only print the @_ decomposition lines if there are any parameters. There's no reason that automatically generated code should look less pretty than code you write yourself!)
Commenting out the debug prints, and running this on eg1, gives the desired output, including 2 blank lines at the end:
    #
    # my( $a, $b ) = wibble( x, y, @z );
    #       do some wibbling with $x, $y and @z
    #       returning a pair ($a,$b)
    #
    sub wibble ($$@)
    {
            my( $x, $y, @z ) = @_;
	    # STUB: write me...
    }
After some further testing (for example eg2 and eg3 input files in the perlstub tarball), I concluded that perlstub passed it's initial inspection.
Gluing perlstub into vi
Now that I had a mostly working perlstub tool, how did I glue it into vi, in particular how did I get vi to pass just the single function comment block into perlstub? First, I prepared a longer test input file (whole-file-eg in the tarball), which contained Perl code and a perlstub format function section in the middle:
    #
    # some stuff..
    #
    sub one ()
    {
	    print "one\n";
    }


    ninc = eek( maxincs, $entry, %wibble )
    {
    Eek modifies the flibbles of a given $entry
    keeping no more than $maxincs flobbles,
    and creating a new empty aardvark-00.
    Returns the number of flubbles now existing.
    }
    #
    # leave me alone you rotten macro..
    #
    sub two ()
    {
	    print "two\n";
    }
Next I invoked vi on this file:
    vi whole-file-eg
and navigated to the eek() stub in the middle. Then I tried to work out a sequence of vi commands to select the entire eek() stub, and pass it into perlstub via vi's '!' command. I came up with:
    ma            (position mark a on the current line)
    /^}/          (search down for the next '}' line, pressing
                   return after the trailing slash)
    !'aperlstub   (pipe the chunk of the editor buffer from
                   mark a to here into perlstub, replacing
	           the input chunk with the output of perlstub,
	           again press return after perlstub)
(note that I made sure not to write out the modified whole-file-eg file after my tests).
Finally, to turn this sequence of commands into a macro and bind it to an unused vi key '=', I added the following into my ~/.exrc - you can clearly see each of the above commands in the macro string:
    map = ma/^}/^M!'aperlstub^M
(note that each ^M is entered as CTRL-v followed by CTRL-m, and represents a literal newline). Then, reedit the unaltered whole-file-eg, sit the cursor on the first line of the eek function stub, and hit '='. Lo and behold, the entire stub is replaced with the output of perlstub, gloriously well formed Perl code, leaving the code above and below perfectly untouched!
First Part Conclusions

Having initially thought that a editor extension tool like perlstub would be very complex; how long did it end up? we wrote 80 lines of Perl code, nearly half of which are comments, and added one mildly tricky vi 'map' command! But that's tiny!.

Writing this web page explaining this has taken me approximately 3 hours, whereas the work of writing those 80 lines of Perl code and gluing it into vi actually took me about 30 minutes, back in 2007.

In the Second Part, we'll talk about Extending Perlstub to handle object oriented calls

d.white@imperial.ac.uk
Back to my Practical Software Development Top Page.
Written: Jan 2013