Creating a Behavior-Driven Development infrastructure for AhatSkin

Motivation

perl

An odd thing has happened in the perl world in the past five (or so) years. We've stopped providing a road map from there (newbie) to here (true expert). There's a missing link, and that missing link has to do with perl module tech. The write-ups the official perl docs and the tutorials out there don't seem to have kept up to date.

[expand]

AhatSkin

As the AhatSkin prototype became more mature, I found the development environment for skins provided by the twiki and even the foswiki fork of twiki completely inadequate. I wanted the behavior-driven development environment to which I have become addicted in the PerL world, with the smooth integration of ideas and journaling to which I have become addicted in the PyThon (especially SciPy) world, with Neal Ford (The Productive Programmer) style integration.

Yup, I'm greedy. But then, I'm perfectly happy to build my own toys

I took a look at a number of bits 'n pieces I had lying about, and started work on a true AhatSkin development tool.

• AhatTWikiInteractionDesign
• OnDBCacheContrib
• TWikiDevEnvironment
• Some Thoughts on Tools and Toolkits
• SkinTemplateLayout

agile software development

[to be written]

Major Components

(esp. how they work in combination, but why it's important that they continue to be developed orthogonally...)

Module::Install

Module::Install is a truly elegant architecture, not to mention extremely perl-ish (makes things easy for experts to do - and even do according to best practices - while still being highly orthogonal.) Modules (and distributions) developed with Module::Install can use Module::Install to bootstrap whatever environment the distribution needs to build itself onto the target machine temporarily. Using differing approaches to configuration, testing, etc., is simple with Module::Install, although not obvious.

Probably the next most important thing to know about Module::Install is that the only reliable documentation for the core distribution is in the Changes files. I sorted this fact out several weeks into working with Module::Install; had I known it earlier, it would have saved me no end of grief. (see below for specific notes)

I have not found the lack of Module::Build support in itself to be a problem, as I have always found better solutions elsewhere than those provided by Module::Build. In contrast, what I found to be the problem was:

1. the documentation for Module::Install, in combination with
2. no roadmap to alternate solutions.

In classic perl tradition, having spotted a problem, I'm trying to do my part to fix it. which is what this discussion (and the accompanying distribution) is all about.)

Module::Install notes

• Module::Build is currently not supported. If you trace Module::Build support, you'll find that &Build->write was initially subsumed into WriteAll; however, maintaining support for two back ends was finally deemed not viable.
• There's an undocumented (as of 4/21/2009) realclean_files command that takes a string of files to add to the make realclean target. Very useful! I cannot image why it isn't documented.
• If you use PL_FILES, be aware that PL files which produce modules are built (as of May 17, 2009) in the wrong order, in contrast to the DWIM (Do What I Mean) behavior that we are accustomed to from !ExtUtils::MakeMaker.
  • A better answer is to use a perl standard configuration approach, such as AppConfig (see configuration discussion)

Test::Base

Exception::Class

A first pass

Orienting myself

I'd been expanding my skills in some other areas for the previous 18 to 24 months, and had not undertaken a new distribution in that time, so I needed to catch up on the latest in perl module tech. After sorting through what goes with what a bit (see AuthoringPerlModules), I settled on module-starter with Module::Install.

Creating a default configuration

Following Larry Wall's LazinessPrinciple, I setup a configuration file in my account for module-starter.

• I stayed with the default location: $HOME/.module-starter/config,
• format of an entry is

       <name-value-pair> ::= name: value <value-list>
       <name-value-pair> ::= name: value
       <value-list> ::= value <value-list>
       <value-list> ::= value

• example:

  author: Hilary J. Holz
  email: hholHYRUQAPZz@cpan.org
  builder: Module::Install

Creating the default development environment

• With the default configuration specified in the previous step, running

  module-starter --module=InstallAhatSkin

• created

  name	type	description
  Changes	<file>	log of changes
  lib	<directory>	where the development module(s) go
  lib/InstallAhatSkin.pm	<module>	module template
  Makefile.PL	<thingy>	cool simplified makefile in Module::Install syntax
  MANIFEST	<text file>	list of files in distribution
  README	<text file>	readme file template
  t	<directory>	where the test scripts go
  t/00-load.t	<test>	default test (does the module load?)
  t/boilerplate.t	<test>	default test (did you replace the boilerplate text in the template with your own?)
  t/pod-coverage.t	<test>	default test (is there pod documentation for all the parts of your code?)
  t/pod.t	<test>	default test (are the pod docs syntactically correct?)

• The Makefile.PL looked like this:

  use inc::Module::Install;
  
  name     'InstallAhatSkin';
  all_from 'lib/InstallAhatSkin.pm';
  author   'Hilary J. Holz <hhoHYRUQAPZlz@cpan.org>';
  license  'perl';
  
  build_requires 'Test::More';
  
  auto_install;
  
  WriteAll;

Customizing the default environment

Basics

1. I'll add some comments to the Makefile.PL (duh!)
2. I'll specify the current version of packages as the minimum versions for my dependencies. As I go along, I may be able to refine this information, but it's better practice to specify current versions as minimum versions than give no information at all.
3. I like Module::Install::DSL's syntactic sugar, because I think it improves the CoderSpeak quality of Makefile.PL. I'll go ahead and change the Makefile.PL syntax, you can judge for yourself. I later ended up dropping this change, because far too much of the work concerning Module::Install is undocumented and it was not worth the extra work to sort out undocumented interactions in undocumented systems. Hey, folks, this is perl. How about writing the docs first?

   # HJH, 3/31/2009, hhoHYRUQAPZzlz@cpan.org
   # all versions are minimum versions on which system has been tested
   # please let me know if you have success with earlier versions
   use inc::Module::Install::DSL 0.81;
   
   # the principal Module
   name            InstallAhatSkin
   
   # not quite all_from, since I specify perl_version
   abstract_from  lib/InstallAhatSkin.pm
   author_from    lib/InstallAhatSkin.pm
   version_from   lib/InstallAhatSkin.pm
   license_from   lib/InstallAhatSkin.pm
   perl_version   5.0088
   
   # build_requires allows flexibility in installer process
   build_requires Test::More 0.86
   
   auto_install
   WriteAll

Developing a script

InstallAhatSkin is has a command line (script) interface. I'm going to follow the common practice of putting the bulk of the intelligence in module(s) and providing an additional minimal script interface to the module(s).

• brian d foy's modulinos (see Chapter 18) are an elegant alternative. My only objection to brian's modulinos is that he doesn't stop being clever (see excellent discussion of cleverness in Perl Best Practices) with the modulinos themselves, but gets clever about all sorts of stuff. In my copious free time (tm), I'd like to come back and build a modulino version of InstallAhatSkin with the standard devel environment and write that up as well.

To develop a script interface to our module, we need to add:

1. an install_script command to Makefile.PL
2. an interface that integrates with Test::More to test command line invocation of the script.
   1. I'll use Test::Script to test that the script compiles;
   2. and Test::Output to test that the script produces the output we expect under various invocations.
   3. There's also Test::Cmd, however, the most recent CPAN release of Test::Cmd (1.05) at the time I wrote this dated back to September 2001. Not a good sign
3. and a script directory to my development environment (only showing modifications):

   name	type	description
   Makefile.PL	<thingy>	coderspeak makefile in Module::Install::DSL syntax
   script	<directory>	development directory for the installahatskin script
   script/installahatskin	<script> (duh)	the actual script itself

The revised Makefile.PL looks like this:

# HJH, 3/31/2009, hhoHYRUQAPZzlz@cpan.org
# all versions are minimum versions on which system has been tested
# please let me know if you have success with earlier versions
use inc::Module::Install::DSL 0.81;

# the principal Module
name            InstallAhatSkin
# the command line script
install_script  installahatskin

# not quite all_from, since I specify perl_version
abstract_from  lib/InstallAhatSkin.pm
author_from    lib/InstallAhatSkin.pm
version_from   lib/InstallAhatSkin.pm
license_from   lib/InstallAhatSkin.pm
perl_version   5.0088

# build_requires allows flexibility in installer process
build_requires Test::More 0.86

# required for testing only
test_requires   Test::Script        1.03
test_requires   Test::Output        0.13

auto_install
WriteAll

Addressing coding standards

Now let's address coding standards. First, we'll incorporate Perl::Critic, which lets us use a variety of policy modules to analyze our code, via Test::Perl::Critic. One excellent quality of Perl::Critic is that we can pick and choose what to follow, or swap in our own policies. For example, I often end up flagging a particular block of code as an intentional violation of the 'no strict refs' policy while generating code with AUTOLOAD. Among the major benefits of using Perl::Critic is that it keeps both our skills and our codebase up to date in the natural course of work and play. Excellent!

1. add to Makefile.PL

   test_requires  Test::Perl::Critic 1.01

2. t/critic.t

   #!perl -T
   
   use Test::Perl::Critic;
   all_critic_ok();
   

Tracking community standards

Next step, kwalitee testing. Yeah, yeah, kwalitee != quality. (It can be reverse engineered, defeated, many situations exist in which there's a good reason not to do the usual thing, etc., etc.) But that isn't the point of kwalitee. And if you really find yourself ranting about kwalitee, then it's probably time to take a vacation. When you get back, (re?)read Larry's Diligence, Patience, and Humility.

1. Ideally, I would add to Makefile.PL

   test_requires Test::Kwalitee 1.01

   , however, at the time I wrote this, Module::CPANTS::Analyse, upon which Test::Kwalitee depends, caused problems with taint checking. So I added a commented out line to the Makefile.PL, and handled the dependency question in the test.
2. t/kwalitee.t

   #!/usr/bin/perl
   use Carp;
   
   eval { require Test::Kwalitee; };
   if ($@) {
       croak "failed to compile: $@\n";
   }
   Test::Kwalitee->import();

Note that I've dropped the taint check -T in the invocation of perl in the test, as well. The irony of the taint checking being the problem is too funny, but let's not go there, shall we?

A closing snapshot

Let's take a moment to review the environment before we start to look at behavior-driven development of the tests for the script (and module) in earnest.

• I ended up having to switch back to inc::Module::Install from inc::Module::Install::DSL because there is simply too much of Module::Install that is undocumented. The added layer of having to divine how inc::Module::Install::DSL interacts with other undocumented materials was more hassle than the benefit of the syntactic sugar promised.

• The revised Makefile.PL looks like this:

  # HJH, 3/31/2009, hhoHYRUQAPZzlz@cpan.org
  # all versions are minimum versions on which system has been tested
  # please let me know if you have success with earlier versions
  use inc::Module::Install 0.81;
  
  # the principal Module
  name            "InstallAhatSkin";
  # the command line script
  install_script  "installahatskin";
  
  # not quite all_from, since I specify perl_version
  abstract_from  "lib/InstallAhatSkin.pm";
  author_from    "lib/InstallAhatSkin.pm";
  version_from   "lib/InstallAhatSkin.pm";
  license_from   "lib/InstallAhatSkin.pm";
  perl_version   "5.0088";
  
  # build_requires allows flexibility in installer process
  build_requires "Test::More" => "0.86";
  
  # required for testing only
  test_requires   "Test::Script"       => "1.03";
  test_requires   "Test::Output"       => "0.13";
  test:requires   "Test::Perl::Critic" =>  "1.01";
  # taint checking broken with Test::Kwalitee under Module::CPANTS::Analyse 0.81
  # test_requires  'Test::Kwalitee'   =>  '1.01';
  
  auto_install;
  WriteAll;

Other nice ideas

• using only to specify the required version numbers robustly, but the module seems to not have a current maintainer

Measuring code coverage

What is going on with code coverage is even more poorly documented than how to create modules. I thrashed around for quite some time, finding partial solutions, presumably inserted by my intrepid predecessors in this quest. I discovered:

• If you simply include

  use ExtUtils::MakeMaker::Coverage 0.05;

  in your Makefile.PL, the resulting Makefile will contain a testcover target. Note that you do not need a MY::postamble in your Makefile.PL, which is good, as determining how to achieve the analogous effect would have taken a lot of work. (There just doesn't seem to be any roadmap to converting to using Module::Install when you have an ongoing real need to be producing systems that push the design edge. That's too bad. I may end up switching over to ExtUtils::ModuleMaker, which is far less elegant, but far, far friendlier.) Right. Back to the main theme. This discovery, cool as it might be, did me no good, as make testcover doesn't help me, at least not without the ability to extensively customize the testcover target, which ExtUtils::MakeMaker::Coverage does not. Why? Because ...
• I need to run the tests under prove with Devel::Cover turned on in order to get coverage on the script interface.
• Conversely, the cover program now contains the equivalent of the old testcover target (cover -test), so in many cases, it is no longer necessary to integrate Devel::Cover directly into the Makefile. This solution still did not solve my problem of needing to run prove rather than make test.

A strawman solution

I installed Johan Lindström's Devel::CoverX::Covered module, and wrote the following testcover shell script, based on Johan's synopsis

#!/bin/sh
#
# HJH, April 5, 2009
#   see also Johan Lindstrom's Devel::CoverX::Covered
#   at http://search.cpan.org/perldoc?Devel::CoverX::Covered
#
# erase old coverage database
cover -delete

# run test suite with coverage turned on
#   --blib add blib to path
#   --recurse to walk all of the test directory
#   --ignore-exit lets you use pod2usage and the like
#   --verbose because coverage runs take forever anyway...
PERL5OPT=-MDevel::Cover prove --blib --recurse --verbose --ignore-exit

# now go back and pick up info on script
covered runs

# output report
#   -ignore switch means report won't include coverage on prove, perldoc, etc.
cover -ignore "/usr/bin/" -outputdir /var/www/html/ahatskin/installahatskin 

This script got me a set of working code coverage reports, which I could then use to bootstrap myself into a more coherent, integrated solution.

Behavior-driven development

I prefer behavior-driven development to test-driven development as a result of having spent time teaching the technique over several years. The reason it's such a good thing (tm) to write the tests before writing the code is because the tests one is writing are tests for desired behavior. All sorts of spiffy results ensue:

• the tool you end up building actually does what was it was originally intended to do, because that vision was captured in the behavioral tests
• what's more, all the interesting subtle details about the real world (tm) we all have to live in have been captured along the way in the narrative documentation and the additional behavioral and contextual tests you have continued to accumulate.
• those tests are exportable to future projects concerning related problems (that's called a test library). That phenomenon is what patterns are all about. So behavioral tests are the key. Calling a test framework a unit test framework is like calling a car a four wheel sedan. So? Does it support behavior-driven development? PyThon's nose framework is a thing of beauty - it even does test discovery from the internal documentation. Talk about BDD! Calling it a unit test framework is almost heresy.

Test the synopsis (Synopsis the test?)

So let us start there. Write your synopsis describing how the thing should be used and turn that into a test. In our case, that will give us two tests: one for the script and one for the underlying module that will do the work for the script. That also means that we now need to setup and tear down a testing environment.

Years of experience with BDD has taught me that the setup/tear down process needs to be as orthogonal as possible from the tests themselves, in order to support the development of tests that are, themselves, orthogonal.

Managing tests using Test::Base

Test::Base is a (really pretty piece of work!!) ahem a test framework that has been integrated into Module::Install. It is:

• data driven, meaning that it has very nice capabilities for organizing test cases in their own files, facilitating developing and keeping tests orthogonally
• built upon Spiffy, with some nice additional features, to make it particularly easy to subclass Test::Base

a custom framework

I built InstallAhatSkinTestFramework? .pm

A better testcover solution

I converted my shell script to a perl script and used the Test::Base subclass I developed in

Exception handling

1. Perl's built-in eval and $@ system.
2. A. Kumar's Error module, has Java style "try, catch, except, finally, yadayada".
3. Exception::Class, which is what I ended up coming back to and using.
4. A whole bunch of stuff intended to marry Error and Exception::Class together, or to add some part of "try, catch, yada" syntax to Exception::Class

Of the above, Exception::Class is very a perl-ish addition to the built-in stuff.

References

Books

• Perl Testing: A Developer's Notebook, by Ian Langworth & chromatic
• Mastering Perl, by brian d foy
• Perl Best Practices, by Damian Conway - An important contribution to the perl community and a truly wonderful book in many ways, however, a number of his most ambitious ideas do not actually work in practice as he thought they might. Damian ran into some family problems and did not get to carry through some of the development he hoped to finish for the book, so Getopt::Clade, for example, does not exist.

Modules

• module-starter
• Module::Install
• Module::Install::DSL
• Module::Install::MakeMaker
• Module::Install::Share
• Module::Install::Build
• ExtUtils::MakeMaker::Coverage

• Test::Base
• Test::Perl::Critic and Perl::Critic
• Test::Kwalitee
• Test::Script
• Test::Output
• Test::Cmd

• Devel::Cover
• Devel::CoverX::Covered

• AppConfig
• YAML::AppConfig
• File::ShareDir
• Spiffy
• only