here we have a bugfixed hash table implementation that
passes libmem tests.
- let's profile iterate 10000.
- add -pg to CFLAGS and LDLIBS
- make clean all
- run ./iterate 10000
- runs as normal, but a bit slower than normal
  [profiling has to slow it down a bit!],
  produces gmon.out binary data file
- produce profiling report by:
  gprof ./iterate gmon.out > profile.orig

- examine profile.orig; hmm, the top entries are:

   %   cumulative   self              self     total           
    time   seconds   seconds    calls  us/call  us/call  name
 38.71      3.37     3.37    20000   168.37   206.96  hashFree
 22.92      5.36     1.99    10000   199.44   289.14  hashCopy
 11.29      6.34     0.98    10000    98.22    98.22  hashCreate
 10.31      7.24     0.90 325330000    0.00     0.00  copy_tree
  8.87      8.01     0.77 650660000    0.00     0.00  free_tree

- what leaps out?   650 million calls to free_tree() and
  325 million calls to copy_tree() are highly suspicious.

- I initially suspected the recursive free_tree() and copy_tree()
  structure and tried to optimize that.  made it slower!

- after 20 minutes thought, the explanation dawned:  hashFree() and
  hashCopy() (and hashForeach() in fact) have the same structure, iterating
  over the hash table's array of 32533 trees making one call to
  free_tree()/copy_tree()/foreach_tree() per tree.

- In the test program, the vast majority of these trees are empty.
  10k x 32533 is 325 million copy_trees.  free_tree() is called
  twice as many times because the test program create 10k hash
  tables and copies (clones) each one, freeing both copies.

- how long do nearly one billion function calls, 99.9% of which
  follow the "return null if t == null" path immediately, take?

- so I changed free_tree(t) to "if( t != null ) free_tree(t)"
  in hashFree(), and made analagous changes to hashCopy() and hashForeach().

- hash.c-diffs is the diff output showing the changes I made.

- recompile.  rerun.  reprofile:

  make
  ./iterate 10000
  gprof ./iterate gmon.out > profile.optimized

- program runs MORE THAN TWICE AS FAST overall!

- optimized profile reveals:

%   cumulative   self              self     total           
time   seconds   seconds    calls  us/call  us/call  name    
41.92      1.51     1.51    20000    75.66    75.66  hashFree
34.14      2.75     1.23    10000   123.26   123.26  hashCopy
24.15      3.62     0.87    10000    87.19    87.19  hashCreate

- we've roughly doubled the speed of hashFree() and hashCopy().

- note that hashCreate() and hashCopy() now take similar
  amounts of time.  previously hashCopy() was 2.5 times
  slower than hashCreate().

- further profiling-led optimizations might include reducing the
  number of trees (NHASH) to some smaller prime number, investigating
  how well the keys are spread among the trees.  Reduce NHASH too far
  and hash keys may start to cluster in particular trees which become
  too deep and unbalanced.  Other thoughts: is a tree rebalancer worth
  writing?  If so, when it is worth invoking?  Similarly, is a dynamic
  array resize and all-key rehash ever worthwhile?

- note that several of these investigations probably require storing a more
  realistic number of (key,value) pairs in the various hash tables.  Can
  we obtain a realistic (key,value) pair dataset from somewhere?
