Download here (4D v6.0 windows version)

Introduction

One of the most powerful features of 4D is its ability to execute a line of 'built' 4D code in a compiled database. Such execution is achieved by passing a line of 4D code to the EXECUTE command as a text value. Although it is possible to call the EXECUTE command multiple times in order to execute several lines of code, such a technique will not cope with code that requires control flow statements such as the If, Case, For, While or Repeat statements.

This note documents a library of methods for executing multi-line 4D code, a 4D script, that can include control flow statements. We have named the library Script and hope that some of you find it useful.

Methods

The following methods are provided for the execution of 4D scripts:

Script Prepare (TEXT script; ->BLOB speedyscript)

Script Prepare changes a 4D script passed via the script parameter into a "speedy" script returned via the speedyscript parameter. Speedy scripts include additional information that is used to speed up the execution of scripts with control flow statements. Note: the 2nd parameter is a pointer to a blob, not a blob.

Script ExecuteBlob (->BLOB speedyscript) : TEXT

Script ExecuteBlob executes a speedy script. If the script sets the text variable Script_TextResult then the value of this variable is returned by Script ExecuteBlob. If Script_TextResult is not set by the script then Script ExecuteBlob returns an empty text value.

Script ExecuteText (TEXT script) : TEXT

Script ExecuteText executes a 4D script passed as a text value. It does this by calling Script Prepare to change the script into a speedy script and then by calling Script ExecuteBlob to execute the speedy script. The result returned from Script ExecuteText is as for Script ExecuteBlob.

Script ExecuteFile (TEXT filename) : TEXT

Script ExecuteFile executes a 4D script given in a textfile (defined by filename). If filename is an empty string, an Open File dialog is presented. The contents of the file are read into a text variable and passed to Script ExecuteText. The result returned from Script ExecuteFile is as for Script ExecuteBlob. If the textfile cannot be opened, then the error message "Error opening file: FILENAME" is returned by this method.

Script ExecuteRecord (TEXT scriptname) : TEXT

Script ExecuteRecord loads and executes a speedy script found in a table called Script. The speedy script is loaded by querying the Name field of the table with the passed scriptname parameter. The result returned from Script ExecuteRecord is as for Script ExecuteBlob. If the requested script does not exist in the table, then the error message "Unknown script: SCRIPTNAME" is returned. Note: both the Script table and this method can be omitted if desired.

In order to speed up the execution of scripts, scripts must first be "prepared" into a blob that includes additional information about the control flow statements in the script, i.e. their type and a link to the line in the script where a control flow statement may start, or end or continue etc. The Script Prepare method performs this pre-processing and the returned blob value is termed a speedy script. The general approach used to execute a speedy script follows the Fetch-Decode-Execute model used in classical von Neumann computer architectures. The script is executed line-by-line and if a control flow statement is encountered, the line's link information is available for branching to a different line, for example if the boolean test for a While statement yields false then we branch to the line that follows the matching End While statement.

Caveats

1. Scripts must be syntactically and semantically correct! If a script has errors, it's execution will most likely fail. The script library does not attempt to check passed scripts for errors nor does it install a default error handler.
   
   
2. Be wary of doing anything unusual within a script; with over 500 4D commands and functions you should expect that some will not work correctly within the context of an EXECUTE statement.
   
   
3. Scripts must not include local variables or parameters. The Script library does not handle local variables or parameters to scripts :-(
   
   
4. Process variables and interprocess variables "created" within a script are not visible to other 4D code. They can, however, be accessed by other scripts that execute later within the same process. In addition, interprocess variables (but not process variables) created within a script can be accessed by scripts that execute later in other processes.
   
   
5. Scripts can access variables declared in non-script 4D code. To avoid conflicts and unexpected interactions with non-script variables, ensure that all script variables are uniquely named, e.g. by prefixing them with a unique character sequence such as LOCAL_
   
   
6. Scripts must not call each other i.e. scripts must not call any of the methods listed above. Scripts can execute concurrently in different processes however, since no volatile interprocess variables are used by the script library.
   
   
7. Scripts must not exceed 32,000 characters (the limit of a 4D text value).
   
   
8. The script library is coded for English 4D scripts. If you use another language for 4D programming, change the Script_Parser method to handle the different control flow keywords and any syntactic differences.
   
   
9. EXECUTE statements are relatively slow. Scripts that execute a high number of statements will take a long time to finish.
   
   
10. The script library has not been used heavily and may have outstanding bugs. Our usage has been for un-anticipated, multi-line formulas in compiled databases, such as for multi-line formula columns in Quick Reports. If you encounter problems, consider the alternatives listed in the Alternative Implementations section below.

Installation

Currently provided as a V6 Windows database only. Download here

1. Copy and paste the script library methods from the Script database into your own database. If you use compiler methods, copy and paste the method Compiler Script as well.
   
   
2. The Script database includes an optional table called Script, which is used by the Script ExecuteRecord method. If you wish to use this method, create an identical Script table and optionally copy the Script table's Input and Output forms. The compiler method for these forms is Compiler ScriptForms. The table called Ignore is flotsam and should be ignored.
   
   
3. The Script_Execute method calls IDLE every 50 milliseconds (3 ticks). This ensures that processor time is given to other running 4D processes if 4D scripts are executed that do not callback the 4D engine. If you consider 3 ticks to be too little or too high for your database, change the value assigned to the interprocess variable <>SCRIPT_IDLE_EVERY in Script_DefineConstants.
   
   
4. For increased robustness, carry out exercise 1 below.

Some Exercises for the Reader

1. To increase robustness of script execution incorporate error-handling for the library using On Err Call. Consider adding support to abort execution of speedy scripts if some abort variable is set, and/or if some limit is reached, e.g. a time limit or a limit on the maximum number of script lines to execute.
   
   
2. Add support for local variables. One implementation approach would be for Script Prepare to rewrite all $ characters not in a comment or in a string to a special prefix character sequence, e.g. if LOCAL_ is the prefix, then we would change $var to LOCAL_var. This would essentially convert local variables into process variables. The prefix should be chosen to avoid clashes with process variables and should be short enough to support long local variable names.
   
   
3. Allow scripts to call each other. This requires that the process variables used in the script library be preserved across nested calls (e.g. in a local blob variable) or in 1D and 2D arrays. The latter would require a changing the library's scalar variables into 1D arrays, and the library's 1D arrays into 2D arrays. If local variables are also to be supported then local variable name conflicts would need to handled, e.g. by rewriting local variables with a unique prefix, e.g. LOCAL1_ for level 1 calls, LOCAL2_ for level 2 calls etc. This can be achieved by always calling Script Prepare before executing a nested script. More sophisticated solutions that minimise script rewriting are also possible.
   
   
4. Add support for parameters. This is tricky, but *might* be achievable by rewriting parameterised calls to a nested script, into assignments to process variables followed by the call to the script. For example, the call
   Script ExecuteRecord("MyScript"; "Hello"; 2/X; P->)
   could be changed by Script Prepare to:

        LOCAL_1 := "Hello"
        LOCAL_2 := 2/X
        LOCAL_3 := P->
        Script ExecuteRecord("MyScript")
   

   Within MyScript, $1, $2, $3 would need to be changed to LOCAL_1, LOCAL_2, LOCAL_3. This technique relies on the dynamic typing that 4D provides for interpreted code working within EXECUTEd code. Of course for nested scripts LOCAL_ should be LOCALlevelNo_paramNo.
   
   
5. Add support for optional execution of scripts in a separate process. This would isolate the script from the state of the calling process and may be desirable in some situations.
   
   
6. Provide a plug-in that provides access to 4D callbacks for tokenising lines and for executing tokenised lines. These callbacks are available to plug-in writers. Tokenised lines should execute faster than non-tokenised lines. Given such a plug-in, change Script_ParserEmitLine to tokenise lines and change Script_Execute to use the tokenised line execution callback instead of EXECUTE.
   
   
7. Code optimisations are possible. If you understand the source code, then have a go.

Some Exercises for ACI

1. Extend the EXECUTE command to support execution of multi-line 4D code.
   
   
2. Add to 4D Compiler the capability to produce compiled code libraries that can hold generic methods like those in Script. Allow such compiled libraries to be used in interpreted databases.

Finally

Good luck!

Revision History

1.2 (Mar 2000):

This note written. Many identifiers renamed. Compiler methods for script library added.

1.1 (Dec 1998):

Access to For loop variables in scripts changed to use Execute command. Version 1.0 accessed script For loop variables via pointers. Unfortunately 4D Compiler does not correctly handle pointers to variables-created-in-EXECUTEd-code.

1.0 (Dec 1998):

Written.

Alternative Implementations

4D Zine (www.4dzine.com) lists 2 alternative multi-line execution libraries:

1. ExecuteIt, by Aaron Braunstein, Keith Goebel and John Spencer. This is the most mentioned implementation on the 4D NUG mailing list.
   
   
2. Execute MultiLines, by Dave Merrill. Dave's implementation supports local variables and allows scripts to call other scripts.

Both the above implementations are older and probably more robust than Script, so do try them if you encounter problems with Script.