runregress - Runs the regression suite and generates reports
runregress [options] configuration_name [options]
runregress [options] -setup_generate|-setup [options]
runregress -help|-h|-?
runregress -listreports|-lr
#option('name', value);
to each query.
run
, roxieconfig is executed in RunECLFile mode. If yes
, roxieconfig is executed in DeployECLFile mode and then testsocket is run. Otherwise, testsocket is run (the queries are assumed to be deployed already). Defaults to yes
.
.log
appended, or to setup_generate.log
where that argument is omitted by the option -setup is used.
yes
; for roxieconfig when deploy_roxie_queries is run
or for testsocket, the output contains the results and is captured).
local.ini
and a non-optional file regress.ini
, with local.ini
taking priority (allowing users to set local overrides or additions to a centrally deployed base configuration). See CONFIGURATION for more details.
All of these may be used (in their long forms) as configuration variables too (see below). A configuration variable overrides the built-in default, but is overridden by a command line option.
A note on space-separated lists. To use these on the command line, you will need to prevent your shell from interpreting the space as a break between options (normally by escaping it with a backslash or enclosing the value in quotes). Using the alternative of comma-separated lists may be preferred. The option of space-separated lists is mainly intended when giving the value in a configuration variable instead.
Configuration for the runregress tool is via configuration files (see OPTIONS above). In each file, values are specified in a block corresponding to the specified configuration name, and optionally in a global block named *
. The blocks are started by [configuration_name]
and [*]
, and the values are specifed as name=value
. Earlier named configuration files take priority over later ones; within each file, the named block takes priority over the global *
block. The following configuration variables are understood. Note that additional configuration files for the tools invoked will be read, e.g. eclplus.ini
.
If you specify the option -setup_generate (aka -setup) then you can omit the name of the configuration to use. In that case, only global blocks are loaded, which is okay provided every variable required (including setup_clusters) is specified in the global blocks.
Note that, although the global and named blocks are the only ones loaded, the other blocks in the configuration files will be scanned. This is done to determine the types of the clusters listed in the configuration variable setup_clusters. It is an error for different configurations to specify different types for the same cluster. If a cluster is only ever used for setup (i.e. no configuration names it as its active cluster) it is assumed to be thor (and a warning is issued).
yes
or run
.
run
.
yes
nor run
.
yes
nor run
.
move
, then the previous suite directory is moved to a .bak
directory at the start of a run, instead of being removed. This can be used to compare two result sets, or to back up the previous result set in case of problems. Note that only one back up is kept: mess up twice (with restoring or backing up yourself) and you will have lost your results.
Additionally, configuration variables with the same names as any the long forms of the command line options may be used, with the same effect. These provide defaults, which are overridden by the command line options if present. These include: suite, setup_generate, parallel_queries, deploy_roxie_queries, report, and logfile. For on/off command line options (setup_generate), turn on using a non-empty, non-zero string.
By default, this tool prepares a suite of test queries, runs it, and generates reports on the results. If the -norun option is given (or the setup_generate config value is non-empty), the first two stages are omitted, so that reports can be generated for a previous run suite. If the -setup options is given, the tool prepares and runs a suite of queries which generate datasets and indexes used in the test queries, and reports on the success of these queries. If the -listreports option is given then the tool just lists the available report formats and exits. If the -help option is given then the tool displays its documentation and exits.
The queries and results go into a directory named for the suite. In a test run, the suite name is specified by the -suite option if used, otherwise it the configuration name is used. In a setup run, the suite is always called setup_generate
.
By default, if the suite directory exists then it is removed at the start of a new run. The purge configuration variable changes this behaviour.
During a setup run, queries are taken from the setup
subdirectory of the test directory. All queries are used, in four variants (regular, local, payload, and varload) for each setup cluster.
During a test run, queries are taken from the test directory, and from the subdirectory named for the cluster type. All queries are considered unless the -query option is used. Some queries may be skipped, and some queries may appear in several variants, as described in QUERIES, VARIANTS, AND SKIPPING (below).
During a test run, key results are taken from the key
subdirectory of the test directory, except that when results exist in a key subdirectory of a type or os subdirectory those are used instead (the type subdirectory taking priority).
During a setup run, the queries go into a setup
subdirectory of the suite directory. During a test run, they go directly into the suite directory, and key results go into the key
subdirectory.
Four additional files are created in the suite directory.
manifest.csv
skipped.csv
tbd.csv
skipped.csv
, but only includes those queries which are skipped because of lines flagged TBD
.
wuids.csv
settings.ini
All queries generated during the preparation stage are executed. For roxie queries, roxieconfig and/or testsocket are used, as described under the deploy_roxie_queries configuration variable above. For other queries, eclplus is used, unless an alternative has been specified by the eclplus configuration variable. Arguments can be altered using various configuarion variables.
During a setup run, the results go into the setup
subdirectory of the suite directory. During a test run, they go into a out
subdirectory.
One additional file is created in the suite directory: wuids.csv
is a CSV file which lists the queries which have been run, and has three fields: the query name (blank indicates that this query does not have multiple variants), and the WUID (blank indicates that the WUID could not be determined). It should be possible to determine the WUIDs for non-roxie, provided eclplus outputs it as expected; roxieconfig does not report the WUID (and it would be less useful anyway).
This script can generate any number of arbitrary reports, provided by perl modules. These reports are based only on configuration data and information from files in the suite directory: the output, the key results, and the files manifest.csv
, skipped.csv
, and tbd.csv
. Because of this, the report generation stage can be done separately from the preparation and running stages, using the -norun option. The -report option or report configuration variable determines which reports are generated.
Possible things a report could do include: describing the results on standard output; generating a file showing the results (e.g. in HTML); or invoking an external diff tool (e.g. Beyond Compare or xxdiff). The default behaviour is to summarize the result on standard output.
During a setup run, the values of the -report option and report configuration variable are ignored: the script always just summarizes the results. Because there is no key output for setup queries, all it does is check the output for errors.
If the ECL contains a comment //UseStandardFiles
, the query is generated in a variant for each setup cluster; or, on roxie, two variants, one using dynamic filenames. If it contains a comment //UseIndexes
, it is generated in several variants for each setup cluster: there is a regular one, a payload one, a varload one (using blobs), and for multipart indexes (where the setup cluster was thor rather than hthor) a local one (using noroot
); except when running on thor, these each also come in a trans variant (using record layout translation); and, on roxie, these each also come in a dynamic filename variant.
A comment //class=...
in the ECL puts the query in a class. If the -class
option is used then queries not in that class are skipped. Otherwise, queries in any class are skipped.
Queries with a comment //nohthor
, //nothor
, or //noroxie
are skipped when the type
configuration variable has the appropriate value.
Queries with a comment //nowindows
, or //nolinux
are skipped when the os
configuration variable has the appropriate value.
If the query contains a comment //nolocal
, the local variant is skipped.
More generally, a query can be skipped using a comment starting //skip
, or some variants can be skipped using a comment starting //varskip
. In either case, the rest of the line consists of a condition, or multiple conditions combined by &&
. Each condition may be name==value
(tests for equality), name!=value
(tests for inequality), name
(tests for a true value), or !name
(tests for a false value). In //skip
, the tests may be against any configuration variable (including type
, os
, etc.). In //varskip
, tests may be against any configuration variable or against local
, payload
, varload
, trans
, and dynamic
(which have true values if this variant has the named properties) or setuptype
(which gives the type of the cluster used for setup (assumes thor if not known)). So, for example, //skip type==thor
is equivalent to //nothor
, and //varskip type==roxie && setuptype==thor && !local
skips non-local variants running against thor indexes on roxie.
The characters TBD
may be added at the end of any line specifying a skip. That will cause this skip to be displayed under a separate (and more prominent) section in reports.
It is possible to apply filters to the key and output results, to normalize them in some way. Currently, one such filter is available, to round floating point values: a comment //normalizeFP 10
causes all floating point values spotted in the result set to be rounded to 10 d.p. (and anything within 10^-10 of zero to be replaced by 0, to fix the -0 problem).
This script runs eclplus, roxieconfig, and testsocket as child processes. There are four possible outcomes: the process terminates normally with a zero return status (success); the process terminates normally with a nonzero return status (failure); the process was terminated by a signal; or the process failed to start at all (e.g. the executable was not found). The intention is that the all of these except success will be logged as a warning to standard error, and that none of them should prevent the script continuing.
This intention is not quite achieved on Win32, which does not have real signals. It seems that using pskill to terminate a child process (with or without the -c option) abruptly kills the script itself; while using the windows task manager to terminate a child process correctly allows the script to continue, but incorrectly appears as if the process had terminated with a return status of 1. As far as I know, this behaviour is outside our control (it is in the perl module the IPC::Run manpage).
Some report types (e.g. BC2, xxdiff) launch diff viewers as child processes. The script starts these but does not wait for them to finish, and so does not care if they receive signals or return a nonzero status. A failure to start will be logged as a warning to standard error.