MASSTEST: Automated validation/verification testing
In 2009 I joined UTIA and started working on the Apple-CORE project. The project was already in its second year, hence there had been already plenty of implementation work done. My first assignment was to prepare a test suite of simple assembly-level programs for validation of the UTLEON3 processor. The initial approach was quite naive, though: I produced a test program, observed that it does not run as intended, and reported a bug via e-mail to my colleagues. However, it quickly turned out that bugs were so proliferated that fixing one place often broke other things… Running validation tests had to be automated, and the MASSTEST was born.
MASSTEST is a collection of simple scripts written in Perl for running validation test suites. The main script is called mtest.pl. This script runs test-sets: first it compiles test programs using assembler/makefile toolchain and then it runs ModelSim VHDL simulator to execute the test program in a simulated UTLEON3. Simulation results are placed in dedicated directories. The second important script is called mcollect.pl: it gathers results from the directories, processes ModelSim output files (mainly the transcript) and generates summary tables.
The mtest.pl script is driven by a test-set configuration file. Configuration files contain lines that assign lists of values to parameters. For example:
# directories: software, hardware
prog_dir ../../utbench
hdl_dir ../../integration-V2/designs/utleon3-ml509-hwt03/_scripts
timeout 400
simscr do-sim-c.sh
# hardware parameters
hw-PCNT_KERNEL '1'
hw-RF_1W_PORT 1
# programs to run
program t07-ut t07-ut-unroll2 t07-ut-unroll4 t07-ut-swch t07-ut-unroll2-swch t07-ut-unroll4-swch
# software parameters
sw-BLOCKSIZE_1 2 5 7
sw-BLOCKSIZE_2 6
The parameters prog_dir, hdl_dir, timeout, and simscr specify system values for running the test. The parameter “program” specifies a list of programs that will be run; each program is a directory under “prog_dir”. The parameters that start with “hw-” are passed to VHDL simulator as generics of the top-level test-bench. The parameters that start with “sw-” are passed to compiler/assembler as macro expansions.
Each parameter is really a list of space-separated values. The important thing about MASSTEST is that it creates Cartesian product of the configuration parameters: in the script above all combinations of the parameters ‘program’ and ‘sw-BLOCKSIZE_1’ will create an independent test run.
Often times, however, I don’t need the full cartesian product of several parameters. Indeed, as individual simulation runs can take tens of minutes it becomes too expensive to run so many tests. Therefore, bundled parameters were devised:
hw-TECHNO;hw-SPEED;hw-dfuclkspeed 1;A3M2;1.0 1;A4M3;1.5 1;A5M3;1.66 1;A6M3;2.0 3;A2M3;1.0 3;A4M4;2.5
cmd-WLEN 4 8 16 32 64 128 196
Bundled parameters are separated by semicolon. The example above bundles three parameters ‘hw-TECHNO’, ‘hw-SPEED’, and ‘hw-dfuclkspeed’ and specifies a set of value triples that are assigned (or iterated) together. The other parameter ‘cmd-WLEN’ on the other hand is not bound to those three; hence, MASSTEST will create Cartesian product of the triples and the individual values in ‘cmd-WLEN’.
Even though all test-runs generated and executed in MASSTEST are independent to each other, parallelization of the runs is not so easy. The problem is that program (software) and design (hardware) directories are re-used for each run. Indeed, VHDL compilation for ModelSim takes time and disk space and redoing it for each test-run is wasteful.
So, parallelization is done differently. You first decide on the number of concurrent runs that should be executed together. Usually this is the number of processors in system, typically four. The program and design directories must be replicated four times and specified in the configuration file using a bracket notation:
# directories: software, hardware; replicated four times.
prog_dir[0] ../../utbench_0
prog_dir[1] ../../utbench_1
prog_dir[2] ../../utbench_2
prog_dir[3] ../../utbench_3
hdl_dir[0] ../../integration-V2/designs/utleon3-ml509-hwt03-0/_scripts
hdl_dir[1] ../../integration-V2/designs/utleon3-ml509-hwt03-1/_scripts
hdl_dir[2] ../../integration-V2/designs/utleon3-ml509-hwt03-2/_scripts
hdl_dir[3] ../../integration-V2/designs/utleon3-ml509-hwt03-3/_scripts
This tells MASSTEST that there are four independent copies of program and HDL design directories. Then you specify that the parallelization should be done over those four replicas:
parmodulo 4
Configuration scripts using parallelization need to be run using mptest.pl, not mtest.pl. The mptest.pl is a wrapper that forks off four instances (or any number specified in parmodulo) of mtest.pl and–this is important–tells each instance ‘i’ that it should actually execute only i-th test modulo four. All instances iterate over the whole test space; but the first instance executes tests 0, 4, 8,…, the second instance tests 1, 5, 9, …, and so on. The test runs are interleaved over the instances.
Finally, it is also possible to parallelize test runs over a cluster of computers. This requires that computers in cluster have a unified filesystem view, i.e. a file name on one computer refers to the same object on the other. In practice this means having a shared network disk mounted in the same place in all nodes. Secondly, remote command execution is facilitated by SSH with public key password-less authentication. MASSTEST configuration scripts are extended with a new “remote” directive:
remote[0] blue03
remote[1] blue03
remote[2] blue04
remote[3] blue04
The “remote” directive specifies cluster host names where the mtest.pl instances should be delegated to. The parmodulo mechanism is used to break test-runs into independent groups, and the mtest.pl for each group is run in the remote host via ssh.