Follow-up to commit 08fc25d86
Testing:
$ export MMGEN_TEST_SUITE_DETERMINISTIC=1
$ script -c 'test/test.py -ne regtest' -O run1
$ script -c 'test/test.py -ne regtest' -O run2
$ scripts/gendiff.py run1 run2 > run.diff
Erratum:
The phrase “and the non-deterministic behavior of Bitcoin Core’s regtest
implementation” in the notes to commit 08fc25d86 is incorrect and should
be disregarded.
Affected coins: BTC,LTC,BCH
Fixes a regression introduced by commit `ca8b4dc31` on Aug 5.
During this period, use of the `mmgen-regtest` script would have overwritten
the user bitcoin/litecoin.conf and regtest dir. Other files in the daemon
datadir were unaffected.
Goal: for each test run to produce reproducible output, allowing us to check
for possible stochastic behavior in the scripts, as well as output-related
regressions (for example, garbage or improperly formatted output produced by a
bad format string) that might not be detected by the test scripts.
In practice, bugginess of the pexpect module and the non-deterministic behavior
of Bitcoin Core’s regtest implementation preclude completely identical output
from test run to test run, but the differences are small enough to result in an
easily reviewable diff.
Enable this feature by setting the MMGEN_TEST_SUITE_DETERMINISTIC environment
variable or running test/test-release.sh with the -D switch.
Examples:
$ script -c 'test/test-release.sh -FDv quick' -O run1
$ script -c 'test/test-release.sh -FDv quick' -O run2
# (optionally remove control characters from output files)
$ diff -u run1 run2 > diff
$ export MMGEN_TEST_SUITE_DETERMINISTIC=1
$ script -c 'test/test.py -ne main' -O run1
$ script -c 'test/test.py -ne main' -O run2
# (optionally remove control characters from output files)
$ diff -u run1 run2 > diff
Rationale of this commit: to relocate some ugly test-related code from the MMGen
package tree to the test tree, as well as to enable deterministic testing
(implemented in the next commit).
The overlay tree is a symlinked mirror of the MMGen package dir with a few
monkey-patched modules.
The monkey-patching is conditional, so the modules are certain to get tested in
their unpatched state as well.
