minor fixes, cleanups and additions

2020-03-12 16:38:02 +00:00 · 2020-03-12 16:38:02 +00:00 · 853a24df21
commit 853a24df21
parent c7d15bcacf
18 changed files with 277 additions and 191 deletions
--- a/README.md
+++ b/README.md
@ -80,9 +80,9 @@ standard.
 - **[BIP69 transaction input and output ordering][69]** helps anonymize the
  “signature” of your transactions.
 - **[Full control over transaction fees][M]:** Fees are specified as absolute or
-  sat/byte amounts and can be adjusted interactively, letting you round fees to
-  improve anonymity.  Network fee estimation, [RBF][R] and [fee bumping][B] are
-  supported.
+  satoshi/byte amounts and can be adjusted interactively, letting you round fees
+  to improve anonymity.  Network fee estimation (with selectable estimation
+  mode), [RBF][R] and [fee bumping][B] are supported.
 - **Support for nine wallet formats:** three encrypted (native wallet,
  brainwallet, incognito wallet) and six unencrypted (native mnemonic,
  **BIP39,** mmseed, hexseed, plain hex, dieroll).
@ -99,7 +99,7 @@ standard.
  splits with a single master share.
 - **[Transaction autosigning][X]:** This feature puts your offline signing
  machine into “hands-off” mode, allowing you to transact directly from cold
-  storage securely and conveniently.  Additional LED blinking support is
+  storage securely and conveniently.  Additional LED signaling support is
  provided for Raspbian and Armbian platforms.
 - **[Password generation][G]:** MMGen can be used to generate and manage your
  online passwords.  Password lists are identified by arbitrarily chosen strings
@ -113,9 +113,8 @@ standard.
 - **Wallet-free operation:** All wallet operations can be performed directly
  from your seed phrase at the prompt, allowing you to dispense with a
  physically stored wallet entirely if you wish.
- **Stealth mnemonic entry:** To guard against acoustic side-channel attacks,
-  you can obfuscate your seed phrase with “dead” keystrokes as you enter it from
-  the keyboard.
+- **Stealth mnemonic entry:** This feature allows you to obfuscate your seed
+  phrase with “dead” keystrokes to guard against acoustic side-channel attacks.
 - **Network privacy:** MMGen never “calls home” or checks for upgrades over the
  network.  No information about your wallet installation or crypto assets is
  ever leaked to third parties.
@ -124,10 +123,11 @@ standard.
 - **Terminal-based:** MMGen can be run in a screen or tmux session on your local
  network.
 - **Scriptability:** Most MMGen commands can be made non-interactive, allowing
-  you to automate repetitive tasks using shell scripts.  Most of the
-  `mmgen-tool` utility’s commands can be piped.
- A convenient [**tool API interface**][ta] allows you to use MMGen as a crypto
-  library for your Python project.
+  you to automate repetitive tasks using shell scripts.
+- The project also includes the [`mmgen-tool`][L] utility, a handy “pocket
+  knife” for cryptocurrency developers, along with an easy-to-use [**tool API
+  interface**][ta] providing access to a subset of its commands from within
+  Python.

 #### Supported platforms:

@ -205,3 +205,4 @@ Donate (BTC,BCH): 15TLdmi5NYLdqmtCqczUs5pBPkJDXRs83w
 [ms]: https://github.com/mmgen/mmgen/wiki/seedsplit-[MMGen-command-help]
 [ta]: https://github.com/mmgen/mmgen/wiki/Tool-API
 [ts]: https://github.com/mmgen/mmgen/wiki/Test-Suite
+[L]: https://github.com/mmgen/mmgen/wiki/tool-[MMGen-command-help].md
--- a/data_files/mmgen.cfg
+++ b/data_files/mmgen.cfg
@ -1,9 +1,9 @@
 # Configuration file for the MMGen suite
-# Everything following a '#' is ignored
+# Everything following a '#' is ignored.

-################
-# User options #
-################
+##################
+## User options ##
+##################

 # Uncomment to suppress the GPL license prompt:
 # no_license true
@ -26,16 +26,16 @@
 # Set the RPC host (the host the coin daemon is running on):
 # rpc_host localhost

-# Set the RPC host's port number
+# Set the RPC host's port number:
 # rpc_port 8332

-# Uncomment to override 'rpcuser' from coin daemon config file
+# Uncomment to override 'rpcuser' from coin daemon config file:
 # rpc_user myusername

-# Uncomment to override 'rpcpassword' from coin daemon config file
+# Uncomment to override 'rpcpassword' from coin daemon config file:
 # rpc_password mypassword

-# Uncomment to set the coin daemon datadir
+# Uncomment to set the coin daemon datadir:
 # daemon_data_dir /path/to/datadir

 # Set the default hash preset:
@ -62,9 +62,10 @@
 # Set the maximum input size - applies both to files and standard input:
 # max_input_size 1048576

-###################
-# Altcoin options #
-###################
+
+#####################
+## Altcoin options ##
+#####################

 # Set the maximum transaction fee for BCH:
 # bch_max_tx_fee 0.1
@ -75,10 +76,10 @@
 # Set the maximum transaction fee for ETH:
 # eth_max_tx_fee 0.005

-# Set the Ethereum mainnet name
+# Set the Ethereum mainnet name:
 # eth_mainnet_chain_name foundation

-# Set the Ethereum testnet name
+# Set the Ethereum testnet name:
 # eth_testnet_chain_name kovan

 # Set the Monero wallet RPC host:
@ -90,9 +91,10 @@
 # Set the Monero wallet RPC password to something secure:
 # monero_wallet_rpc_password passw0rd

-#####################################################################
-# The following options are probably of interest only to developers #
-#####################################################################
+
+#######################################################################
+## The following options are probably of interest only to developers ##
+#######################################################################

 # Uncomment to display lots of debugging information:
 # debug true
--- a/mmgen/altcoins/eth/contract.py
+++ b/mmgen/altcoins/eth/contract.py
@ -138,7 +138,7 @@ class Token(MMGenObject): # ERC20
 			die(3,m.format(from_addr,tx.sender.hex()))
 		if g.debug:
 			msg('TOKEN DATA:')
-			pmsg(tx.to_dict())
+			pp_msg(tx.to_dict())
 			msg('PARSED ABI DATA:\n  {}'.format('\n  '.join(parse_abi(tx.data.hex()))))
 		return hex_tx,coin_txid

--- a/mmgen/baseconv.py
+++ b/mmgen/baseconv.py
@ -184,7 +184,7 @@ class baseconv(object):
 				die(2,'{}: invalid length for Monero mnemonic'.format(len(words)))

 			z = cls.monero_mn_checksum(words[:-1])
-			assert z == words[-1],'{!r}: invalid Monero checksum (should be {!r})'.format(words[-1],z)
+			assert z == words[-1],'invalid Monero mnemonic checksum'
 			words = tuple(words[:-1])

 			ret = b''
--- a/mmgen/bip39.py
+++ b/mmgen/bip39.py
@ -2127,7 +2127,7 @@ zoo
 				bitlen = int(k)
 				break
 		else:
-			raise MnemonicError('{}: invalid seed phrase length'.format(len(words)))
+			raise MnemonicError('{}: invalid BIP39 seed phrase length'.format(len(words)))

 		if pad != None:
 			assert pad * 4 == bitlen, '{}: invalid pad length'.format(pad)
@ -2143,7 +2143,7 @@ zoo
 		chk_bin_chk = '{:0{w}b}'.format(int(chk_hex_chk,16),w=256)[:chk_len]

 		if chk_bin != chk_bin_chk:
-			raise MnemonicError('invalid seed phrase checksum')
+			raise MnemonicError('invalid BIP39 seed phrase checksum')

 		return seed_hex

--- a/mmgen/exception.py
+++ b/mmgen/exception.py
@ -31,6 +31,7 @@ class MnemonicError(Exception):           mmcode = 1
 class RangeError(Exception):              mmcode = 1
 class FileNotFound(Exception):            mmcode = 1
 class InvalidPasswdFormat(Exception):     mmcode = 1
+class CfgFileParseError(Exception):       mmcode = 1

 # 2: yellow hl, message only
 class InvalidTokenAddress(Exception):     mmcode = 2
--- a/mmgen/globalvars.py
+++ b/mmgen/globalvars.py
@ -22,6 +22,7 @@ globalvars.py:  Constants and configuration options for the MMGen suite

 import sys,os
 from decimal import Decimal
+from collections import namedtuple
 from mmgen.devtools import *

 # Global vars are set to dfl values in class g.
@ -50,11 +51,13 @@ class g(object):
 	keywords  = 'Bitcoin, BTC, Ethereum, ETH, Monero, XMR, ERC20, cryptocurrency, wallet, BIP32, cold storage, offline, online, spending, open-source, command-line, Python, Linux, Bitcoin Core, bitcoind, hd, deterministic, hierarchical, secure, anonymous, Electrum, seed, mnemonic, brainwallet, Scrypt, utility, script, scriptable, blockchain, raw, transaction, permissionless, console, terminal, curses, ansi, color, tmux, remote, client, daemon, RPC, json, entropy, xterm, rxvt, PowerShell, MSYS, MSYS2, MinGW, MinGW64, MSWin, Armbian, Raspbian, Raspberry Pi, Orange Pi, BCash, BCH, Litecoin, LTC, altcoin, ZEC, Zcash, DASH, Dashpay, SHA256Compress, monerod, EMC, Emercoin, token, deploy, contract, gas, fee, smart contract, solidity, Parity, testnet, devmode, Kovan'
 	max_int   = 0xffffffff

-	stdin_tty = bool(sys.stdin.isatty() or os.getenv('MMGEN_TEST_SUITE_POPEN_SPAWN'))
+	stdin_tty = sys.stdin.isatty()
 	stdout = sys.stdout
 	stderr = sys.stderr

 	http_timeout = 60
+	err_disp_timeout = 0.7
+	short_disp_timeout = 0.3

 	# Variables - these might be altered at runtime:

@ -121,9 +124,9 @@ class g(object):

 	color = sys.stdout.isatty()

-	if os.getenv('HOME'):                             # Linux or MSYS
+	if os.getenv('HOME'):   # Linux or MSYS
 		home_dir = os.getenv('HOME')
-	elif platform == 'win': # Windows native:
+	elif platform == 'win': # non-MSYS Windows - not supported
 		die(1,'$HOME not set!  {} for Windows must be run in MSYS environment'.format(proj_name))
 	else:
 		die(2,'$HOME is not set!  Unable to determine home directory')
@ -154,7 +157,6 @@ class g(object):
 	)
 	incompatible_opts = (
 		('bob','alice'),
-		('quiet','verbose'),
 		('label','keep_label'),
 		('tx_id','info'),
 		('tx_id','terse_info'),
@ -200,8 +202,11 @@ class g(object):
 		'MMGEN_DISABLE_COLOR',
 		'MMGEN_DISABLE_MSWIN_PW_WARNING',
 	)
-	opt_values = { # first value is used as default
-		'fee_estimate_mode': ('nocase_str', ('conservative','economical')),
+	# Auto-typechecked and auto-set opts - incompatible with global_sets_opt and opt_sets_global
+	# First value in list is the default
+	ov = namedtuple('autoset_opt_info',['type','choices'])
+	autoset_opts = {
+		'fee_estimate_mode': ov('nocase_pfx', ('conservative','economical')),
 	}

 	min_screen_width = 80
@ -221,7 +226,7 @@ class g(object):
 	mmenc_ext      = 'mmenc'
 	salt_len       = 16
 	aesctr_iv_len  = 16
-	aesctr_dfl_iv  = b'\x00' * (aesctr_iv_len-1) + b'\x01'
+	aesctr_dfl_iv  = int.to_bytes(1,aesctr_iv_len,'big')
 	hincog_chk_len = 8

 	key_generators = ('python-ecdsa','libsecp256k1') # '1','2'
@ -240,3 +245,14 @@ class g(object):
 		'6': [17, 8, 20],
 		'7': [18, 8, 24],
 	}
+
+	if os.getenv('MMGEN_TEST_SUITE'):
+		err_disp_timeout = 0.1
+		short_disp_timeout = 0.1
+		if os.getenv('MMGEN_TEST_SUITE_POPEN_SPAWN'):
+			stdin_tty = True
+
+	if os.getenv('MMGEN_DEBUG_ALL'):
+		for name in env_opts:
+			if name[:11] == 'MMGEN_DEBUG':
+				os.environ[name] = '1'
--- a/mmgen/main_txcreate.py
+++ b/mmgen/main_txcreate.py
@ -38,7 +38,7 @@ opts_data = {
 -d, --outdir=      d  Specify an alternate directory 'd' for output
 -D, --contract-data=D Path to hex-encoded contract data (ETH only)
 -E, --fee-estimate-mode=M Specify the network fee estimate mode.  Choices:
-                      '{fec}'.  Default: '{fe}'
+                      {fe[1]}. Default: '{fe[1][0]}'
 -f, --tx-fee=      f  Transaction fee, as a decimal {cu} amount or as
                      {fu} (an integer followed by {fl}).
                      See FEE SPECIFICATION below.  If omitted, fee will be
@ -64,9 +64,8 @@ opts_data = {
 		'options': lambda s: s.format(
 			fu=help_notes('rel_fee_desc'),
 			fl=help_notes('fee_spec_letters'),
+			fe=g.autoset_opts['fee_estimate_mode'],
 			cu=g.coin,
-			fec="','".join(g.opt_values['fee_estimate_mode'][1]),
-			fe=g.opt_values['fee_estimate_mode'][1][0],
 			g=g),
 		'notes': lambda s: s.format(
 			help_notes('txcreate'),
--- a/mmgen/main_txdo.py
+++ b/mmgen/main_txdo.py
@ -42,7 +42,7 @@ opts_data = {
 -D, --contract-data= D Path to hex-encoded contract data (ETH only)
 -e, --echo-passphrase  Print passphrase to screen when typing it
 -E, --fee-estimate-mode=M Specify the network fee estimate mode.  Choices:
-                      '{fec}'.  Default: '{fe}'
+                       {fe[1]}. Default: '{fe[1][0]}'
 -f, --tx-fee=        f Transaction fee, as a decimal {cu} amount or as
                       {fu} (an integer followed by {fl}).
                       See FEE SPECIFICATION below.  If omitted, fee will be
@ -99,8 +99,7 @@ column below:
 			fu=help_notes('rel_fee_desc'),
 			fl=help_notes('fee_spec_letters'),
 			ss=g.subseeds,ss_max=SubSeedIdxRange.max_idx,
-			fec="','".join(g.opt_values['fee_estimate_mode'][1]),
-			fe=g.opt_values['fee_estimate_mode'][1][0],
+			fe=g.autoset_opts['fee_estimate_mode'],
 			kg=g.key_generator,
 			cu=g.coin),
 		'notes': lambda s: s.format(
--- a/mmgen/opts.py
+++ b/mmgen/opts.py
@ -21,13 +21,18 @@ opts.py:  MMGen-specific options processing after generic processing by share.Op
 """
 import sys,os,stat

-class opt(object): pass
+class opt(object):
+	pass

 from mmgen.globalvars import g
 import mmgen.share.Opts
 from mmgen.util import *

-def usage(): Die(1,'USAGE: {} {}'.format(g.prog_name,usage_txt))
+def usage():
+	Die(1,'USAGE: {} {}'.format(g.prog_name,usage_txt))
+
+def fmt_opt(o):
+	return '--' + o.replace('_','-')

 def die_on_incompatible_opts(incompat_list):
 	for group in incompat_list:
@ -35,8 +40,6 @@ def die_on_incompatible_opts(incompat_list):
 		if len(bad) > 1:
 			die(1,'Conflicting options: {}'.format(', '.join(map(fmt_opt,bad))))

-def fmt_opt(o): return '--' + o.replace('_','-')
-
 def _show_hash_presets():
 	fs = '  {:<7} {:<6} {:<3}  {}'
 	msg('Available parameters for scrypt.hash():')
@ -55,7 +58,8 @@ def opt_preproc_debug(short_opts,long_opts,skipped_opts,uopts,args):
 		('Cmd args',           args),
 	)
 	Msg('\n=== opts.py debug ===')
-	for e in d: Msg('    {:<20}: {}'.format(*e))
+	for e in d:
+		Msg('    {:<20}: {}'.format(*e))

 def opt_postproc_debug():
 	a = [k for k in dir(opt) if k[:2] != '__' and getattr(opt,k) != None]
@ -106,7 +110,6 @@ def get_cfg_template_data():
 		return ''

 def get_data_from_cfg_file():
-	from mmgen.util import msg,die,check_or_create_dir
 	check_or_create_dir(g.data_dir_root) # dies on error
 	template_data = get_cfg_template_data()
 	data = {}
@ -134,32 +137,45 @@ def get_data_from_cfg_file():

 	return data['cfg']

-def override_from_cfg_file(cfg_data):
-	from mmgen.util import die,strip_comments,set_for_type
+def override_globals_from_cfg_file(cfg_data):
 	import re
 	from mmgen.protocol import CoinProtocol
-	for n,l in enumerate(cfg_data.splitlines(),1): # DOS-safe
-		l = strip_comments(l)
-		if l == '': continue
-		m = re.match(r'(\w+)\s+(\S+)$',l)
-		if not m: die(2,"Parse error in file '{}', line {}".format(g.cfg_file,n))
-		name,val = m.groups()
-		if name in g.cfg_file_opts:
-			pfx,cfg_var = name.split('_',1)
-			if pfx in CoinProtocol.coins:
-				tn = False
-				cv1,cv2 = cfg_var.split('_',1)
-				if cv1 in ('mainnet','testnet'):
-					tn,cfg_var = (cv1 == 'testnet'),cv2
-				cls,attr = CoinProtocol(pfx,tn),cfg_var
-			else:
-				cls,attr = g,name
-			setattr(cls,attr,set_for_type(val,getattr(cls,attr),attr,src=g.cfg_file))
-		else:
-			die(2,"'{}': unrecognized option in '{}'".format(name,g.cfg_file))
+	from mmgen.util import strip_comments

-def override_from_env():
-	from mmgen.util import set_for_type
+	for n,l in enumerate(cfg_data.splitlines(),1):
+
+		l = strip_comments(l)
+		if l == '':
+			continue
+
+		try:
+			m = re.match(r'(\w+)(\s+(\S+)|(\s+\w+:\S+)+)$',l) # allow multiple colon-separated values
+			name = m[1]
+			val = dict([i.split(':') for i in m[2].split()]) if m[4] else m[3]
+		except:
+			raise CfgFileParseError('Parse error in file {!r}, line {}'.format(g.cfg_file,n))
+
+		if name in g.cfg_file_opts:
+			ns = name.split('_')
+			if ns[0] in CoinProtocol.coins:
+				nse,tn = (ns[2:],True) if len(ns) > 2 and ns[1] == 'testnet' else (ns[1:],False)
+				cls = CoinProtocol(ns[0],tn)
+				attr = '_'.join(nse)
+			else:
+				cls = g
+				attr = name
+			refval = getattr(cls,attr)
+			if type(refval) is dict and type(val) is str: # catch single colon-separated value
+				try:
+					val = dict([val.split(':')])
+				except:
+					raise CfgFileParseError('Parse error in file {!r}, line {}'.format(g.cfg_file,n))
+			val_conv = set_for_type(val,refval,attr,src=g.cfg_file)
+			setattr(cls,attr,val_conv)
+		else:
+			die(2,'{!r}: unrecognized option in {!r}'.format(name,g.cfg_file))
+
+def override_globals_from_env():
 	for name in g.env_opts:
 		if name == 'MMGEN_DEBUG_ALL': continue
 		disable = name[:14] == 'MMGEN_DISABLE_'
@ -176,7 +192,7 @@ def common_opts_code(s):
 		cu_all=' '.join(CoinProtocol.coins) )

 common_opts_data = {
-	# most, but not all, of these set the corresponding global var
+	# Most but not all of these set the corresponding global var
 	'text': """
 --, --accept-defaults     Accept defaults at all prompts
 --, --coin=c              Choose coin unit. Default: {cu_dfl}. Options: {cu_all}
@ -213,41 +229,47 @@ def init(opts_data,add_opts=[],opt_filter=None,parse_only=False):
 	if parse_only:
 		return uopts,args,short_opts,long_opts,skipped_opts

-	if g.debug_opts: opt_preproc_debug(short_opts,long_opts,skipped_opts,uopts,args)
+	if g.debug_opts:
+		opt_preproc_debug(short_opts,long_opts,skipped_opts,uopts,args)

-	# Save this for usage()
+	# Copy parsed opts to opt, setting values to None if not set by user
+	for o in  (
+			tuple(s.rstrip('=') for s in long_opts)
+			+ tuple(add_opts)
+			+ tuple(skipped_opts)
+			+ g.required_opts
+			+ g.common_opts ):
+		setattr(opt,o,uopts[o] if o in uopts else None)
+
+	# Make this available to usage()
 	global usage_txt
 	usage_txt = opts_data['text']['usage']

-	# Transfer uopts into opt, setting program's opts + required opts to None if not set by user
-	for o in  ( tuple([s.rstrip('=') for s in long_opts] + add_opts + skipped_opts)
-				+ g.required_opts
-				+ g.common_opts ):
-		setattr(opt,o,uopts[o] if o in uopts else None)
-
-	if opt.version: Die(0,"""
-    {pn} version {g.version}
-    Part of the {g.proj_name} suite, an online/offline cryptocoin wallet for the command line.
-    Copyright (C) {g.Cdates} {g.author} {g.email}
-	""".format(g=g,pn=g.prog_name.upper()).lstrip('\n').rstrip())
-
+	if opt.version:
+		Die(0,fmt("""
+			{pn} version {g.version}
+			Part of the {g.proj_name} suite, an online/offline cryptocurrency wallet for the
+			command line.  Copyright (C){g.Cdates} {g.author} {g.email}
+		""".format(g=g,pn=g.prog_name.upper()),indent='    ').rstrip())

 	if os.getenv('MMGEN_DEBUG_ALL'):
 		for name in g.env_opts:
 			if name[:11] == 'MMGEN_DEBUG':
 				os.environ[name] = '1'

-	# === Interaction with global vars begins here ===
+	# === begin global var initialization === #

 	# NB: user opt --data-dir is actually g.data_dir_root
 	# cfg file is in g.data_dir_root, wallet and other data are in g.data_dir
 	# We must set g.data_dir_root and g.cfg_file from cmdline before processing cfg file
 	set_data_dir_root()
 	if not opt.skip_cfg_file:
-		override_from_cfg_file(get_data_from_cfg_file())
-	override_from_env()
+		override_globals_from_cfg_file(get_data_from_cfg_file())
+	override_globals_from_env()

-	# User opt sets global var - do these here, before opt is set from g.global_sets_opt
+	# Set globals from opts, setting type from original global value
+	# Do here, before opts are set from globals below
+	# g.coin is finalized here
 	for k in (g.common_opts + g.opt_sets_global):
 		if hasattr(opt,k):
 			val = getattr(opt,k)
@ -261,18 +283,20 @@ def init(opts_data,add_opts=[],opt_filter=None,parse_only=False):
 	from mmgen.protocol import init_genonly_altcoins,CoinProtocol
 	altcoin_trust_level = init_genonly_altcoins(opt.coin or 'btc')

-	# g.testnet is set, so we can set g.proto
+	# g.testnet is finalized, so we can set g.proto
 	g.proto = CoinProtocol(g.coin,g.testnet)

-	# global sets proto
-	if g.daemon_data_dir: g.proto.daemon_data_dir = g.daemon_data_dir
+	# this could have been set from long opts
+	if g.daemon_data_dir:
+		g.proto.daemon_data_dir = g.daemon_data_dir

 	# g.proto is set, so we can set g.data_dir
 	g.data_dir = os.path.normpath(os.path.join(g.data_dir_root,g.proto.data_subdir))

-	# If user opt is set, convert its type based on value in mmgen.globalvars (g)
-	# If unset, set it to default value in mmgen.globalvars (g)
-	setattr(opt,'set_by_user',[])
+	# Set user opts from globals:
+	# - if opt is unset, set it to global value
+	# - if opt is set, convert its type to that of global value
+	opt.set_by_user = []
 	for k in g.global_sets_opt:
 		if k in opt.__dict__ and getattr(opt,k) != None:
 			setattr(opt,k,set_for_type(getattr(opt,k),getattr(g,k),'--'+k))
@ -284,7 +308,8 @@ def init(opts_data,add_opts=[],opt_filter=None,parse_only=False):
 		_show_hash_presets()
 		sys.exit(0)

-	if opt.verbose: opt.quiet = None
+	if opt.verbose:
+		opt.quiet = None

 	die_on_incompatible_opts(g.incompatible_opts)

@ -313,19 +338,20 @@ def init(opts_data,add_opts=[],opt_filter=None,parse_only=False):
 	if not check_opts(uopts):
 		die(1,'Options checking failed')

-	# Check user-set opts against g.opt_values, setting opt if unset:
+	# Check user-set opts against g.autoset_opts, setting opt if unset:
 	if not check_opts2(uopts):
 		die(1,'Options checking failed')

 	if hasattr(g,'cfg_options_changed'):
 		ymsg("Warning: config file options have changed! See '{}' for details".format(g.cfg_file+'.sample'))
 		if not g.test_suite:
-			from mmgen.util import my_raw_input
 			my_raw_input('Hit ENTER to continue: ')

 	if g.debug and g.prog_name != 'test.py':
 		opt.verbose,opt.quiet = (True,None)
-	if g.debug_opts: opt_postproc_debug()
+
+	if g.debug_opts:
+		opt_postproc_debug()

 	warn_altcoins(g.coin,altcoin_trust_level)

@ -355,9 +381,9 @@ def opt_is_tx_fee(val,desc):
 def check_opts2(usr_opts): # Returns false if any check fails

 	for key in [e for e in opt.__dict__ if not e.startswith('__')]:
-		if key in g.opt_values:
+		if key in g.autoset_opts:
 			val = getattr(opt,key)
-			d = g.opt_values[key]
+			d = g.autoset_opts[key]
 			if d[0] == 'nocase_str':
 				if val == None:
 					setattr(opt,key,d[1][0])
@ -426,7 +452,6 @@ def check_opts(usr_opts): # Returns false if any check fails

 		desc = "parameter for '{}' option".format(fmt_opt(key))

-		from mmgen.util import check_infile,check_outfile,check_outdir
 		# Check for file existence and readability
 		if key in ('keys_from_file','mmgen_keys_from_file',
 				'passwd_file','keysforaddrs','comment_file'):
--- a/mmgen/seed.py
+++ b/mmgen/seed.py
@ -728,10 +728,8 @@ an empty passphrase, just hit ENTER twice.
 	}

 	def _get_hash_preset_from_user(self,hp,desc_suf=''):
-# 					hp=a,
 		n = ('','old ')[self.op=='pwchg_old']
-		m,n = (('to accept the default',n),('to reuse the old','new '))[
-						int(self.op=='pwchg_new')]
+		m,n = (('to accept the default',n),('to reuse the old','new '))[self.op=='pwchg_new']
 		fs = "Enter {}hash preset for {}{}{},\n or hit ENTER {} value ('{}'): "
 		p = fs.format(
 			n,
@ -1075,27 +1073,33 @@ class DieRollSeedFile(SeedSourceUnenc):
 		seed_bitlen = self._choose_seedlen(self.wclass,seed_bitlens,self.mn_type)
 		nDierolls = self.conv_cls.seedlen_map['b6d'][seed_bitlen // 8]

-		m  = 'For a {sb}-bit seed you must roll the die {nd} times.  After each die roll,\n'
-		m += 'enter the result on the keyboard as a digit.  If you make an invalid entry,\n'
-		m += "you'll be prompted to re-enter it."
-
-		msg('\n'+m.format(sb=seed_bitlen,nd=nDierolls)+'\n')
+		m = """
+			For a {sb}-bit seed you must roll the die {nd} times.  After each die roll,
+			enter the result on the keyboard as a digit.  If you make an invalid entry,
+			you'll be prompted to re-enter it.
+		"""
+		msg('\n'+fmt(m.strip()).format(sb=seed_bitlen,nd=nDierolls)+'\n')

 		b6d_digits = self.conv_cls.digits['b6d']

-		from mmgen.term import get_char,get_char
+		cr = '\n' if g.test_suite else '\r'
+		prompt_fs = '\b\b\b   {}Enter die roll #{{}}: {}'.format(cr,CUR_SHOW)
+		clear_line = '' if g.test_suite else '\r' + ' ' * 25
+		invalid_msg = CUR_HIDE + cr + 'Invalid entry' + ' ' * 11
+
+		from mmgen.term import get_char
 		def get_digit(n):
-			p = '\b\b\b   \rEnter die roll #{}: '+ CUR_SHOW
-			sleep = 0.3
+			p = prompt_fs
+			sleep = g.short_disp_timeout
 			while True:
 				ch = get_char(p.format(n),num_chars=1,sleep=sleep).decode()
 				if ch in b6d_digits:
 					msg_r(CUR_HIDE + ' OK')
 					return ch
 				else:
-					msg_r(CUR_HIDE + '\rInvalid entry           ')
-					sleep = 0.7
-					p = '\r' + ' '*25 + CUR_SHOW + p
+					msg_r(invalid_msg)
+					sleep = g.err_disp_timeout
+					p = clear_line + prompt_fs

 		dierolls,n = [],1
 		while len(dierolls) < nDierolls:
--- a/mmgen/tool.py
+++ b/mmgen/tool.py
@ -540,7 +540,7 @@ class MMGenToolCmdMnemonic(MMGenToolCmdBase):
 			return baseconv.frombytes(bytestr,fmt,'seed',tostr=True)

 	def mn2hex( self, seed_mnemonic:'sstr', fmt:mn_opts_disp = dfl_mnemonic_fmt ):
-		"convert a 12, 18 or 24-word mnemonic seed phrase to a hexadecimal number"
+		"convert a mnemonic seed phrase to a hexadecimal number"
 		if fmt == 'bip39':
 			from mmgen.bip39 import bip39
 			return bip39.tohex(seed_mnemonic.split(),fmt)
--- a/mmgen/util.py
+++ b/mmgen/util.py
@ -96,6 +96,21 @@ def pp_fmt(d):
 def pp_msg(d):
 	msg(pp_fmt(d))

+def fmt(s,indent=''):
+	"de-indent multiple lines of text, or indent with specified string"
+	return indent + ('\n'+indent).join([l.strip() for l in s.strip().splitlines()]) + '\n'
+
+def fmt_list(l,fmt='dfl',indent=''):
+	"pretty-format a list"
+	sep,lq,rq = {
+		'utf8':      ("“, ”",      "“",    "”"),
+		'dfl':       ("', '",      "'",    "'"),
+		'bare':      (' ',         '',     '' ),
+		'no_quotes': (', ',        '',     '' ),
+		'col':       ('\n'+indent, indent, '' ),
+	}[fmt]
+	return lq + sep.join(l) + rq
+
 CUR_HIDE = '\033[?25l'
 CUR_SHOW = '\033[?25h'

@ -103,35 +118,46 @@ def warn_altcoins(coinsym,trust_level):
 	if trust_level > 3:
 		return

-	tl = (red('COMPLETELY UNTESTED'),red('LOW'),yellow('MEDIUM'),green('HIGH'))
+	tl_str = (
+		red('COMPLETELY UNTESTED'),
+		red('LOW'),
+		yellow('MEDIUM'),
+		green('HIGH'),
+	)[trust_level]
+
 	m = """
-Support for coin '{}' is EXPERIMENTAL.  The {pn} project assumes no
-responsibility for any loss of funds you may incur.
-This coin's {pn} testing status: {}
-Are you sure you want to continue?
-""".strip().format(coinsym.upper(),tl[trust_level],pn=g.proj_name)
+		Support for coin {!r} is EXPERIMENTAL.  The {pn} project
+		assumes no responsibility for any loss of funds you may incur.
+		This coin’s {pn} testing status: {}
+		Are you sure you want to continue?
+	"""
+	m = fmt(m).strip().format(coinsym.upper(),tl_str,pn=g.proj_name)

 	if g.test_suite:
-		qmsg(m); return
+		qmsg(m)
+		return
+
 	if not keypress_confirm(m,default_yes=True):
 		sys.exit(0)

 def set_for_type(val,refval,desc,invert_bool=False,src=None):
-	src_str = (''," in '{}'".format(src))[bool(src)]
+
 	if type(refval) == bool:
 		v = str(val).lower()
-		if v in ('true','yes','1'):          ret = True
-		elif v in ('false','no','none','0'): ret = False
-		else: die(1,"'{}': invalid value for '{}'{} (must be of type '{}')".format(
-				val,desc,src_str,'bool'))
-		if invert_bool: ret = not ret
+		ret = True if v in ('true','yes','1') else False if v in ('false','no','none','0') else None
+		if ret is not None:
+			return not ret if invert_bool else ret
 	else:
 		try:
-			ret = type(refval)((val,not val)[invert_bool])
+			return type(refval)(not val if invert_bool else val)
 		except:
-			die(1,"'{}': invalid value for '{}'{} (must be of type '{}')".format(
-				val,desc,src_str,type(refval).__name__))
-	return ret
+			pass
+
+	die(1,'{!r}: invalid value for {!r}{} (must be of type {!r})'.format(
+		val,
+		desc,
+		' in {!r}'.format(src) if src else '',
+		type(refval).__name__) )

 # From 'man dd':
 # c=1, w=2, b=512, kB=1000, K=1024, MB=1000*1000, M=1024*1024,
@ -196,16 +222,31 @@ def Vmsg_r(s,force=False):
 def dmsg(s):
 	if opt.debug: msg(s)

-def suf(arg,suf_type='s'):
-	suf_types = { 's': '', 'es': '', 'ies': 'y' }
-	assert suf_type in suf_types,'invalid suffix type'
+def suf(arg,suf_type='s',verb='none'):
+	suf_types = {
+		'none': {
+			's':   ('s',  ''),
+			'es':  ('es', ''),
+			'ies': ('ies','y'),
+		},
+		'is': {
+			's':   ('s are',  ' is'),
+			'es':  ('es are', ' is'),
+			'ies': ('ies are','y is'),
+		},
+		'has': {
+			's':   ('s have',  ' has'),
+			'es':  ('es have', ' has'),
+			'ies': ('ies have','y has'),
+		},
+	}
 	if isinstance(arg,int):
 		n = arg
 	elif isinstance(arg,(list,tuple,set,dict)):
 		n = len(arg)
 	else:
 		die(2,'{}: invalid parameter for suf()'.format(arg))
-	return suf_types[suf_type] if n == 1 else suf_type
+	return suf_types[verb][suf_type][n == 1]

 def get_extension(f):
 	a,b = os.path.splitext(f)
--- a/test/test.py
+++ b/test/test.py
@ -396,7 +396,8 @@ def list_tmpdirs():
 	return {k:cfgs[k]['tmpdir'] for k in cfgs}

 def clean(usr_dirs=None):
-	if opt.skip_deps: return
+	if opt.skip_deps:
+		return
 	all_dirs = list_tmpdirs()
 	dirnums = map(int,(usr_dirs if usr_dirs is not None else all_dirs))
 	dirlist = list(map(str,sorted(dirnums)))
@ -460,40 +461,32 @@ def set_restore_term_at_exit():

 class CmdGroupMgr(object):

-	cmd_groups = {
+	cmd_groups_dfl = {
 		'helpscreens':      ('TestSuiteHelp',{'modname':'misc','full_data':True}),
 		'main':             ('TestSuiteMain',{'full_data':True}),
 		'conv':             ('TestSuiteWalletConv',{'is3seed':True,'modname':'wallet'}),
+		'ref':              ('TestSuiteRef',{}),
 		'ref3':             ('TestSuiteRef3Seed',{'is3seed':True,'modname':'ref_3seed'}),
 		'ref3_addr':        ('TestSuiteRef3Addr',{'is3seed':True,'modname':'ref_3seed'}),
-		'ref':              ('TestSuiteRef',{}),
 		'ref_altcoin':      ('TestSuiteRefAltcoin',{}),
 		'seedsplit':        ('TestSuiteSeedSplit',{}),
 		'tool':             ('TestSuiteTool',{'modname':'misc','full_data':True}),
 		'input':            ('TestSuiteInput',{'modname':'misc','full_data':True}),
 		'output':           ('TestSuiteOutput',{'modname':'misc','full_data':True}),
+		'autosign':         ('TestSuiteAutosign',{}),
 		'regtest':          ('TestSuiteRegtest',{}),
 #		'chainsplit':       ('TestSuiteChainsplit',{}),
 		'ethdev':           ('TestSuiteEthdev',{}),
-		'autosign':         ('TestSuiteAutosign',{}),
+	}
+
+	cmd_groups_extra = {
 		'autosign_btc':     ('TestSuiteAutosignBTC',{'modname':'autosign'}),
 		'autosign_live':    ('TestSuiteAutosignLive',{'modname':'autosign'}),
 		'create_ref_tx':    ('TestSuiteRefTX',{'modname':'misc','full_data':True}),
 	}

-	dfl_groups =  ( 'helpscreens',
-					'main',
-					'conv',
-					'ref',
-					'ref3',
-					'ref3_addr',
-					'ref_altcoin',
-					'tool',
-					'input',
-					'output',
-					'autosign',
-					'regtest',
-					'ethdev')
+	cmd_groups = cmd_groups_dfl.copy()
+	cmd_groups.update(cmd_groups_extra)

 	def load_mod(self,gname,modname=None):
 		clsname,kwargs = self.cmd_groups[gname]
@ -561,7 +554,7 @@ class CmdGroupMgr(object):
 			ginfo = [g for g in ginfo
 						if network_id in g[1].networks
 							and not g[0] in exclude
-							and g[0] in self.dfl_groups + tuple(usr_args) ]
+							and g[0] in self.cmd_groups_dfl + tuple(usr_args) ]

 		for name,cls in ginfo:
 			msg('{:17} - {}'.format(name,cls.__doc__))
@ -631,7 +624,7 @@ class TestSuiteRunner(object):

 		passthru_opts = ['--{}{}'.format(k.replace('_','-'),
 							'=' + getattr(opt,k) if getattr(opt,k) != True else '')
-								for k in ('data_dir',) + self.ts.passthru_opts if getattr(opt,k)]
+								for k in self.ts.base_passthru_opts + self.ts.passthru_opts if getattr(opt,k)]

 		args = [cmd] + passthru_opts + self.ts.extra_spawn_args + args

@ -755,9 +748,9 @@ class TestSuiteRunner(object):
 			if opt.exclude_groups:
 				exclude = opt.exclude_groups.split(',')
 				for e in exclude:
-					if e not in self.gm.dfl_groups:
+					if e not in self.gm.cmd_groups_dfl:
 						die(1,'{!r}: group not recognized'.format(e))
-			for gname in self.gm.dfl_groups:
+			for gname in self.gm.cmd_groups_dfl:
 				if opt.exclude_groups and gname in exclude: continue
 				if not self.init_group(gname): continue
 				clean(self.ts.tmpdir_nums)
--- a/test/test_py_d/common.py
+++ b/test/test_py_d/common.py
@ -146,30 +146,36 @@ def get_label(do_shuffle=False):
 		label_iter = iter(labels)
 		return next(label_iter)

-def stealth_mnemonic_entry(t,mn,fmt):
-	wnum = 1
-	max_wordlen = { 'words': 12, 'bip39': 8 }[fmt]
+def stealth_mnemonic_entry(t,mn,fmt,pad_entry=False):

-	def get_pad_chars(n):
-		ret = ''
-		for i in range(n):
-			m = int.from_bytes(os.urandom(1),'big') % 32
-			ret += r'123579!@#$%^&*()_+-=[]{}"?/,.<>|'[m]
+	def pad_mnemonic(mn,ss_len):
+		def get_pad_chars(n):
+			ret = ''
+			for i in range(n):
+				m = int.from_bytes(os.urandom(1),'big') % 32
+				ret += r'123579!@#$%^&*()_+-=[]{}"?/,.<>|'[m]
+			return ret
+		ret = []
+		for w in mn:
+			if len(w) > (3,5)[ss_len==12]:
+				w = w + '\n'
+			else:
+				w = (
+					get_pad_chars(2 if randbool() else 0)
+					+ w[0] + get_pad_chars(2) + w[1:]
+					+ get_pad_chars(9) )
+				w = w[:ss_len+1]
+			ret.append(w)
 		return ret

-	for i in range(len(mn)):
-		w = mn[i]
-		if len(w) > (3,5)[max_wordlen==12]:
-			w = w + '\n'
-		else:
-			w = (
-				get_pad_chars(2 if randbool() else 0)
-				+ w[0] + get_pad_chars(2) + w[1:]
-				+ get_pad_chars(9) )
-			w = w[:max_wordlen+1]
-		em,rm = 'Enter word #{}: ','Repeat word #{}: '
+	mn = ['fzr'] + mn[:5] + ['grd','grdbxm'] + mn[5:]
+	mn = pad_mnemonic(mn,(12,8)[fmt=='bip39'])
+
+	wnum,em,rm = 1,'Enter word #{}: ','Repeat word #{}: '
+	for w in mn:
 		ret = t.expect((em.format(wnum),rm.format(wnum-1)))
-		if ret == 0: wnum += 1
+		if ret == 0:
+			wnum += 1
 		for j in range(len(w)):
 			t.send(w[j])
 			time.sleep(0.005)
--- a/test/test_py_d/ts_autosign.py
+++ b/test/test_py_d/ts_autosign.py
@ -70,7 +70,6 @@ class TestSuiteAutosign(TestSuiteBase):
 			t.expect('OK? (Y/n): ','\n')
 			mn_file = dfl_words_file
 			mn = read_from_file(mn_file).strip().split()
-			mn = ['foo'] + mn[:5] + ['realiz','realized'] + mn[5:]
 			stealth_mnemonic_entry(t,mn,fmt='words')
 			wf = t.written_to_file('Autosign wallet')
 			t.ok()
--- a/test/test_py_d/ts_base.py
+++ b/test/test_py_d/ts_base.py
@ -28,6 +28,7 @@ from test.test_py_d.common import *

 class TestSuiteBase(object):
 	'initializer class for the test.py test suite'
+	base_passthru_opts = ('data_dir',)
 	passthru_opts = ()
 	extra_spawn_args = []
 	networks = ()
--- a/test/test_py_d/ts_misc.py
+++ b/test/test_py_d/ts_misc.py
@ -92,7 +92,7 @@ class TestSuiteHelp(TestSuiteBase):
 		return self._run_cmd('test.py',['-l'],cmd_dir='test',extra_desc='(cmd list)')

 class TestSuiteOutput(TestSuiteBase):
-	'screen output tests'
+	'screen output'
 	networks = ('btc',)
 	tmpdir_nums = []
 	cmd_group = (
@ -117,12 +117,12 @@ class TestSuiteInput(TestSuiteBase):
 	networks = ('btc',)
 	tmpdir_nums = []
 	cmd_group = (
-		('password_entry_noecho', (1,"utf8 password entry", [])),
-		('password_entry_echo',   (1,"utf8 password entry (echoed)", [])),
-		('mnemonic_entry_mmgen',  (1,"stealth mnemonic entry (MMGen native)", [])),
-		('mnemonic_entry_bip39',  (1,"stealth mnemonic entry (BIP39)", [])),
-		('dieroll_entry',         (1,"dieroll entry (base6d)", [])),
-		('dieroll_entry_usrrand', (1,"dieroll entry (base6d) with added user entropy", [])),
+		('password_entry_noecho',         (1,"utf8 password entry", [])),
+		('password_entry_echo',           (1,"utf8 password entry (echoed)", [])),
+		('mnemonic_entry_mmgen',          (1,"stealth mnemonic entry (mmgen)", [])),
+		('mnemonic_entry_bip39',          (1,"stealth mnemonic entry (bip39)", [])),
+		('dieroll_entry',                 (1,"dieroll entry (base6d)", [])),
+		('dieroll_entry_usrrand',         (1,"dieroll entry (base6d) with added user entropy", [])),
 	)

 	def password_entry(self,prompt,cmd_args):
@ -149,14 +149,13 @@ class TestSuiteInput(TestSuiteBase):
 			return ('skip_warn',m)
 		return self.password_entry('Enter passphrase (echoed): ',['--echo-passphrase'])

-	def _user_seed_entry(self,fmt,usr_rand=False,out_fmt=None):
+	def _user_seed_entry(self,fmt,usr_rand=False,out_fmt=None,mn=None):
 		wcls = SeedSource.fmt_code_to_type(fmt)
 		wf = os.path.join(ref_dir,'FE3C6545.{}'.format(wcls.ext))
 		if wcls.wclass == 'mnemonic':
-			mn = read_from_file(wf).strip().split()
-			mn = ['foo'] + mn[:5] + ['grac','graceful'] + mn[5:]
+			mn = mn or read_from_file(wf).strip().split()
 		elif wcls.wclass == 'dieroll':
-			mn = list(read_from_file(wf).strip().translate(dict((ord(ws),None) for ws in '\t\n ')))
+			mn = mn or list(read_from_file(wf).strip().translate(dict((ord(ws),None) for ws in '\t\n ')))
 			for idx,val in ((5,'x'),(18,'0'),(30,'7'),(44,'9')):
 				mn.insert(idx,val)
 		t = self.spawn('mmgen-walletconv',['-r10','-S','-i',fmt,'-o',out_fmt or fmt])