Asynchronous HTTP significantly speeds up operations involving multiple
JSON-RPC calls to the server, such as tracking wallet views for wallets
with a large number of outputs.
This patch adds base-level asyncio infrastructure plus aiohttp support to all
applicable MMGen commands.
The aiohttp package is not currently supported by MSYS2, so Windows users will
have to choose one of the other backends ('curl' is the default).
Tested on: Linux, Armbian, Windows; Python 3.6, 3.7, 3.8
New user features:
- configurable RPC backends via the 'rpc_backend' option. Supported
options are 'aiohttp' (Linux-only), 'httplib', 'requests' and 'curl'
- configurable RPC queue size via the 'aiohttp_rpc_queue_len' option
The patch also includes a rewrite/redesign of large parts of the MMGen code
base, most importantly:
- rpc.py - full rewrite of RPC library, new RPCBackends class
- main_addrimport.py - full rewrite
- main_autosign.py - LED code now handled by new LEDControl class
- eth/tw.py, eth/tx.py - reworked logic for resolving token symbols and
addresses
- eth/tx.py - separate classes for signed and unsigned transactions
Testing:
# Set a backend (choose one):
$ export MMGEN_RPC_BACKEND='aiohttp' # Linux-only
$ export MMGEN_RPC_BACKEND='curl' # Windows
$ export MMGEN_RPC_BACKEND='httplib' # compare performance with 'aiohttp'
# Bitcoin:
$ test/unit_tests.py rpc btc
$ test/test.py main regtest autosign
# Ethereum:
$ test/unit_tests.py rpc eth
$ test/tooltest2.py --coin=eth --testnet=1 txview
$ test/test.py --coin=eth ethdev
# Monero wallet:
$ test/unit_tests.py rpc xmr_wallet
$ test/test-release.sh -F xmr
- provides a convenient interface to selected methods in the mmgen.tool module
- Type `pydoc3 mmgen.tool.tool_api` for available methods and call signatures
## Examples:
### Initialize:
from mmgen.tool import tool_api
tool = tool_api()
### Utility methods:
# Skip user entropy gathering (not recommended)
tool.usr_randchars = 0
# Generate a random hex secret:
hexsec = tool.randhex()
# Reverse the hex string:
hexsec_rev = tool.hexreverse(hexsec)
# Get the HASH160 of the value:
sec_hash160 = tool.hash160(hexsec)
# Convert the value to base58 format:
sec_b58 = tool.hextob58(hexsec)
# Convert the value to base58 check format:
sec_b58chk = tool.hextob58chk(hexsec)
# Convert the byte specification '4G' to an integer:
four_g = tool.bytespec('4G')
# Convert the byte specification '4GB' to an integer:
four_gb = tool.bytespec('4GB')
### Key/address generation:
# List available coins:
print(' '.join(tool.coins))
# Initialize a coin/network pair:
proto = tool.init_coin('btc','mainnet')
# Print the available address types for current coin/network, along with a
# description. If tool.addrtype is unset, the first-listed will be used:
tool.print_addrtypes()
# Set the address type to P2PKH with compressed public key:
tool.addrtype = 'compressed'
# Generate the key and address:
wif = tool.hex2wif(hexsec)
addr = tool.wif2addr(wif)
# Generate an LTC regtest Segwit key and address:
proto = tool.init_coin('ltc','regtest')
tool.addrtype = 'segwit'
wif = tool.hex2wif(hexsec)
addr = tool.wif2addr(wif)
# Generate a random LTC regtest Bech32 key/address pair:
tool.addrtype = 'bech32'
wif,addr = tool.randpair()
### Mnemonic seed phrase generation:
# Generate an MMGen native mnemonic seed phrase:
mmgen_seed = tool.hex2mn(hexsec)
# Generate a BIP39 mnemonic seed phrase:
bip39_seed = tool.hex2mn(hexsec,fmt='bip39')
- only 256-bit (25-word) new-style mnemonics are supported
Testing:
$ test/unit_tests.py baseconv
$ test/tooltest2.py hex2mn mn2hex
$ test/scrambletest.py pw
$ test/test.py ref_xmrseed_25_passwdgen_3
$ test/test.py ref_passwdfile_chk_xmrseed_25
The following operations are supported:
Generate a random Monero mnemonic:
$ mmgen-tool mn_rand256 fmt=xmrseed
Generate a Monero mnemonic from hexadecimal data:
$ mmgen-tool hex2mn deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef fmt=xmrseed
Convert the resulting mnemonic back to hexadecimal data:
$ mmgen-tool mn2hex 'viewpoint donuts ardent template unveil agile meant unafraid urgent athlete rustled mime azure jaded hawk baby jagged haystack baby jagged haystack ramped oncoming point template' fmt=xmrseed
Note that the result of the reversal does not match the original input. This
is because input data is reduced to a spendkey before conversion so that a
canonical seed phrase is produced. This is required because Monero seeds,
unlike ordinary wallet seeds, are tied to a concrete key/address pair. The
spendkey can be generated directly using the `hex2wif` command:
$ mmgen-tool --coin=xmr hex2wif deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
Generate a list of passwords in Monero mnemonic format with ID 'mymonero':
$ mmgen-passgen -f xmrseed 'mymonero' 1-10
- these are required due to the lack of the following external testing tools
in MSYS2:
+ ethkey (requires build)
+ moneropy (requires pysha3)
+ zcash-mini (requires golang)
This patch introduces the commands `mmgen-seedsplit` and `mmgen-seedjoin`.
The first creates shares one at a time, while the second joins them to
recover the original seed. By default, the default wallet is operated upon.
Shares are ordinary MMGen wallets and as such may be saved in any MMGen wallet
format, with one minor limitation: only one share in a given split may be in
hidden incognito format, and it must be the master share in the case of a
master share split.
For usage information and examples, see:
mmgen-seedsplit --help
mmgen-seedjoin --help
Relevant tests:
test/test.py -e seedsplit
ls -lrt test/tmp23/* # list the created files
test/objtest.py SeedSplitSpecifier
Related commits:
7311f474 - seed splitting: seed-level infrastructure
237567bc - master shares
- provided as an alternative to MMGen's native mnemonic format
# Run the BIP39 unit test:
$ test/unit_tests.py -v bip39
# Generate a random 128-bit BIP39 seed phrase:
$ mmgen-tool mn_rand128 fmt=bip39
# Export your default wallet to BIP39 format:
$ mmgen-walletconv -o bip39
...
BIP39 mnemonic data written to file '98831F3A[256].bip39'
# Generate ten addresses from the exported wallet:
$ mmgen-addrgen '98831F3A[256].bip39' 1-10
...
Addresses written to file '98831F3A[1-10].addrs'
# Generate ten addresses directly from your BIP39 seed phrase:
$ mmgen-addrgen -q -i bip39 1-10
...
Addresses written to file '98831F3A[1-10].addrs'
# Export subwallet 10L of your default wallet to BIP39 format:
$ mmgen-subwalletgen -o bip39 10L
...
BIP39 mnemonic data written to file 'A17F8E90[256].bip39'
Beginning with this commit, every MMGen wallet now has a two sets of associated
subwallets with “long“ and “short” seeds.
MMGen wallets and subwallets are functionally equivalent and externally
indistinguishable. This has benefits, especially for real-world security, as
well as drawbacks. For more information, see the `mmgen-subwalletgen` help
screen: https://github.com/mmgen/mmgen/wiki/subwalletgen-[MMGen-command-help]
This patch provides subwallet generation functionality and subseed display
utilities. Support for transaction signing and address generation using a
subwallet's parent wallet will be added in a forthcoming patch.
Examples:
# Create a bogus wallet in mnemonic format for testing purposes:
$ echo $(yes bee | head -n24) > bogus.mmwords
# List the wallet's first five subseed pairs:
$ mmgen-tool list_subseeds 1-5 wallet=bogus.mmwords
Parent Seed: DF449DA4 (256 bits)
Long Subseeds Short Subseeds
------------- --------------
1L: FC9A8735 1S: 930E1AD5
2L: 62B02F54 2S: DF14AB49
3L: 9E884E99 3S: AD3ABD98
4L: DB595AE1 4S: 3E885EC4
5L: 36D5A0D1 5S: 30D66FF5
# Generate the 5th short (128-bit) subwallet from the wallet:
$ mmgen-subwalletgen bogus.mmwords 5S
# Same as above, but output subwallet to mnemonic (seed phrase) format:
$ mmgen-subwalletgen -o mn bogus.mmwords 5S
...
Mnemonic data written to file '30D66FF5[128].mmwords'
# View the subwallet's seed phrase:
$ cat 30D66FF5[128].mmwords
object capture field heart page observe road bond mother loser really army
# Generate 10 addresses from the subwallet seed phrase:
$ mmgen-addrgen 30D66FF5[128].mmwords 1-10
...
Addresses written to file '30D66FF5[1-10].addrs'
- When parsing opts, opts.init() now looks only at string values from
opts_data. Global variables are evaluated only when printing help text,
after the variables are initialized
tooltest.py - bugfixes, remove some commands covered in tooltest2.py
mmgen-tool - bugfixes, cleanups, rename some commands, change some command
options
- all commands taking binary input can now receive it from file
or stdin
+ numerous minor fixes throughout
- test groups are now separate classes in separate modules
- test data and code is loaded on an as-needed basis
- new TestSuiteRunner and CmdGroupMgr classes
- simplified invocation: if arguments are omitted, all default tests relevant
for given network and option are run. The following set of invocations
provides nearly complete coverage of MMGen's core functionality:
test/test.py
test/test.py --segwit-random
test/test.py --bech32
test/test.py --coin=ltc
test/test.py --coin=ltc --segwit-random
test/test.py --coin=ltc --bech32
test/test.py --coin=bch
test/test.py --coin=eth
test/test.py --coin=etc
- group tool commands into classes
- add docstrings to command classes and methods
- annotate command method args
- generate call signatures + help and usage screens on the fly
- API changes:
- listaddresses,twview: new 'age_fmt' option replaces 'show_days' and 'show_age'
- addrfile_chksum and friends: 'mmtype' option removed
- tooltest.py, tooltest2.py and test.py have been updated accordingly
- 2 modes of operation:
- test tool commands by spawning the executable 'mmgen-tool'
- test tool commands directly, by importing 'mmgen.tool'
- all test vectors are contained in the script
- no dependency chains or IPC via the filesystem
