[WIP] Merge Stable to Master #1514

* using the correct mechanism to choose random port * show node registry only when it is non empty and do not show the message of missing pool transaction file on startup (#68) * update help message for 'missing genesis txns file' * Revert "update help message for 'missing genesis txns file'" This reverts commit 4b15c4b. * update help message for 'missing genesis txns file' * show help info aoubt when trigger * Revert "show help info aoubt when trigger" This reverts commit 730e74c. * show 'missing genesis txn' help text only when 'connect test' is trigger * show node registery only if available

New stable release

* proposed abstraction for network interface * initial commit for zstack * test in progress * fixing problem with verify key location * change in remote * test passing in promiscous mode * adding linger time but it does not help * using disconnect on remote and changing from old style to new style coroutines * completing test * comments * removed sleeps * refactoring tests and KITZStack * making basic node tests work * fixing tests in test_node_connection.py * changes for pool transactions * more changes * fixing more tests * fixing more tests * adding whitelist message * adding whitelist message for node catchup tests * added ELEC tag to logs for troubleshooting * fixing a bug where a node with a primary was nominating itself during re-elections * in the middle * patching non-test node and client objects to make script tests work * added dummy placeholders to allow tests to run until abstraction of networking is complete * all tests except one work * changes in scripts * changes for endpoint stack * commenting a log message which makes some tests run slow * fixes in scripts * not printing some logs on CLI * creating node params file * digest recalculation preventaion * checking for port in use for both tcp and udp * ensured new log entries didn't show up in CLI * printing exception with the log * beginnig to abolish key sharing * changes for key sharing * fixing a bug with req reply store and some tests * log messages * fixing logs * removing accidentally committed code * add pattern for hidden files to gitignore * adding disconnect detection * fixing bug for 3 phase messages prior to stable checkpoint * 'gc'ing requests only when checkpoint achived on protocol instances * fixing bug * resolving some race conditions around checkpoints * add option for supresing stdout logging * removing linger and handling exception while sending message through listener * closing auth, stopping prodables when logging error results in BlowUp and upgrating tests * Consistency proof request should specify the target seqNo * skipping a test and raising timeout * skipping test * skipping test * skipping test * skipping test * initial refactoring to move secure transport abstraction into a different project * minor changes * moved some modules to stp * moving node and client stacks * fixed some improper imports * more changes * fixed keystroke error * not calling sorted container's method from a newer api, resolving a bug in catchup when transactions are requested after timeout * [SOV-768] Fixed port not available exception getting ignored and tests failing on windows and linux based system * Removed unused import * flexible timeout and catching exception * Fixed error for port not available exception on windows * Removed extra lines at the end of exceptions file * Moved common logic related to transport to stp * using new names from stp * changes for stp * removing redundant close in orientdb socket * adding tests for catchup scenario and 3 phase message outside water mark * fixing import in test * Fix 'infinit' looping on receiving messages when their amount is very high (#97) * fix 'infinit' looping on receiving messages when their amount is very high * move quotas to config * make zstack get params from config only once * add test for quota * increase message quotas to 100 * Feature Added: Jenkins (#99) * Moved util methods to NetwrorkInterfaces; fixes in raet stack * fix key generation * Hotfix: Deps (#101) * - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (#98) - Corrected Cli.restoreWalletByPath method. * Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp * handling infinite loop problem and fixing test delay * fixing import * bugfix in test * raising timeout * merge * change in ordered * move connection related exceptions to stp project * fix imports of moved exceptions * fixed dependencies * temporarily disabling tests on Windows * renamed init_plenum_raet_keep to init_plenum_keys * removing unused code and changing script mode * Introduced failing of the pytets session in case any unexpected warnings have been made (#104) * changes to troubleshoot testing for coro not awaited * added support for warning checking across all tests * fixed scope for warnfilters fixture * - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points. - Sorted the rules in the warnings filter alphabetically. * Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine). * Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources. * Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter. * - Removed the rules not being actuated currently from the warnings filter. - Replaced use of line numbers with use of message patterns in the warnings filter rules. * Corrected the message pattern for a rule from the warnings filter. * Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter. * Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules. * adding warning * removing looper and checking while closing orientdb * increasing a timeout * replacing warn with warning methods * fixed stp dependency * fixed tests * skip failing test, created SOV-881. * changing logs to print alias * setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * adding timeout to test utility method * Versioning fix (#113) * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * moving error codes to stp * upgrading version of stp * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * build * build * build * build

* proposed abstraction for network interface * initial commit for zstack * test in progress * fixing problem with verify key location * change in remote * test passing in promiscous mode * adding linger time but it does not help * using disconnect on remote and changing from old style to new style coroutines * completing test * comments * removed sleeps * refactoring tests and KITZStack * making basic node tests work * fixing tests in test_node_connection.py * changes for pool transactions * more changes * fixing more tests * fixing more tests * adding whitelist message * adding whitelist message for node catchup tests * added ELEC tag to logs for troubleshooting * fixing a bug where a node with a primary was nominating itself during re-elections * in the middle * patching non-test node and client objects to make script tests work * added dummy placeholders to allow tests to run until abstraction of networking is complete * all tests except one work * changes in scripts * changes for endpoint stack * commenting a log message which makes some tests run slow * fixes in scripts * not printing some logs on CLI * creating node params file * digest recalculation preventaion * checking for port in use for both tcp and udp * ensured new log entries didn't show up in CLI * printing exception with the log * beginnig to abolish key sharing * changes for key sharing * fixing a bug with req reply store and some tests * log messages * fixing logs * removing accidentally committed code * add pattern for hidden files to gitignore * adding disconnect detection * fixing bug for 3 phase messages prior to stable checkpoint * 'gc'ing requests only when checkpoint achived on protocol instances * fixing bug * resolving some race conditions around checkpoints * add option for supresing stdout logging * removing linger and handling exception while sending message through listener * closing auth, stopping prodables when logging error results in BlowUp and upgrating tests * Consistency proof request should specify the target seqNo * skipping a test and raising timeout * skipping test * skipping test * skipping test * skipping test * initial refactoring to move secure transport abstraction into a different project * minor changes * moved some modules to stp * moving node and client stacks * fixed some improper imports * more changes * fixed keystroke error * not calling sorted container's method from a newer api, resolving a bug in catchup when transactions are requested after timeout * [SOV-768] Fixed port not available exception getting ignored and tests failing on windows and linux based system * Removed unused import * flexible timeout and catching exception * Fixed error for port not available exception on windows * Removed extra lines at the end of exceptions file * Moved common logic related to transport to stp * using new names from stp * changes for stp * removing redundant close in orientdb socket * adding tests for catchup scenario and 3 phase message outside water mark * fixing import in test * Fix 'infinit' looping on receiving messages when their amount is very high (#97) * fix 'infinit' looping on receiving messages when their amount is very high * move quotas to config * make zstack get params from config only once * add test for quota * increase message quotas to 100 * Feature Added: Jenkins (#99) * Moved util methods to NetwrorkInterfaces; fixes in raet stack * fix key generation * Hotfix: Deps (#101) * - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (#98) - Corrected Cli.restoreWalletByPath method. * Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp * handling infinite loop problem and fixing test delay * fixing import * bugfix in test * raising timeout * merge * change in ordered * move connection related exceptions to stp project * fix imports of moved exceptions * fixed dependencies * temporarily disabling tests on Windows * renamed init_plenum_raet_keep to init_plenum_keys * removing unused code and changing script mode * Introduced failing of the pytets session in case any unexpected warnings have been made (#104) * changes to troubleshoot testing for coro not awaited * added support for warning checking across all tests * fixed scope for warnfilters fixture * - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points. - Sorted the rules in the warnings filter alphabetically. * Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine). * Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources. * Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter. * - Removed the rules not being actuated currently from the warnings filter. - Replaced use of line numbers with use of message patterns in the warnings filter rules. * Corrected the message pattern for a rule from the warnings filter. * Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter. * Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules. * adding warning * removing looper and checking while closing orientdb * increasing a timeout * replacing warn with warning methods * fixed stp dependency * fixed tests * skip failing test, created SOV-881. * changing logs to print alias * setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * adding timeout to test utility method * Versioning fix (#113) * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * moving error codes to stp * upgrading version of stp * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * updated to use new API from shared lib * Enabled the tests in plenum.test.test_log_rotation module. (#135) * check for close method before calling it (#136) * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * added missed __init__.py for common/member * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * fix the catchup tests (#140) * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * up stp to 0.1.42 (#148) * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * fix generation of trustee txns * fix generation of trustee txns (#151) * Deps updated

* proposed abstraction for network interface * initial commit for zstack * test in progress * fixing problem with verify key location * change in remote * test passing in promiscous mode * adding linger time but it does not help * using disconnect on remote and changing from old style to new style coroutines * completing test * comments * removed sleeps * refactoring tests and KITZStack * making basic node tests work * fixing tests in test_node_connection.py * changes for pool transactions * more changes * fixing more tests * fixing more tests * adding whitelist message * adding whitelist message for node catchup tests * added ELEC tag to logs for troubleshooting * fixing a bug where a node with a primary was nominating itself during re-elections * in the middle * patching non-test node and client objects to make script tests work * added dummy placeholders to allow tests to run until abstraction of networking is complete * all tests except one work * changes in scripts * changes for endpoint stack * commenting a log message which makes some tests run slow * fixes in scripts * not printing some logs on CLI * creating node params file * digest recalculation preventaion * checking for port in use for both tcp and udp * ensured new log entries didn't show up in CLI * printing exception with the log * beginnig to abolish key sharing * changes for key sharing * fixing a bug with req reply store and some tests * log messages * fixing logs * removing accidentally committed code * add pattern for hidden files to gitignore * adding disconnect detection * fixing bug for 3 phase messages prior to stable checkpoint * 'gc'ing requests only when checkpoint achived on protocol instances * fixing bug * resolving some race conditions around checkpoints * add option for supresing stdout logging * removing linger and handling exception while sending message through listener * closing auth, stopping prodables when logging error results in BlowUp and upgrating tests * Consistency proof request should specify the target seqNo * skipping a test and raising timeout * skipping test * skipping test * skipping test * skipping test * initial refactoring to move secure transport abstraction into a different project * minor changes * moved some modules to stp * moving node and client stacks * fixed some improper imports * more changes * fixed keystroke error * not calling sorted container's method from a newer api, resolving a bug in catchup when transactions are requested after timeout * [SOV-768] Fixed port not available exception getting ignored and tests failing on windows and linux based system * Removed unused import * flexible timeout and catching exception * Fixed error for port not available exception on windows * Removed extra lines at the end of exceptions file * Moved common logic related to transport to stp * using new names from stp * changes for stp * removing redundant close in orientdb socket * adding tests for catchup scenario and 3 phase message outside water mark * fixing import in test * Fix 'infinit' looping on receiving messages when their amount is very high (#97) * fix 'infinit' looping on receiving messages when their amount is very high * move quotas to config * make zstack get params from config only once * add test for quota * increase message quotas to 100 * Feature Added: Jenkins (#99) * Moved util methods to NetwrorkInterfaces; fixes in raet stack * fix key generation * Hotfix: Deps (#101) * - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (#98) - Corrected Cli.restoreWalletByPath method. * Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp * handling infinite loop problem and fixing test delay * fixing import * bugfix in test * raising timeout * merge * change in ordered * move connection related exceptions to stp project * fix imports of moved exceptions * fixed dependencies * temporarily disabling tests on Windows * renamed init_plenum_raet_keep to init_plenum_keys * removing unused code and changing script mode * Introduced failing of the pytets session in case any unexpected warnings have been made (#104) * changes to troubleshoot testing for coro not awaited * added support for warning checking across all tests * fixed scope for warnfilters fixture * - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points. - Sorted the rules in the warnings filter alphabetically. * Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine). * Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources. * Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter. * - Removed the rules not being actuated currently from the warnings filter. - Replaced use of line numbers with use of message patterns in the warnings filter rules. * Corrected the message pattern for a rule from the warnings filter. * Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter. * Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules. * adding warning * removing looper and checking while closing orientdb * increasing a timeout * replacing warn with warning methods * fixed stp dependency * fixed tests * skip failing test, created SOV-881. * changing logs to print alias * setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * adding timeout to test utility method * Versioning fix (#113) * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * moving error codes to stp * upgrading version of stp * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * updated to use new API from shared lib * Enabled the tests in plenum.test.test_log_rotation module. (#135) * check for close method before calling it (#136) * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * added missed __init__.py for common/member * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * fix the catchup tests (#140) * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * up stp to 0.1.42 (#148) * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * fix generation of trustee txns * fix generation of trustee txns (#151) * fix

* changes for stp * removing redundant close in orientdb socket * adding tests for catchup scenario and 3 phase message outside water mark * fixing import in test * Fix 'infinit' looping on receiving messages when their amount is very high (#97) * fix 'infinit' looping on receiving messages when their amount is very high * move quotas to config * make zstack get params from config only once * add test for quota * increase message quotas to 100 * Feature Added: Jenkins (#99) * Moved util methods to NetwrorkInterfaces; fixes in raet stack * fix key generation * Hotfix: Deps (#101) * - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (#98) - Corrected Cli.restoreWalletByPath method. * Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp * handling infinite loop problem and fixing test delay * fixing import * bugfix in test * raising timeout * merge * change in ordered * move connection related exceptions to stp project * fix imports of moved exceptions * fixed dependencies * fixing name * temporarily disabling tests on Windows * renamed init_plenum_raet_keep to init_plenum_keys * removing unused code and changing script mode * Introduced failing of the pytets session in case any unexpected warnings have been made (#104) * changes to troubleshoot testing for coro not awaited * added support for warning checking across all tests * fixed scope for warnfilters fixture * - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points. - Sorted the rules in the warnings filter alphabetically. * Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine). * Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources. * Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter. * - Removed the rules not being actuated currently from the warnings filter. - Replaced use of line numbers with use of message patterns in the warnings filter rules. * Corrected the message pattern for a rule from the warnings filter. * Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter. * Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules. * adding warning * removing looper and checking while closing orientdb * increasing a timeout * replacing warn with warning methods * fixed stp dependency * fixed tests * skip failing test, created SOV-881. * adding load script * updating setup * changing logs to print alias * setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * some corrects to gc * setting the lastOrdered sequence number of the replica after election * fixing obsolete imports * adding temporary info logs to show up during load testing, problem identified for PREPREPARE getting rejects, state tree has different root at beginning for different nodes * reverting config changes * using ordered json encoder * add declarations of common request handling methods to ReqHandler * refactor method commit * add docstring for ReqHandler, rearrange methods * rename *ReqHandlers to *RequestHandlers * add TODOs about fixing hierarchy of State and PruningState * rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * fix * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * Added another test scenario for catchup fix * stable release

* setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * some corrects to gc * setting the lastOrdered sequence number of the replica after election * fixing obsolete imports * adding temporary info logs to show up during load testing, problem identified for PREPREPARE getting rejects, state tree has different root at beginning for different nodes * reverting config changes * using ordered json encoder * add declarations of common request handling methods to ReqHandler * refactor method commit * add docstring for ReqHandler, rearrange methods * rename *ReqHandlers to *RequestHandlers * add TODOs about fixing hierarchy of State and PruningState * rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (#184) * updates to test helpers and change in forwarding requests logic (#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (#197) * Enable and update testNodeDiscardMessageFromUnknownView (#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * fix README * fix setup.py * up stp version to 0.1.10

Resolved problems introduced from earlier merge

* rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (#184) * updates to test helpers and change in forwarding requests logic (#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (#197) * Enable and update testNodeDiscardMessageFromUnknownView (#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * Make random string generation efficient The current algorithm requires calling the libsodium random function 'size' times where size is the expected length of random string Changed this so that this will require at the most '(size/64)' number of calls. Also we do not need the chars list now we always use only alphabets and digits available in hex this makes it very easy and still gives us what we need. Every instance of usage of this function has been checked and nowhere the 'chars' parameter is used * Change to pass exact size to libsodium randombytes() function This probably is cryptographically more accurate * Fix catchup when commit (#203) * remove 3PC messages that are already ordered during catch-up * Added more tests * fix catch-up in between 3pc msgs * fix tests * update validation rule for verkey (#208) * fix an incorrect test (#207) * fix an incorrect test * refactoring test * remove un-necessary parameter (#210) * Revert unordered batches before starting catch-up (not after) (#209) * Revert unordered batches before starting catch-up (not after) * add test for revert of the state before catchup * improve the test * Do not enqueue old pre-prepares (#211) * ignore pp requests with ppSeqNo less than expected * minor refactor * Optimisations and a testing util (#212) * add todo, fixing log levels, commenting out an expensive area of code and adding capability to diable monitor in tests * an optimisation avoiding re-calculation of merkle proof * remove commented code and add test * update names * add test fils for validators * add test for commit message * add test for ConsistencyProof message * update test for Commit message * add test for InstanceChange message * add test for LedgerStatus message * add test for nomination message * add test for Ordered message * add tests for Prepare and PrePrepare messages * add test for Primary message * add test for PROPAGATE message * add test for Reelection message * remove unused imports * Election bug fix (#215) * Election bug fix if `f` nodes do not send Primary and the node is the last one to send Primary, then it was not able to decide the primary, since a node was doing "primary setting" only on receiving a Primary message, changed that to try "setting primary" after sending Primary too * update test with more checks

* add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (#184) * updates to test helpers and change in forwarding requests logic (#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (#197) * Enable and update testNodeDiscardMessageFromUnknownView (#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * Make random string generation efficient The current algorithm requires calling the libsodium random function 'size' times where size is the expected length of random string Changed this so that this will require at the most '(size/64)' number of calls. Also we do not need the chars list now we always use only alphabets and digits available in hex this makes it very easy and still gives us what we need. Every instance of usage of this function has been checked and nowhere the 'chars' parameter is used * Change to pass exact size to libsodium randombytes() function This probably is cryptographically more accurate * Fix catchup when commit (#203) * remove 3PC messages that are already ordered during catch-up * Added more tests * fix catch-up in between 3pc msgs * fix tests * update validation rule for verkey (#208) * fix an incorrect test (#207) * fix an incorrect test * refactoring test * remove un-necessary parameter (#210) * Revert unordered batches before starting catch-up (not after) (#209) * Revert unordered batches before starting catch-up (not after) * add test for revert of the state before catchup * improve the test * Do not enqueue old pre-prepares (#211) * ignore pp requests with ppSeqNo less than expected * minor refactor * Optimisations and a testing util (#212) * add todo, fixing log levels, commenting out an expensive area of code and adding capability to diable monitor in tests * an optimisation avoiding re-calculation of merkle proof * remove commented code and add test * update names * add test fils for validators * add test for commit message * add test for ConsistencyProof message * update test for Commit message * add test for InstanceChange message * add test for LedgerStatus message * add test for nomination message * add test for Ordered message * add tests for Prepare and PrePrepare messages * add test for Primary message * add test for PROPAGATE message * add test for Reelection message * remove unused imports * Election bug fix (#215) * Election bug fix if `f` nodes do not send Primary and the node is the last one to send Primary, then it was not able to decide the primary, since a node was doing "primary setting" only on receiving a Primary message, changed that to try "setting primary" after sending Primary too * update test with more checks * removed obsolete test testStopScriptIfNodeIsRunning (#220) * Fix processing Propagate msg, INDY-245 (#221) * Bugfix/non human readable error (#222) * handling case with odd-length verkey * updated exception message, check it in test * Stable release

re-pin indy dependencies

* add test for SerializedValueField * process stashed ledger status only once * calling correct method in force_ordered * add documentation for method * added common constants for input validation, tests for verkeys * optimized error output for incorrect b58 chars * updated tests for fields input validation * renamed decodedLengthConstraints to byte_lengths * revert uncommitted ledger on setting primary of master replica * checking for exception in case of ledger not present * fixed PR comments * reformat code and comment in a test * made API to get b58 strings with specified byte length * fixed pep8 warnings * fix primary selection * Add pool_upgrade force field; allow to send forced requests without waiting for sufficient number of nodes; (#228) * fix test * Commented out raet logging * Add test for verkey check (#230) * add new test that checks requests from different ledgers interleave, skip an probably incorrect test and refactoring * fix test * add log to test helper * Set commit quorum N-f Ref: testNodeRejectingInvalidTxns Fix: testNodeRejectingInvalidTxns Remove excess 'sendRandomRequests' Ref: add quorums, propagate Ref: use quorums in replica class Ref: quorums election Ref: remove old quorum code * Ref: remove useless condition * Ref: quorums for client * Ref: propagator Ref: minor Fix ReqState class * Fix: do not count node's propagate for quorum * Fix testNonPrimarySendsAPrePrepare * Fix propagator * workaround for byte sender names * fix removing name of node from senders * Fix testNodeDoesNotParticipateUntilCaughtUp * Skip test_primary_election_case6 * small clean-ups in ledger manager * use Base58Field for CONS_PROOF and MapField ionstead of IterableField for txns * create StringifiedNonNegativeNumberField and use it in CatchupRep * fix a bug with processing commit and catchup req * Change view_change, prepare quorums * view_change N-f * prepare N-f-1 ('-1' - except the primary) * update fixture to work even if catchup was tried again * Merge branches 'master' and 'simple-election' of https://github.com/evernym/plenum into simple-election * Merge branches 'master' and 'simple-election' of https://github.com/evernym/plenum into simple-election * small cleaning in fields.py * adding get_by_rank in test util and update a new test * fix test * dump * change schema of ViewChangeDone message and send acceptabted view change * reformat code and comment in a test * fix test * fix intermediately failing tests Catchup can be marked completed if got all txns from stashed ordered requests * fix test * add a test for catchup requests not needed and add ability to process specific delayed messages * remove unused imports and add a update test * re-factor slow_node tests: make them independent * remove debug print * update error message for validation * add missing blank line * update logic in CatchupOperationField * remove redundant blank line * changes for validation * create a build from a branch * create a build from a branch: disable tests temporarily * enable tests back * set name for exception for debug purpose * During view change, cathcup only certain number of times if catchup did not result in any new txns * bump dependency version * remove commented code, propagate regardless of participation, use correct wallet for node and some refactoring * add CatchupOperationField and use it in CatchupRep * use ClientMessageValidator instead of CatchupOperationField - it is possible because transactions in txn field are reformated back to request format * uncomment digest field of client request * remove fields from 'operation' when rcopying them to root level of document * remove CatchupOperationField, it is not needed anymore * test * add support of not strict schema * add field 'optional' to FieldValidator since all its successors have it * fix testNodeRejectingInvalidTxns * fix tests in test_client_node_op.py * fix tests in test_catchuprep_message.py * Validation for transaction sequence number and sha256 hash (#233) * add input validation for transaction sequence number field * adding sha256 validation * splitting test * adding TRUSTEE as an allowed role * adding perf tests * Added LimitedLengthStringField (#234) * Added LimitedLengthStringField * Code review fixes * Code review fixes * Code review fixes * update doc * refactoring * Add MessageFactory * Add node_message_factory * Fix the validation error messages, make "_raise_invalid*" static * Ref: MessageFactory * Add TestMsg to the message factory * Use node_message_factory for client, refactoring * fix timeout in tests * Make rest messages MessageBase * Fix testClientNotRetryRequestWhenReqnackReceived * Remove TaggedTuples * Ref: input validation * move node messages into a separate file * remove TaggedTuple * Add update_schemas_by_field_type method * Validation for transaction sequence number and sha256 hash (#233) * add input validation for transaction sequence number field * adding sha256 validation * splitting test * adding TRUSTEE as an allowed role * Added LimitedLengthStringField (#234) * Added LimitedLengthStringField * Code review fixes * Code review fixes * Code review fixes * adding perf tests * update doc * rename file * Fix reqToTxn, roll back catchup_rep validation * Update setup.py * Fix test notes from indy-111 * Fix test_has_expected_validators * Remove runner.py * fix jenkinsfile * fix test name and remove commented skip (#240) * Version update * fix tests * ZStack takes config (#237) * pass config to ZStacks and move utility to stp * remove unused imports * Test fix * Roll back f.TXN_ROOT validation for PrePrepares * Improve validation error massages * Fix tests * Fix tests and merge issues * Update readme for new home in Hyperledger. * Ledger using generators (#246) * pass config to ZStacks and move utility to stp * remove unused imports * midcommit * ledger iteration is done using generators * bump dependency version * add test for state re-creation from ledger * Added predicates constant * log level change, deleting useless test and updating tests (#247) * log level change, deleting useless test and updating tests * bum dependency version * Rearrange tests * Reverted unusued changes * handle corrupt pool ledger files (INDY-150) * Fix test testInstanceChangeMsgTypeChecking * Make stateRootHash and txnRootHash fields MerkleRootField type * update test with descriptive name * Make stateRootHash and txnRootHash fields nullable * Fix update_schemas_by_field_type, node message factory * Cli did support (#239) * added some tests for genesis pool files, showing some brittlness * Switched CLI to use DID instead of Cryptonyms Required a few changes to some signers to handle some of the use case in the CLI * Added output for new Signer to include verfication keys in addtion to identifiers * fixed test to match current CLI output * made signer_did handle case where verkey is not passed a little better * changed veriable to non-keyword * changed veriable to not be a keyword * add test for full_verkey * Version fieldvalidator (#243) * Add validator for vrsion field * CHnages based on code review * Add check spaces in the midle of version * Add more testcases; fix conditions; * Simplify the version component check * Moved handleGetTnxReq function in plenum * Corrected params in getReplyFromLedger function * Fix test_message_factory tests * Add runner.py back * GET_TXN request handling * Fix memory leaks in Replica (#248) * refactor creation of inBoxRouter * remove view_ends_at * add test for cleaning of ordered * add removal of outdated ordereds * add cleaning for primaryNames * rename test_ordered_cleaning to test_buffers_cleaning * remove stashed checkpoints when they were processed * remove self.stashedRecvdCheckpoints[key] if it is empty * move calling of self.compact_ordered() to gc() * do gc() manually * Fixed comments * Changed GET_TXN response * manage permissions for keyring files and directories INDY-323 (#232) * helper class to manage keyrings and tests for it * fixed tests * allow save wallet by absolute path, updated tests * default values for files and dirs permissions in keyring * added logging about saved wallet * switch cli to use new API to save wallets, added test * removed saveGivenWallet API as obsolete * added loadWallet api and tests * swicthed wallet loading logic to new API, removed old one * added TODO comment * replace TypeError with more appropriate ValueError * improved style: variable name for imported class * pathlib onstead of os for WalletStorageHelper * updated exception message check * delayed walletSaver initialization * Added test that resu; of get_txn transaction same as expected * Ability to turn off spyables for benchmarking and memory tracking utils (#254) * print size of objects * add method to inspect collection size of objects and add option to disable spies for benchmarking * add more options to memory usage methods * do checkpointing frequently * Update imports from stp (#256) * fix imports * up stp to 0.2.42 * reduce delay in test_view_change_in_between_3pc_all_nodes_random_delays_long_delay because it runs for too long * Support GET_TXN transaction (#249) * test * Added predicates constant * Reverted unusued changes * Moved handleGetTnxReq function in plenum * Corrected params in getReplyFromLedger function * GET_TXN request handling * Fixed comments * Changed GET_TXN response * Added test that resu; of get_txn transaction same as expected * Add capability to request messages (#251) * log level change, deleting useless test and updating tests * bum dependency version * Add capability to request messages A node can request certain kinds of messages from other nodes. Currently PRE-PREPARE, LEDGER_STATUS and CONSISTENCY_PROOF are supported types. The old way of LEDGER_STATUS and CONSISTENCY_PROOF is removed. A replica can request PRE-PREPAREs from other nodes if it is missing them but has sufficient PREPAREs * remove commented code and commits are queued even when prepares missing * add test for checking malicious behavior while requesting pre-prepare * refactor * adding more tests * rename objects * add new test for view change and use fixture in a logging test * fix broken merge * update imports * add comment to revisit * Bump dependencies * Refactor: Use correct error code for instance change after timeout, use correct method name for get transaction handler, separate method to send acknowledgement and minor improvement in test utility (#258) * log level change, deleting useless test and updating tests * bum dependency version * Add capability to request messages A node can request certain kinds of messages from other nodes. Currently PRE-PREPARE, LEDGER_STATUS and CONSISTENCY_PROOF are supported types. The old way of LEDGER_STATUS and CONSISTENCY_PROOF is removed. A replica can request PRE-PREPAREs from other nodes if it is missing them but has sufficient PREPAREs * remove commented code and commits are queued even when prepares missing * add test for checking malicious behavior while requesting pre-prepare * refactor * adding more tests * rename objects * add new test for view change and use fixture in a logging test * fix broken merge * update imports * add comment to revisit * Refactor: Use correct error code for instance change after timeout, use correct method name for get transaction handler, separate method to send acknowledgement and minor improvement in test utility * update test to check if pool is functional * let GET_TXN handler return reply * removing unused config variables and imports (#263) * removing unused config variables and imports * remove commented code * Feature Added: Repo consolidation (#262) * [WIP] Repo merge Merged in ledger * [WIP] Repo merge Merged in state, stp * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * Update Jenkinsfile * Update setup.py * Fix consensus tests; add catchup consensus to Quorums (#260) * Fix consensus tests; add catchup consensus to Quorums - fixed consensus tests - added new tests - added catchup consensus values to Quorums - added checkpoint consensus values to Quorums * add catchup consensus to Quorums * add ledger_status quorum * Checkpoint bug (#264) * removing unused config variables and imports * remove commented code * fix bug with checkpoint and gc after view change * Feature Added: Repo consolidation (#262) * [WIP] Repo merge Merged in ledger * [WIP] Repo merge Merged in state, stp * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * Update Jenkinsfile * Update setup.py * update test * Corrected GET_TXN Responce type * Corrected get_Txn response * Test network did (#261) * added some tests for genesis pool files, showing some brittlness * Changed test_network_setup to use DIDs test_network_setup used cryptonym but we are switching over to DID style Ids. Cli no longer easily support cryptonym so we need the test genisus txn to also use DID. * Signature is not required for GET_TXN transaction. Corrected GET_TXN tets * Fix serialization of messages when ujson < 1.35 is used (#266) * add test for serialization of messages * override __dir__ method for MessageBase to exclude service fields * add test for dir over message * Corrected checking * Node requests Propagates if needed (#269) * removing unused config variables and imports * remove commented code * fix bug with checkpoint and gc after view change * Feature Added: Repo consolidation (#262) * [WIP] Repo merge Merged in ledger * [WIP] Repo merge Merged in state, stp * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * [WIP] Repo merge Test fixes * Update Jenkinsfile * Update setup.py * update test * Node requests PROPAGATEs If a node receives a PRE-PREPARE but does not enough finalised requests, it requests PROPAGATEs * use fixture params in test and update test with more checks * use correct type * add and update tests * Deleted reference to sovrin_common * Corrected GET_TXN Response type (#265) * test * Added predicates constant * Reverted unusued changes * Moved handleGetTnxReq function in plenum * Corrected params in getReplyFromLedger function * GET_TXN request handling * Fixed comments * Changed GET_TXN response * Added test that resu; of get_txn transaction same as expected * Corrected GET_TXN Responce type * Corrected get_Txn response * Signature is not required for GET_TXN transaction. Corrected GET_TXN tets * Corrected checking * Deleted reference to sovrin_common * HashStore is used to recover when the ledger starts, fixed the calcu… (#271) * Hash store is used to recover when the ledger starts, fixed the calculation for consistency Recovering from transaction log is slow as each transaction is applied again to the tree. * change log level * put imports at top and add a property to HashStore * fix ujson version to have the same serialization of composite messages (#272) * fix ujson version to have the same serialization of composite messages * fix serialization test * Set consensus for catchup and ladger_status to f + 1 * use epoch for time in PRE-PREPARE and PREPARE * Authenticator will not authenticate cryptonyms not on ledger - INDY 213 (#267) * CRYPTONYMS WILL NOW CHECK ON LEDGER When a cryptonym was given that was not on the ledger, the authenticator would not verfiy and use the credentials supplied by the client. * RYPTONYMS WILL NOW CHECK ON LEDGER - INDY 213 When a cryptonym not on the ledger is attempted to be authenticated, it should check the ledger and not depend on the credentials provided by the client. When this occurs, the authenticator should throw an exception (because the verkey is None) * Removed unittest, switched for proper pytest.raises * Feature/indy 399 (#274) * Feature Added: Package renaming support * Feature Added: Package renaming support * Feature Added: Package renaming support * Feature Added: Package renaming support * Merge preparations * Update Jenkinsfile * Update Jenkinsfile * first working draft of timestamps * refactor, clear previous view checkpoint and ignore checkpoint digest… (#273) * refactor, clear previous view checkpoint and ignore checkpoint digest check for non-master * rename method, fix spelling and indentation * Catchup if 3pc messages are missed, INDY-335 * start the catchup procedure if a node going to stash second CHECKPOINT * minor refactor * add TODOs * Do not start catchup if already started * Do not start catchup for primary or non-master Start the catchup procedure only for master non-primary replicas * Fix stashed_checkpoints_with_quorum * Updated ci helpers (#276) * storing time in ledger * add error log to global whitelist and handle exception while processing PRE-PREPARE * rename and add missed parameter * update test * Corrected GET_TXN response * more tests * not ignoring txn time * Message for notifying lagged nodes about previously selected primary (#277) * add test for CurrentState message * create CurrentState message * send CurrentState to lagged nodes * make message tests check order of fields correctly * refactor creation of router for node messages * handler received CurrentState message * update name of method for sending current state * update CurrentState message * add CurrentState message to whitelist * add test for propagation of current state * change type of field in CurrentState * add methods of node related to CurrentState message to spylog * add debug log message for processing of current state * add verbose messag for assertion * unwrap ViewChangeDone message from LedgerStatus and send it to elector * remove commented sendElectionMsgsToLaggingNode method * add MessageField field validator * make _validate_message instane method * discard LedgerStatus if cannot parse internal messages * set todo about changing validation of internal messages * Add explicit log message on successful Notifier Plugin Import (#281) * adding item to quorums * Stable

Merge from master for release

Release merge

…lenum into rc-1.1

Rc 1.1

* INDY-733: Fixed bug in WalletCompatibilityBackend - Fixed a bug with decoding non-string keys of dictionaries in WalletCompatibilityBackend. * Added migration of genesis pool transactions to the client.

* Node status tool base support * Improve node status tool support * Add OOP, renaming * Remove unsupported fields * Renaming * Add dynamic tests * Make node status tool safe * Fix primary selection after primary demotion (INDY-463) (#354) * modified rank routine, added test cases for primary demotion * fixed pep8 * fixed nodes oreder initialization * added checks that pool is functional after recofigurations * added additional view change to ensure original master is skipped * Merged hotfixes from stable for wallet and genesis pool transactions file migration and readme file. * Fix validator info tool (#365) * Don't add ip addresses, one-shot dump on start * Parameterize zmq network protocol * Cherry-pick cb7d068 8642be5 c37e835 from master

Rc 3.1

Rc 3.2

Signed-off-by: Dmitry Surnin <[email protected]>

Rc 17.23 28 Ignore DCO failure for some commits: they passed at the time accordant were merged, don't want to edit them forcefully.

Signed-off-by: Dmitry Surnin <[email protected]>

Fix client basedir initialization

Signed-off-by: dsurnin <[email protected]>

Stable 51. Ignore DCO check failure - checked it manually for the commits where it failed.

…nto 1.2.31-stable

1.2.31 stable. DCO check failure was verified manually.

Signed-off-by: ashcherbakov <[email protected]>

fix setting of 3PC key after catch-up

Signed-off-by: ashcherbakov <[email protected]>

reset last_ordered_3PC for future views during view change.

Signed-off-by: Lovesh <[email protected]> (cherry picked from commit 5db7629) Signed-off-by: ashcherbakov <[email protected]>

fix KeyError for "services" when pool ledger contains node update txns

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.2.35

Signed-off-by: Andrew Nikitin <[email protected]>

[Increment libindy-crypto] Bumped libindy-crypto version to 0.4.0

Signed-off-by: Andrew Nikitin <[email protected]>

…ement-crypto Signed-off-by: Andrew Nikitin <[email protected]>

[Freeze pip] Pip version < 10.0.0

Add from current state flag to be able to process veiw change done on zero view Signed-off-by: dsurnin <[email protected]>

Extend testing timeout Signed-off-by: dsurnin <[email protected]>

Fix current_state processing Signed-off-by: dsurnin <[email protected]>

Signed-off-by: dsurnin <[email protected]>

Release hotfix1

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.4.43 DCO failure was reviewed manually. PR is passed on Hyperledger Server.

Signed-off-by: ashcherbakov <[email protected]>

use protocolVersion=None in test genesis files

Signed-off-by: ashcherbakov <[email protected]>

RC-1.4.45 with "fix get value from state if there is no BLS multi-sig"

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.5.46 DCO check was verified manually

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.5.46

Test case when one node restart and in catchup receive Ledger Status without protocol version from the one of nodes in pool. Signed-off-by: toktar <[email protected]>

Add check discarding a Ledger Statuses from the break_node for all ledgers Signed-off-by: toktar <[email protected]>

- Added handling of TypeError to the logic of replied message creation in the handler of replies to message requests. Signed-off-by: Nikita Spivachuk <[email protected]>

Add test for catchup with receiving incorrect ledger status

- Added unit tests for some unhandled cases in message request / response handling logic. - Added handling for these cases to message request / response processing logic. Signed-off-by: Nikita Spivachuk <[email protected]>

INDY-1519: Fixed handling of message replies

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.6.49

Signed-off-by: lovesh harchandani <[email protected]>

get request handler by txn type

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.6.51

new version 1.6.554-master release Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.6.52

Signed-off-by: Andrew Nikitin <[email protected]>

[bump indy-crypto] bump version for python3-indy-crypto

Signed-off-by: Andrew Nikitin <[email protected]>

1.6.54 rc

Signed-off-by: toktar <[email protected]>

Changes: - move check in unpackClientMsg and add check for request type - add tests Signed-off-by: toktar <[email protected]>

Signed-off-by: toktar <[email protected]>

1.6.55 rc

Signed-off-by: toktar <[email protected]>

Changes: - change an interval - fix tests Signed-off-by: toktar <[email protected]>

Changed: -add tests -change time counter Signed-off-by: toktar <[email protected]>

Signed-off-by: toktar <[email protected]>

1.6.56 rc

ToleratePrimaryDisconnection increased up to 60 seconds. The bind re-try time increased up to 10 seconds. Signed-off-by: Sergey Shilov <[email protected]>

Signed-off-by: Sergey Shilov <[email protected]>

…-1926 Cherry-pick INDY-1836/INDY-1926

Signed-off-by: Andrew Nikitin <[email protected]>

Rc 1.6.58

Signed-off-by: Andrew Nikitin <[email protected]>

[RC-1.7.0] RC build

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[RC-1.7.1.rc1] Hotfix release candidate

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[RC1-1.8.0] Release candidate for 1.8.0

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[Hotfix 1.8.1] RC for 1.8.1

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrey Kononykhin <[email protected]>

[INDY-2095] RC 1.9.0.rc1

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Sergey Khoroshavin <[email protected]>

INDY-2162: Rc 1.9.1.rc1

Signed-off-by: Sergey Khoroshavin <[email protected]>

INDY-2162: 1.9.1.rc2

Signed-off-by: Sergey Khoroshavin <[email protected]>

INDY-2162: 1.9.1.rc3

Signed-off-by: Sovbot <[email protected]>

INDY-1954 -- Implement GET_TXN audit proof read Signed-off-by: Andrew Nikitin <[email protected]>

INDY-1954 Add multisig check Signed-off-by: Andrew Nikitin <[email protected]>

INDY-2215: add re-ask LedgerStatuses for the init catchup Signed-off-by: Andrew Nikitin <[email protected]>

[INDY-2218] Add check for identifier decoding Signed-off-by: Andrew Nikitin <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[1.9.2.rc1] Release candidate for 1.9.2

Signed-off-by: Sovbot <[email protected]>

[INDY-2213]: Bump pyzmq version Signed-off-by: Andrew Nikitin <[email protected]>

[INDY-2213]: Fix FPM dependencies Signed-off-by: Andrew Nikitin <[email protected]>

Signed-off-by: Nemanja Patrnogic <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

INDY-2233 -- Fix audit proof for GET_TXN Signed-off-by: Andrew Nikitin <[email protected]>

[Rc 1.10.0] Release candidate for indy-plenum

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[Rc 1.11.0.rc1] Release candidate for 1.11.0

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[Rc 1.12.0.rc1] Release candidate for 1.12.0

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: ashcherbakov <[email protected]>

RC 1.12.1.rc1

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Andrew Nikitin <[email protected]>

[RC1-1.12.2] Release candidate for 1.12.2 stable

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: toktar <[email protected]> (cherry picked from commit dbbc0ca) Signed-off-by: toktar <[email protected]>

Signed-off-by: toktar <[email protected]>

[RC1-1.12.3] Release candidate for 1.12.3 stable

Signed-off-by: Sovbot <[email protected]>

Signed-off-by: Wade Barnes <[email protected]>

[RC1-1.12.4] Release candidate for 1.12.4 stable

Signed-off-by: Sovbot <[email protected]>

Commits on Mar 27, 2017

Feature Added: Jenkins (#100 )

Andrei Goncharov authored Mar 27, 2017

Configuration menu

View commit details

Copy full SHA for f4784e6

Browse repository at this point

Copy the full SHA

f4784e6 View commit details

Browse the repository at this point in the history

Commits on Mar 28, 2017

Hotfix: Deps (#102 )

Andrei Goncharov authored Mar 28, 2017

Configuration menu

View commit details

Copy full SHA for b639a44

Browse repository at this point

Copy the full SHA

b639a44 View commit details

Browse the repository at this point in the history

Commits on Apr 6, 2017

Versioning fix (#112 )

Andrei Goncharov authored Apr 6, 2017

Configuration menu

View commit details

Copy full SHA for e4bd9b0

Browse repository at this point

Copy the full SHA

e4bd9b0 View commit details

Browse the repository at this point in the history

[WIP] Merge Stable to Master #1514

[WIP] Merge Stable to Master #1514

Commits on Feb 21, 2017

Commits on Mar 27, 2017

Commits on Mar 28, 2017

Commits on Mar 30, 2017

Commits on Apr 6, 2017

Commits on Apr 21, 2017

Commits on May 3, 2017

Commits on Jun 1, 2017

Commits on Jun 5, 2017

Commits on Jun 15, 2017

Commits on Jun 21, 2017

Commits on Jul 6, 2017

Commits on Jul 18, 2017

Commits on Jul 25, 2017

Commits on Jul 26, 2017

Commits on Aug 16, 2017

Commits on Aug 25, 2017

Commits on Aug 28, 2017

Commits on Aug 29, 2017

Commits on Sep 6, 2017

Commits on Sep 12, 2017

Commits on Sep 13, 2017

Commits on Nov 16, 2017

Commits on Nov 21, 2017

Commits on Feb 9, 2018

Commits on Feb 15, 2018

Commits on Feb 21, 2018

Commits on Feb 22, 2018

Commits on Feb 26, 2018

Commits on Apr 12, 2018

Commits on Apr 13, 2018

Commits on Apr 16, 2018

Commits on Apr 17, 2018

Commits on May 16, 2018

Commits on May 17, 2018

Commits on Jun 20, 2018

Commits on Jun 25, 2018

Commits on Jun 27, 2018

Commits on Jun 28, 2018

Commits on Jul 18, 2018

Commits on Jul 19, 2018

Commits on Jul 22, 2018

Commits on Jul 25, 2018

Commits on Jul 26, 2018

Commits on Aug 9, 2018

Commits on Aug 21, 2018

Commits on Aug 31, 2018

Commits on Oct 5, 2018

Commits on Oct 10, 2018

Commits on Oct 11, 2018

Commits on Nov 28, 2018

Commits on Nov 29, 2018

Commits on Dec 11, 2018

Commits on Dec 12, 2018

Commits on Dec 18, 2018

Commits on Dec 19, 2018

Commits on Feb 5, 2019

Commits on Feb 6, 2019

Commits on Apr 23, 2019

Commits on Apr 24, 2019

Commits on May 28, 2019

Commits on Jun 4, 2019

Commits on Jun 5, 2019

Commits on Jun 27, 2019

Commits on Jul 29, 2019

Commits on Aug 28, 2019

Commits on Sep 30, 2019

Commits on Oct 1, 2019

Commits on Oct 2, 2019

Commits on Oct 3, 2019

Commits on Oct 29, 2019

Commits on Oct 30, 2019

Commits on Nov 26, 2019

Commits on Nov 27, 2019

Commits on Nov 28, 2019

Commits on Dec 26, 2019

Commits on Jan 27, 2020

Commits on May 24, 2020