Protoniuman FR patch 1 #6
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
protoniuman
commented
Dec 19, 2021
protoniuman
commented
- Fixes the return to line encoding issue
- some corrections and rephrases
Now only skipping the HTLC redemption if the AT is finished and the balance has been redeemed by the buyer. This allows HTLCs to be refunded for ATs that have been refunded or cancelled.
This was accidentally missed out of the original code. Some pre-updated nodes on the network will be missing this index, but we can use the upcoming "auto-bootstrap" feature to get those back.
Initially just deleting old and unused AT states, to get this table under control. I have had to delete them individually as the table can't handle complex queries due to its size. Nodes in pruning mode will be unable to serve older blocks to peers.
By default, this leaves only the last 1450 blocks in the database. Only applies when pruning mode is enabled.
…ent in the db to make decisions. As a side effect, this now prunes ATs up the the pruneBlockLimit too, rather than keeping the last 35 days or so. Will review this later but I don't think we will need the missing ones.
…mode is enabled. When switching from a full node to a pruning node, we need to delete most of the database contents. If we do this entirely as a background process, it is very slow and can interfere with syncing. However, if we take the approach of transferring only the necessary rows to a new table and then deleting the original table, this makes the process much faster. It was taking several days to delete the AT states in the background, but only a couple of minutes to copy them to a new table. The trade off is that we have to go through a form of "reshape" when starting the app for the first time after enabling pruning mode. But given that this is an opt-in mode, I don't think it will be a problem. Once the pruning is complete, it automatically performs a CHECKPOINT DEFRAG in order to shrink the database file size down to a fraction of what it was before. From this point, the original background process will run, but can be dialled right down so not to interfere with syncing.
…by more than one class. Note - the rebuildLatestAtStates() must never be used by two different classes at the same time, or AT states could be incorrectly deleted. It is okay at the moment as we don't run the AT states trimmer and pruner in the same app session. However we should probably synchronize this method so that we don't accidentally call it from two places in the future.
…controller.repository)
onlineAccountSignaturesMinLifetime reduced from 720 hours to 12 hours onlineAccountSignaturesMaxLifetime reduced from 888 hours to 24 hours These were using up too much space in the database and so it makes sense to trim them more aggressively (assuming testing goes well). We will now stop validating online account signatures after 12 hours, which should be more than enough confirmations, and we will discard them after 24 hours. Note: this will create some complexity once some of the network is running this code. It could cause out-of-sync nodes on old versions to start treating blocks as invalid from updated peers. It's likely not worth the complexity of a hard fork though, given that almost all nodes will be synced to the chain tip and will therefore be unaffected. And even with a hard fork, we'd still face this problem on out of date nodes.
This takes all trimmed blocks (which should now be all but the last 1450 or so) and moves them into flat files. Each file contains the serialized bytes of as many blocks that can fit within the file size target of 100MiB. As a result, the HSQLDB size drops to less than 1GB, making it much faster and easier to maintain. It also significantly reduces the total size of each full node, because the data is stored in a highly optimized way. HSQLDB then works similarly to the way it does in pruning mode - it holds all transactions, the latest state of every AT, as well as the full AT states data and hashes for the past 1450 blocks. Each archive file contains headers and indexes in order to quickly locate blocks. When a peer requests a block that is within the archive, the serialized bytes are sent directly without the need to go via a BlockData object. Now that there are no slow queries or data serialization processes needed, it should greatly speed up the block serving. The /block API endpoints have been modified in such a way that they will also check and retrieve blocks from the archive when needed. A lightweight "BlockArchive" table is needed in HSQLDB to map block heights to signatures minters and timestamps. It made more sense to keep SQL support for these basic attributes of each block. These are located in a separate table from the full blocks, in order to create a clear distinction between HSQLDB blocks and archived blocks, and also to speed up query times in the Blocks table, which is the one we are using 99% of the time. There is currently a restriction on the /admin/orphan API endpoint to prevent orphaning beyond the threshold of the block archive.
Block 536140 fix (same situation as block 535658)
…tches. Whilst not ideal, this is necessary to prevent the chain from getting stuck on future blocks due to duplicate name registrations. See Block535658.java for full details on this problem - this is simply a "catch-all" implementation of that class in order to futureproof this fix. There is still a database inconsistency to be solved, as some nodes are failing to add a registered name to their Names table the first time around, but this will take some time. Once fixed, this commit could potentially be reverted. Also added unit tests for both scenarios (same and different creator). TLDR: this allows all past and future invalid blocks caused by NAME_ALREADY_REGISTERED (by the same creator) to now be valid.
This reverts commit 278201e.
…the value the UI uses.
…of minters at each level
…ntroduction of ACCTv2
…troduction of ACCTv2
…the introduction of ACCTv2
…CTv3 and DogecoinACCTv3. It turns out that when you call SLEEP_UNTIL_MESSAGE, the AT resumes from that very same line on the next execution. The original code incorrectly assumed that it would execute from the restart position (SET_PCS). So sleeping can be thought of as pausing one execution half way through, rather than ending it. This caused a bug, because once the AT receives a transaction it wakes up and resumes from the SLEEP_UNTIL_MESSAGE line, which is after the refund check. Even when it loops back around again it lands on labelRedeemTxnLoop = codeByteBuffer.position(); which is again after the refund check. For now, the simplest fix is to only sleep when listed. We could have alternatively moved the SLEEP_UNTIL_MESSAGE above GET_BLOCK_TIMESTAMP, but this would still require users to send a random transaction to the AT to trigger the refund. Given that the ATs are only "alive" for 30 minutes once the trade begins, it's simpler to just execute every block and therefore allow the refunds to happen automatically.
… working as intended.
…offers APIs where 0 was being treated as zero instead of unlimited. In 2.1.1, unlimited results can be requested by removing the `limit` query string parameter completely, e.g: http://127.0.0.1:12391/crosschain/trades?foreignBlockchain=LITECOIN&minimumTimestamp=1638835200000&reverse=false
The main log4j version cannot be increased yet due to some incompatibilities with the Qortal code.
This reverts commit 6a9904f.
…ight} unless requested using the includesignatures=true query string parameter. This should fix issue where it would take up to 30 seconds to return for a recent block, and would consume masses of CPU due to having to base58 encode the online accounts signatures. Base58 is very slow and made this API endpoint almost unusable for recent blocks, due to them having untrimmed online accounts signatures.
Also renamed query string parameter to "includeOnlineSignatures" to make it clearer.
credit: stchoupi
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.