Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release Zebra 1.4.0 #7911

Merged
merged 4 commits into from
Nov 6, 2023
Merged

Release Zebra 1.4.0 #7911

merged 4 commits into from
Nov 6, 2023

Conversation

teor2345
Copy link
Contributor

@teor2345 teor2345 commented Nov 5, 2023

Close #7764

Prepare for the Release

  • Make sure there has been at least one successful full sync test since the last state change, or start a manual full sync.
  • Make sure the PRs with the new checkpoint hashes and missed dependencies are already merged.
    (See the release ticket checklist for details)

Summarise Release Changes

These steps can be done a few days before the release, in the same PR:

Change Log

Important: Any merge into main deletes any edits to the draft changelog.
Once you are ready to tag a release, copy the draft changelog into CHANGELOG.md.

We use the Release Drafter workflow to automatically create a draft changelog. We follow the Keep a Changelog format.

To create the final change log:

  • Copy the latest draft changelog into CHANGELOG.md (there can be multiple draft releases)
  • Delete any trivial changes
    • Put the list of deleted changelog entries in a PR comment to make reviewing easier
  • Combine duplicate changes
  • Edit change descriptions so they will make sense to Zebra users
  • Check the category for each change
    • Prefer the "Fix" category if you're not sure

README

README updates can be skipped for urgent releases.

Update the README to:

  • Remove any "Known Issues" that have been fixed since the last release.
  • Update the "Build and Run Instructions" with any new dependencies.
    Check for changes in the Dockerfile since the last tag: git diff <previous-release-tag> docker/Dockerfile.
  • If Zebra has started using newer Rust language features or standard library APIs, update the known working Rust version in the README, book, and Cargo.tomls

You can use a command like:

fastmod --fixed-strings '1.58' '1.65'

Create the Release PR

  • Push the updated changelog and README into a new branch
    for example: bump-v1.0.0 - this needs to be different to the tag name
  • Create a release PR by adding &template=release-checklist.md to the comparing url (Example).
  • Freeze the batched queue using Mergify.
  • Mark all the release PRs as Critical priority, so they go in the urgent Mergify queue.
  • Mark all non-release PRs with do-not-merge, because Mergify checks approved PRs against every commit, even when a queue is frozen.

Update Versions and End of Support

Update Zebra Version

Choose a Release Level

Zebra follows semantic versioning. Semantic versions look like: MAJOR.MINOR.PATCH[-TAG.PRE-RELEASE]

Choose a release level for zebrad. Release levels are based on user-visible changes from the changelog:

  • Mainnet Network Upgrades are major releases
  • significant new features or behaviour changes; changes to RPCs, command-line, or configs; and deprecations or removals are minor releases
  • otherwise, it is a patch release

Zebra's Rust API doesn't have any support or stability guarantees, so we keep all the zebra-* and tower-* crates on a beta pre-release version.

Update Crate Versions

If you're publishing crates for the first time, log in to crates.io,
and make sure you're a member of owners group.

Check that the release will work:

  • Update crate versions, commit the changes to the release branch, and do a release dry-run:
cargo release version --verbose --execute --allow-branch '*' --workspace --exclude zebrad beta
cargo release version --verbose --execute --allow-branch '*' --package zebrad patch # [ major | minor | patch ]
cargo release replace --verbose --execute --allow-branch '*' --package zebrad
cargo release commit --verbose --execute --allow-branch '*'

Crate publishing is automatically checked in CI using "dry run" mode.

Update End of Support

The end of support height is calculated from the current blockchain height:

  • Find where the Zcash blockchain tip is now by using a Zcash explorer or other tool.
  • Replace ESTIMATED_RELEASE_HEIGHT in end_of_support.rs with the height you estimate the release will be tagged.
Optional: calculate the release tagging height
  • Add 1152 blocks for each day until the release
  • For example, if the release is in 3 days, add 1152 * 3 to the current Mainnet block height

Update the Release PR

  • Push the version increments and the release constants to the release branch.

Publish the Zebra Release

Create the GitHub Pre-Release

  • Wait for all the release PRs to be merged
  • Create a new release using the draft release as a base, by clicking the Edit icon in the draft release
  • Set the tag name to the version tag,
    for example: v1.0.0
  • Set the release to target the main branch
  • Set the release title to Zebra followed by the version tag,
    for example: Zebra 1.0.0
  • Replace the prepopulated draft changelog in the release description with the final changelog you created;
    starting just after the title ## [Zebra ... of the current version being released,
    and ending just before the title of the previous release.
  • Mark the release as 'pre-release', until it has been built and tested
  • Publish the pre-release to GitHub using "Publish Release"
  • Delete all the draft releases from the list of releases

Test the Pre-Release

Publish Release

Publish Crates

  • Run cargo login
  • Run cargo clean in the zebra repo (optional)
  • Publish the crates to crates.io: cargo release publish --verbose --workspace --execute
  • Check that Zebra can be installed from crates.io:
    cargo install --locked --force --version 1.minor.patch zebrad && ~/.cargo/bin/zebrad
    and put the output in a comment on the PR.

Publish Docker Images

Release Failures

If building or running fails after tagging:

Tag a new release, following these instructions...
  1. Fix the bug that caused the failure
  2. Start a new patch release
  3. Skip the Release Preparation, and start at the Release Changes step
  4. Update CHANGELOG.md with details about the fix
  5. Follow the release checklist for the new Zebra version

@teor2345 teor2345 added P-Critical 🚑 C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG A-release Area: Zebra releases and release management labels Nov 5, 2023
@teor2345 teor2345 self-assigned this Nov 5, 2023
@teor2345 teor2345 requested review from a team as code owners November 5, 2023 22:28
@teor2345 teor2345 requested review from arya2 and upbqdn and removed request for a team November 5, 2023 22:28
@teor2345
Copy link
Contributor Author

teor2345 commented Nov 5, 2023

Here is a list of deleted changelog entries:

  • build(deps): bump Swatinem/rust-cache from 2.7.0 to 2.7.1 (#7825)
  • build(deps): bump actions/checkout from 4.1.0 to 4.1.1 (#7766)
  • build(deps): bump shimataro/ssh-key-action from 2.5.1 to 2.6.1 (#7767)
  • build(deps): bump the app group with 1 update (#7802)
  • build(deps): bump the async group with 2 updates (#7872)
  • build(deps): bump the data-structures group with 1 update (#7747)
  • build(deps): bump the devops group with 1 update (#7887)
  • build(deps): bump the formats group with 1 update (#7874)
  • build(deps): bump the formats group with 1 update (#7803)
  • build(deps): bump the formats group with 1 update (#7746)
  • build(deps): bump the prod group with 2 updates (#7881)
  • build(deps): bump the test group with 1 update (#7876)
  • build(deps): bump tj-actions/changed-files from 39.2.1 to 39.2.3 (#7768)
  • build(deps): bump tj-actions/changed-files from 39.2.3 to 39.2.4 (#7826)
  • change(ci): Remove duplicate CI jobs for getblocktemplate-rpcs (#7753)
  • change(deps): cargo update on commit bd06294 on 2023-11-01 (#7885)
  • change(dev doc): Document how to change branch protection rules (#7773)
  • change(devops): Split dependency upgrades into 4 large categories (#7848)
  • change(doc): Add instructions for new placeholder crates (#7751)
  • change(doc): Delete fixed macOS known issue from README.md (#7884)]
  • change(doc): Document how to fix deny.toml warnings (change(doc): Document how to fix deny.toml warnings #7806)07
  • change(doc): Document state format upgrade implementation and testing (#7799)
  • change(doc): Update README.md for known progress bar issue (#7761)
  • change(docs): Add changelog entry for stable mining feature (#7788)
  • change(docs): Replace doc.zebra.zfnd.org API documentation with docs.rs (#7886)
  • change(ops): Update full sync test URL in release ticket template (#7830)
  • change(pr): Add an author checklist to the PR template (#7832)
  • change(state): Refactor the naming of note commitment subtrees (#7855)
  • chore(clippy): Cleanup nightly clippy warnings from 2023-10-30 (#7868)
  • chore(consensus): Update mainnet and testnet checkpoints on 2023-10-30 (#7866)
  • cleanup(clippy): Resolve new nightly clippy and rustc lints (#7860)
  • cleanup(deps): Remove unused deny.toml duplicate dependency exceptions (#7818)
  • cleanup(state): Refactor subtree read functions to accept ranges and check memory first (#7739)
  • doc(devops): Explain how to change branch protection rules (#7883)
  • docs(build): Add ECC dependencies documentation (#7794)
  • fix(ci): Add missing timeouts to CI and CD tests (#7730)
  • fix(ci): In config tests, look for the config being successfully loaded by Zebra (#7900)
  • fix(ci): Increase Docker unit test timeout (#7814)
  • fix(ci): Limit parallel jobs launched by the crates workflow (#7754)
  • fix(ci): Skip mempool checks if it wasn't queried by lightwalletd (#7744)
  • fix(doc): Fix outdated Dockerfile features comment (#7815)
  • fix(docs): Fix some typos (#7862)
  • ref(workflows): consolidate workflows based on their purpose (#7616)
  • ref(workflows): use a single job to run GCP tests (#7682)

@teor2345
Copy link
Contributor Author

teor2345 commented Nov 5, 2023

  • If Zebra has started using newer Rust language features or standard library APIs, update the known working Rust version in the README, book, and Cargo.tomls

Zebra still compiles with Rust 1.70 with no warnings.

Clippy 1.70 fails with an unknown lint warning, which is expected, because we're using a newer lint for security reasons. So I'm going to upgrade the required Rust version to the latest stable release, which has no clippy warnings.

fastmod --fixed-strings '1.70' '1.73'

Copy link
Member

@upbqdn upbqdn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good.

mergify bot added a commit that referenced this pull request Nov 6, 2023
@mergify mergify bot merged commit 477da1a into main Nov 6, 2023
158 checks passed
@mergify mergify bot deleted the release140 branch November 6, 2023 12:51
@teor2345
Copy link
Contributor Author

teor2345 commented Nov 7, 2023

$ cargo release publish --verbose --workspace --execute
...
   Uploading zebra-network v1.0.0-beta.31 (/home/dev/zebra/zebra-network)
error: failed to publish to registry at https://crates.io

Caused by:
  failed to get a 200 OK response, got 500
  headers:
        HTTP/2 500
        content-type: text/plain; charset=utf-8
        content-length: 21
        server: Cowboy
        report-to: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1699321629&sid=af571f24-03ee-46d1-9f90-ab9030c2c74c&s=eoXUUHtjQSLpBF%2FUOS0CGOYhY%2FxP77np4Lg%2FXl7F3mk%3D"}]}
        reporting-endpoints: heroku-nel=https://nel.heroku.com/reports?ts=1699321629&sid=af571f24-03ee-46d1-9f90-ab9030c2c74c&s=eoXUUHtjQSLpBF%2FUOS0CGOYhY%2FxP77np4Lg%2FXl7F3mk%3D
        nel: {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
        access-control-allow-origin: *
        strict-transport-security: max-age=31536000; includeSubDomains
        date: Tue, 07 Nov 2023 01:47:19 GMT
        via: 1.1 vegur, 1.1 18973cd357a68e16bd20873be51e8596.cloudfront.net (CloudFront)
        x-cache: Error from cloudfront
        x-amz-cf-pop: SYD62-P1
        x-amz-cf-id: L9yEN9Xc8lLZFsePNwqv8xwrfGZ6zaP_reJroSV8aJgjC1yotyvyUg==
  
  body:
  Internal Server Error
Exit 101

I'll try again.

@teor2345
Copy link
Contributor Author

teor2345 commented Nov 7, 2023

Ok that worked the second time:

    Finished dev [optimized + debuginfo] target(s) in 2m 19s
    Packaged 117 files, 1.0MiB (238.7KiB compressed)
   Uploading zebrad v1.4.0 (/home/dev/zebra/zebrad)
    Uploaded zebrad v1.4.0 to registry `crates-io`
note: Waiting for `zebrad v1.4.0` to be available at registry `crates-io`.
You may press ctrl-c to skip waiting; the crate should be available shortly.
   Published zebrad v1.4.0 at registry `crates-io`

@teor2345
Copy link
Contributor Author

teor2345 commented Nov 7, 2023

There was an issue with chain history root validation during the release test, but I don't think we've changed that code recently:

2023-11-07T02:07:30.986898Z  INFO {net="Main"}:sync:try_to_sync:obtain_tips:checkpoint: zebra_consensus::checkpoint: verified checkpoint range block_count=361 current_range=(Excluded(Height(906888)), Included(Height(907249)))
2023-11-07T02:07:30.987788Z  INFO zebra_state::service::write: committing a block to the finalized state failed, resetting state queue error=CloneError { source: InvalidBlockCommitment(InvalidChainHistoryRoot { expected: [160, 116, 16, 229, 140, 235, 54, 202, 75, 21, 105, 110, 209, 37, 215, 125, 49, 25, 115, 18, 177, 49, 205, 198, 154, 185, 194, 110, 157, 70, 168, 38], actual: [173, 220, 236, 66, 148, 7
5, 229, 131, 172, 231, 193, 95, 187, 98, 186, 250, 118, 159, 142, 11, 8, 16, 227, 146, 234, 76, 210, 2, 251, 75, 12, 213] }) } last_valid_height=Some(Height(906888)) last_valid_hash=Some(block::Hash("000
00000009a21b1ba2e9e649cdbc8397ced98c6dcd1cdd336f2176532c20028"))
2023-11-07T02:07:31.681666Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_attempt: Some(Instant { tv_sec: 20056350, tv_nsec: 826457443 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to peer error=Serialization(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) candidate.addr=v4redacted:8233
2023-11-07T02:07:33.258533Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_
attempt: Some(Instant { tv_sec: 20056351, tv_nsec: 230622659 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to p
eer error=Elapsed(()) candidate.addr=v4redacted:8233
2023-11-07T02:07:34.084284Z  WARN {net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: error downloading and verifying block e=Invalid { error: Checkpoint { source: 
CommitCheckpointVerified(CloneError { source: InvalidBlockCommitment(InvalidChainHistoryRoot { expected: [160, 116, 16, 229, 140, 235, 54, 202, 75, 21, 105, 110, 209, 37, 215, 125, 49, 25, 115, 18, 177, 
49, 205, 198, 154, 185, 194, 110, 157, 70, 168, 38], actual: [173, 220, 236, 66, 148, 75, 229, 131, 172, 231, 193, 95, 187, 98, 186, 250, 118, 159, 142, 11, 8, 16, 227, 146, 234, 76, 210, 2, 251, 75, 12,
 213] }) }) }, height: Height(906889), hash: block::Hash("000000000153edfc9bda9d1b27fcb0c7b86b2dab4d15da4ce310211ce45ed56f") }
2023-11-07T02:07:34.276173Z  INFO {net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(906888))

This could be caused by peers distributing bad blocks on the network, or a corrupt local state. I'll try again.

full log of `cargo install --locked --force --version 1.minor.patch zebrad && ~/.cargo/bin/zebrad`
$ cargo install --locked --force --version 1.minor.patch zebrad && ~/.cargo/bin/zebrad
  Downloaded zebrad v1.4.0         
  Downloaded 1 crate (244.4 KB) in 1.90s
    Updating crates.io index        
  Installing zebrad v1.4.0           
    Updating crates.io index      
    Updating crates.io index    
   Compiling libc v0.2.149              
   Compiling proc-macro2 v1.0.69
   Compiling unicode-ident v1.0.12
   Compiling cfg-if v1.0.0            
   Compiling autocfg v1.1.0              
   Compiling version_check v0.9.4   
   Compiling serde v1.0.190             
   Compiling once_cell v1.18.0      
   Compiling typenum v1.17.0         
   Compiling subtle v2.4.1         
   Compiling log v0.4.20                    
   Compiling pin-project-lite v0.2.13  
   Compiling memchr v2.6.4            
   Compiling spin v0.5.2                       
   Compiling scopeguard v1.2.0           
   Compiling arrayvec v0.7.4            
   Compiling ppv-lite86 v0.2.17       
  Compiling byteorder v1.5.0
   Compiling futures-core v0.3.29
   Compiling radium v0.7.0
   Compiling crossbeam-utils v0.8.16
   Compiling nonempty v0.7.0
   Compiling tap v1.0.1
   Compiling funty v2.0.0
   Compiling syn v1.0.109
   Compiling cpufeatures v0.2.11
   Compiling arrayref v0.3.7
   Compiling constant_time_eq v0.3.0
   Compiling futures-sink v0.3.29
   Compiling static_assertions v1.1.0
   Compiling either v1.9.0
   Compiling rayon-core v1.12.0
   Compiling futures-channel v0.3.29
   Compiling zerocopy v0.7.21
   Compiling bytes v1.5.0
   Compiling futures-task v0.3.29
   Compiling semver v1.0.20
   Compiling glob v0.3.1
   Compiling futures-util v0.3.29
   Compiling futures-io v0.3.29
   Compiling crunchy v0.2.2
   Compiling lazy_static v1.4.0
   Compiling wyz v0.5.1
   Compiling pin-utils v0.1.0
   Compiling tinyvec_macros v0.1.1
   Compiling prettyplease v0.2.15
   Compiling itoa v1.0.9
   Compiling memuse v0.2.1
   Compiling tinyvec v1.6.0
   Compiling thiserror v1.0.50
   Compiling regex-syntax v0.8.2
   Compiling allocator-api2 v0.2.16
   Compiling tracing-core v0.1.32
   Compiling minimal-lexical v0.2.1
   Compiling blake2b_simd v1.0.2
   Compiling libloading v0.7.4
   Compiling ident_case v1.0.1
   Compiling opaque-debug v0.3.0
   Compiling smallvec v1.11.1
   Compiling equivalent v1.0.1
   Compiling generic-array v0.14.7
   Compiling ahash v0.8.6
   Compiling memoffset v0.9.0
   Compiling num-traits v0.2.17
   Compiling crossbeam-epoch v0.9.15
   Compiling slab v0.4.9
   Compiling num-integer v0.1.45
   Compiling num-bigint v0.4.4
   Compiling libm v0.2.8
   Compiling peeking_take_while v0.1.2
   Compiling shlex v1.2.0
   Compiling halo2_legacy_pdqsort v0.1.0
   Compiling bitflags v2.4.1
   Compiling lazycell v1.3.0
   Compiling clang-sys v1.6.1                                                                                                                                                                              
   Compiling base64ct v1.0.1
   Compiling rustc-hash v1.1.0
   Compiling lock_api v0.4.11
   Compiling incrementalmerkletree v0.5.0
   Compiling platforms v3.1.2
   Compiling untrusted v0.7.1
   Compiling aho-corasick v1.1.2
   Compiling serde_json v1.0.108
   Compiling unicode-normalization v0.1.22
   Compiling bitvec v1.0.1
   Compiling fnv v1.0.7
   Compiling rustc_version v0.4.0
   Compiling strsim v0.10.0
   Compiling zcash_encoding v0.2.0
   Compiling blake2s_simd v1.0.2
   Compiling ryu v1.0.15
   Compiling byte-slice-cast v1.2.2
   Compiling nom v7.1.3
   Compiling f4jumble v0.1.0
   Compiling bech32 v0.9.1
   Compiling equihash v0.2.0
   Compiling quote v1.0.33
   Compiling curve25519-dalek v4.1.1
   Compiling pkg-config v0.3.27
   Compiling rustix v0.38.21
   Compiling rustc-hex v2.1.0
   Compiling tower-service v0.3.2
   Compiling hashbrown v0.14.2
   Compiling signature v2.1.0
   Compiling bindgen v0.65.1
   Compiling vcpkg v0.2.15
   Compiling getrandom v0.1.16
   Compiling linux-raw-sys v0.4.10
   Compiling syn v2.0.38
   Compiling crossbeam-deque v0.8.3
   Compiling bridgetree v0.4.0
   Compiling jobserver v0.1.27
   Compiling getrandom v0.2.10
   Compiling num_cpus v1.16.0
   Compiling signal-hook-registry v1.4.1
   Compiling socket2 v0.5.5
   Compiling mio v0.8.9
   Compiling rand_core v0.6.4
   Compiling cc v1.0.83
   Compiling bitflags v1.3.2
   Compiling iana-time-zone v0.1.58
   Compiling humantime v2.1.0
   Compiling bindgen v0.68.1
   Compiling home v0.5.5
   Compiling unicode-width v0.1.11
   Compiling rand_chacha v0.3.1
   Compiling password-hash v0.3.2
   Compiling portable-atomic v1.5.1
   Compiling termcolor v1.3.0
   Compiling regex-syntax v0.6.29
   Compiling cxxbridge-flags v1.0.107
   Compiling parking_lot_core v0.9.9
   Compiling rand v0.8.5
   Compiling crossbeam-channel v0.5.8
   Compiling codespan-reporting v0.11.1
   Compiling rayon v1.8.0
   Compiling itertools v0.11.0
   Compiling tower-layer v0.3.2
   Compiling hdrhistogram v7.5.2
   Compiling overload v0.1.1
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling inout v0.1.3
   Compiling rustversion v1.0.14
   Compiling universal-hash v0.5.1
   Compiling aead v0.5.2
   Compiling digest v0.10.7
   Compiling regex-automata v0.4.3
   Compiling poly1305 v0.8.0
   Compiling console v0.15.7
   Compiling nu-ansi-term v0.46.0
   Compiling sha2 v0.10.8
   Compiling pbkdf2 v0.10.1
   Compiling cexpr v0.6.0
   Compiling hmac v0.12.1
   Compiling ripemd v0.1.3
   Compiling fixed-hash v0.7.0
   Compiling rand_core v0.5.1
   Compiling nanorand v0.7.0
   Compiling spin v0.9.8
   Compiling bs58 v0.5.0
   Compiling tracing-log v0.1.4
   Compiling thread_local v1.1.7
   Compiling sharded-slab v0.1.7
   Compiling option-ext v0.2.0
   Compiling unicode-bidi v0.3.13
   Compiling zcash_address v0.3.0
   Compiling secp256k1-sys v0.8.1
   Compiling ring v0.16.20
   Compiling link-cplusplus v1.0.9
   Compiling libz-sys v1.1.12
   Compiling lz4-sys v1.9.4
   Compiling which v4.4.2
   Compiling cxx v1.0.107
   Compiling libgit2-sys v0.16.1+1.7.1
   Compiling number_prefix v0.4.0
   Compiling percent-encoding v2.3.0
   Compiling form_urlencoded v1.2.0
   Compiling indicatif v0.17.7
   Compiling ff v0.13.0
   Compiling regex-automata v0.1.10
   Compiling idna v0.4.0
   Compiling dirs-sys v0.4.1
   Compiling parking_lot v0.12.1
   Compiling rand_chacha v0.2.2
   Compiling group v0.13.0
   Compiling unicase v2.7.0
   Compiling xdg v2.5.2
   Compiling parking_lot_core v0.8.6
   Compiling known-folders v1.0.1
   Compiling powerfmt v0.2.0                                                                                                                                                                               
   Compiling pairing v0.23.0
   Compiling pasta_curves v0.5.1
   Compiling bls12_381 v0.8.0
   Compiling fastrand v2.0.1
   Compiling regex v1.10.2
   Compiling rlimit v0.10.1
   Compiling utf8parse v0.2.1
   Compiling httparse v1.8.0
   Compiling anyhow v1.0.75
   Compiling anstyle-parse v0.2.2
   Compiling maybe-rayon v0.1.1
   Compiling bellman v0.14.0
   Compiling url v2.4.1
   Compiling deranged v0.3.9
   Compiling rand v0.7.3
   Compiling dirs v5.0.1
   Compiling backtrace v0.3.69
   Compiling http v0.2.9
   Compiling matchers v0.1.0
   Compiling bstr v1.7.0
   Compiling jubjub v0.10.0
   Compiling instant v0.1.12
   Compiling try-lock v0.2.4
   Compiling adler v1.0.2
   Compiling anstyle v1.0.4
   Compiling colorchoice v1.0.0
   Compiling vergen v8.2.5
   Compiling anstyle-query v1.0.0
   Compiling eyre v0.6.8
   Compiling num_threads v0.1.6
   Compiling gimli v0.28.0
   Compiling darling_core v0.20.3
   Compiling cxx-gen v0.7.110
   Compiling time-core v0.1.2
   Compiling anstream v0.6.4
   Compiling env_logger v0.7.1
   Compiling http-body v0.4.5
   Compiling globset v0.4.13
   Compiling time v0.3.30
   Compiling tempfile v3.8.1
   Compiling quickcheck v0.9.2
   Compiling miniz_oxide v0.7.1
   Compiling want v0.3.1
   Compiling socket2 v0.4.10
   Compiling object v0.32.1
   Compiling mset v0.1.1
   Compiling cfg-if v0.1.10
   Compiling wagyu-zcash-parameters-2 v0.2.0
   Compiling clap_lex v0.6.0
   Compiling heck v0.4.1
   Compiling indenter v0.3.3
   Compiling httpdate v1.0.3
   Compiling addr2line v0.21.0
   Compiling wagyu-zcash-parameters-1 v0.2.0
   Compiling wagyu-zcash-parameters-4 v0.2.0
   Compiling rustc-demangle v0.1.23
   Compiling wagyu-zcash-parameters-3 v0.2.0
   Compiling toml v0.5.11
   Compiling wagyu-zcash-parameters-6 v0.2.0
   Compiling hex-literal v0.4.1
   Compiling wagyu-zcash-parameters-5 v0.2.0
   Compiling unicode-xid v0.2.4
   Compiling wagyu-zcash-parameters v0.2.0
   Compiling clap_builder v4.4.7
   Compiling synstructure v0.12.6
   Compiling net2 v0.2.39
   Compiling parking_lot v0.11.2
   Compiling serde_derive v1.0.190
   Compiling tracing-attributes v0.1.27
   Compiling zeroize_derive v1.4.2
   Compiling futures-macro v0.3.29
   Compiling tokio-macros v2.1.0
   Compiling thiserror-impl v1.0.50
   Compiling pin-project-internal v1.1.3
   Compiling darling_macro v0.20.3
   Compiling impl-trait-for-tuples v0.2.2
   Compiling curve25519-dalek-derive v0.1.1
   Compiling displaydoc v0.2.4
   Compiling metrics-macros v0.7.0
   Compiling cxxbridge-macro v1.0.107
   Compiling quickcheck_macros v0.9.1
   Compiling proc-macro-crate v0.1.5
   Compiling clap_derive v4.4.7
   Compiling winnow v0.5.18
   Compiling owo-colors v3.5.0
   Compiling abscissa_derive v0.7.0
   Compiling wait-timeout v0.2.0
   Compiling jsonrpc-derive v18.0.0
   Compiling darling v0.20.3
   Compiling zeroize v1.6.0
   Compiling arc-swap v1.6.0
   Compiling serde_with_macros v3.4.0
   Compiling cipher v0.4.4
   Compiling tokio v1.33.0
   Compiling bip0039 v0.10.1
   Compiling chacha20 v0.9.1
   Compiling cbc v0.1.2
   Compiling aes v0.8.3
   Compiling fpe v0.6.1
   Compiling ordered-map v0.4.2
   Compiling fs-err v2.9.0
   Compiling metrics v0.21.1
   Compiling chacha20poly1305 v0.10.1
   Compiling canonical-path v2.0.2
   Compiling zcash_note_encryption v0.4.0
   Compiling pin-project v1.1.3
   Compiling flume v0.10.14
   Compiling atty v0.2.14
   Compiling tracing v0.1.40
   Compiling howudoin v0.1.2
   Compiling halo2_proofs v0.3.0
   Compiling tracing-subscriber v0.3.17
   Compiling tracing-futures v0.2.5
   Compiling color-eyre v0.6.2
   Compiling clap v4.4.7
   Compiling tracing-error v0.2.0
   Compiling zcash_script v0.1.14
   Compiling tracing-appender v0.2.2
   Compiling librocksdb-sys v0.11.0+8.1.1
   Compiling futures-executor v0.3.29
   Compiling futures v0.3.29
   Compiling tokio-util v0.7.10
   Compiling tokio-util v0.6.10
   Compiling hyper v0.14.27
   Compiling tower v0.4.13
   Compiling tokio-stream v0.1.14
   Compiling tower-batch-control v0.2.41-beta.7
   Compiling tower-fallback v0.2.41-beta.7
   Compiling hex v0.4.3
   Compiling indexmap v2.1.0
   Compiling parity-scale-codec v3.6.5
   Compiling ed25519 v2.2.3
   Compiling serde-big-array v0.5.1
   Compiling chrono v0.4.31
   Compiling serde_with v3.4.0
   Compiling bitflags-serde-legacy v0.1.1
   Compiling humantime-serde v1.1.1
   Compiling bincode v1.3.3
   Compiling serde_spanned v0.6.4
   Compiling toml_datetime v0.6.5
   Compiling secrecy v0.8.0
   Compiling uint v0.9.5
   Compiling reddsa v0.5.1
   Compiling halo2_gadgets v0.3.0
   Compiling redjubjub v0.7.0
   Compiling toml_edit v0.20.7
   Compiling impl-codec v0.6.0
   Compiling primitive-types v0.11.1
   Compiling abscissa_core v0.7.0
   Compiling jsonrpc-core v18.0.0
   Compiling x25519-dalek v2.0.0
   Compiling ed25519-zebra v4.0.3
   Compiling secp256k1 v0.26.0
   Compiling git2 v0.18.1
   Compiling zcash_history v0.3.0
   Compiling jsonrpc-server-utils v18.0.0
   Compiling hdwallet v0.4.1
   Compiling jsonrpc-http-server v18.0.0
   Compiling toml v0.8.6
   Compiling zebrad v1.4.0
   Compiling orchard v0.6.0
   Compiling zcash_primitives v0.13.0-rc.1
   Compiling zebra-chain v1.0.0-beta.31
   Compiling zcash_proofs v0.13.0-rc.1
   Compiling zebra-node-services v1.0.0-beta.31
   Compiling zebra-network v1.0.0-beta.31
   Compiling zebra-script v1.0.0-beta.31
   Compiling rocksdb v0.21.0                                                                                                                                                                               
   Compiling zebra-state v1.0.0-beta.31                                                                                                                                                                    
   Compiling zebra-consensus v1.0.0-beta.31                                                                                                                                                                
   Compiling zebra-rpc v1.0.0-beta.31                                                                                                                                                                      
    Finished release [optimized + debuginfo] target(s) in 2m 11s                                                                                                                                           
   Replacing /home/dev/.cargo/bin/zebrad                                                                                                                                                                   
    Replaced package `zebrad v1.3.0` with `zebrad v1.4.0` (executable `zebrad`)                                                                                                                            
                                                                                   
             X@8:::::::@X                      S888@t             S888%        
          @::X:;SX;%8S8@@:X::8              S:         .8      @.         .@    
       @:;@[email protected]  ;;t;. XX.;@@;:@         ;               %  ;               X  
     8:@88     .%@[email protected]:     888:8      8                 t%                 @ 
    :;SX    XS8;::@X@8::;8t8:   8;;:@                                           
  @::@    88:::8 8::88::::::X@    8::8  @                                      8
 @:XS8   ;::::::8.88%8@S88::::8:  8S8:8 X                                      X
 ::;8 t8S8::::@St 88SX @SS@8:8S;8S @@:: 8                                      8
8:XX  %%@@::X% 8:  X@ t:8t X ..%t8  ;8:8                                        
::8t:  tX:X8tS 8: :@ ;[email protected] tt888:  @8:: X                                    % 
::8S:  @:8Xtt% 8: S 8 8S.   .8;::.  88::  %                                  t  
::X8   S@8   8:8.@8@8Xt S  .%8;S    %8::   8                                8   
 ::@8  S .   @:::::::X;@ 8.88S8    @8;:X     %                            %     
 8:X8X .@:X88:::::::::888 @;:X    %88::        %                        %       
  @::S    888@8:::::::::8X; 8     8::8           8                    8         
   X:;S@   tXX8;::::::::S::@8   8;S:8              S.              :t           
     ::8S@     :S88XX@8X:     @XX::                   %          %              
       8:;%X8.:S        t; X8%S:8X                      @      S                
         X8::8SS8SX8@@X@8;8S:8X                           S  %                  
              X@8:::::::@XS                                %                    

Thank you for running a mainnet zebrad 1.4.0 node! 
You're helping to strengthen the network and contributing to a social good :)
2023-11-07T02:07:15.254903Z  INFO zebrad::components::tracing::component: started tracing component filter="info" TRACING_STATIC_MAX_LEVEL=LevelFilter::TRACE LOG_STATIC_MAX_LEVEL=Trace
2023-11-07T02:07:15.254963Z  INFO zebrad::components::tracing::component: set 'tracing.progress_bar ="summary"' in zebrad.toml to activate progress bars
2023-11-07T02:07:15.254975Z  INFO zebrad::application: Diagnostic Metadata:
version: 1.4.0
Zcash network: Mainnet
running state version: 25.3.0
initial disk state version: 25.3.0
features: default,getblocktemplate_rpcs,howudoin,indicatif,progress_bar,release_max_level_info
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.75.0-nightly
rust release date: 2023-11-02
optimization level: 3
debug checks: true
2023-11-07T02:07:15.254988Z  INFO zebrad::application: loaded zebrad config config_path=Some("/home/dev/.config/zebrad.toml") config=ZebradConfig { consensus: Config { checkpoint_sync: true }, metrics: C
onfig { endpoint_addr: None }, network: Config { listen_addr: 0.0.0.0:8233, network: Mainnet, initial_mainnet_peers: {"dnsseed.z.cash:8233", "dnsseed.str4d.xyz:8233", "mainnet.seeder.zfnd.org:8233", "mai
nnet.is.yolo.money:8233"}, initial_testnet_peers: {"dnsseed.testnet.z.cash:18233", "testnet.seeder.zfnd.org:18233", "testnet.is.yolo.money:18233"}, cache_dir: IsEnabled(true), peerset_initial_target_size
: 25, crawl_new_peer_interval: 61s, max_connections_per_ip: 1 }, state: Config { cache_dir: "/home/dev/.cache/zebra", ephemeral: false, delete_old_database: true, debug_stop_at_height: None, debug_validi
ty_check_interval: None }, tracing: Config { inner: InnerConfig { use_color: true, force_use_color: false, filter: Some("info"), buffer_limit: 128000, endpoint_addr: None, flamegraph: None, progress_bar:
 None, log_file: None, use_journald: false } }, sync: Config { download_concurrency_limit: 50, checkpoint_verify_concurrency_limit: 1000, full_verify_concurrency_limit: 20, parallel_cpu_threads: 0 }, mem
pool: Config { tx_cost_limit: 80000000, eviction_memory_time: 3600s, debug_enable_at_height: None }, rpc: Config { listen_addr: None, parallel_cpu_threads: 1, debug_force_finished_sync: false }, mining: 
Config { miner_address: None, extra_coinbase_data: None, debug_like_zcashd: true } }
2023-11-07T02:07:15.255155Z  INFO {net="Main"}: zebrad::application: initialized rayon thread pool for CPU-bound tasks num_threads=36
2023-11-07T02:07:15.256463Z  INFO {net="Main"}: zebrad::commands::start: Starting zebrad
2023-11-07T02:07:15.256535Z  INFO {net="Main"}: zebrad::commands::start: initializing node state
2023-11-07T02:07:15.262748Z  INFO {net="Main"}: zebrad::commands::start: opening database, this may take a few minutes
2023-11-07T02:07:15.262894Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open current database format running_version=25.3.0
2023-11-07T02:07:15.263059Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1024 min_limit=512 ideal_limit=1024
2023-11-07T02:07:15.760570Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /home/dev/.cache/zebra/state/v25/mainnet
2023-11-07T02:07:15.761563Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(906888), block::Hash("00000000009a21b1ba2e9e649cdbc8397ced98c6dcd1cdd336f2176532c20028")
))
2023-11-07T02:07:15.761695Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: checking database format produced by a previous zebra instance is current and valid running_version=25.3.0
2023-11-07T02:07:15.762401Z  INFO zebra_state::service: starting legacy chain check
2023-11-07T02:07:15.762729Z  INFO zebra_state::service: cached state consensus branch is valid: no legacy chain found
2023-11-07T02:07:15.762950Z  INFO {net="Main"}: zebrad::commands::start: initializing network
2023-11-07T02:07:15.763019Z  INFO {net="Main"}:open_listener{addr=0.0.0.0:8233}: zebra_network::peer_set::initialize: Trying to open Zcash protocol endpoint at 0.0.0.0:8233...
2023-11-07T02:07:15.763086Z  INFO {net="Main"}:open_listener{addr=0.0.0.0:8233}: zebra_network::peer_set::initialize: Opened Zcash protocol endpoint at 0.0.0.0:8233
2023-11-07T02:07:15.763193Z  INFO {net="Main"}: zebra_network::address_book_updater: starting the address book updater
2023-11-07T02:07:16.168068Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: resolved seed peer IP addresses seed="dnsseed.z.cash:8233" remote_ip_count=25
2023-11-07T02:07:17.237281Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: resolved seed peer IP addresses seed="dnsseed.str4d.xyz:8233" remote_ip_count=30
2023-11-07T02:07:17.817735Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=25.3.0 inital_disk_version=25.3.0
2023-11-07T02:07:20.765189Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: DNS timeout resolving peer IP addresses host="mainnet.is.yolo.money:8233" e=Elapsed(())
2023-11-07T02:07:20.765243Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: Seed peer DNS resolution failed, checking for addresses from other seed peers host="mainnet.is.yolo.money:8233" at
tempts=1
2023-11-07T02:07:20.765250Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: DNS timeout resolving peer IP addresses host="mainnet.seeder.zfnd.org:8233" e=Elapsed(())
2023-11-07T02:07:20.765252Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: Seed peer DNS resolution failed, checking for addresses from other seed peers host="mainnet.seeder.zfnd.org:8233" 
attempts=1
2023-11-07T02:07:20.765981Z  INFO {net="Main"}:add_initial_peers: zebra_network::config: loaded cached peer IP addresses cached_ip_count=75 peer_cache_file="/home/dev/.cache/zebra/network/mainnet.peers"
2023-11-07T02:07:20.766008Z  INFO {net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: limiting the initial peers list from 96 to 25
2023-11-07T02:07:20.766173Z  INFO {net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: connecting to initial peer set ipv4_peer_count=24 ipv6_peer_count=1
2023-11-07T02:07:21.768116Z  INFO {net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=5 errors=1 addr=v6redacted:8233 e=Network is unreachable
 (os error 101)
2023-11-07T02:07:25.669046Z  INFO {net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=23 errors=2 addr=v4redacted:8233 e=request timed out
2023-11-07T02:07:25.669124Z  INFO {net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: finished connecting to initial seed and disk cache peers handshake_success_total=23 handshake_error_
total=2 outbound_connections=23
2023-11-07T02:07:25.669351Z  INFO {net="Main"}: zebra_network::peer_set::initialize: sending initial request for peers active_initial_peer_count=23
2023-11-07T02:07:25.670082Z  INFO {net="Main"}: zebrad::commands::start: initializing verifiers
2023-11-07T02:07:25.670104Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}: zebra_network::peer_set::initialize: starting the peer address crawler crawl_new_peer_interval=61s outbound_connectio
ns=23
2023-11-07T02:07:25.670138Z  INFO {net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: starting state checkpoint validation
2023-11-07T02:07:25.675308Z  INFO {net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: initializing block verifier router tip=Some((Height(906888), block::
Hash("00000000009a21b1ba2e9e649cdbc8397ced98c6dcd1cdd336f2176532c20028"))) max_checkpoint_height=Height(2275898)
2023-11-07T02:07:25.675449Z  INFO {net="Main"}: zebrad::commands::start: initializing syncer
2023-11-07T02:07:25.675493Z  INFO {net="Main"}: zebrad::commands::start: initializing mempool
2023-11-07T02:07:25.675513Z  INFO {net="Main"}: zebrad::commands::start: fully initializing inbound peer request handler
2023-11-07T02:07:25.675517Z  INFO {net="Main"}: zebrad::commands::start: spawning RPC server
2023-11-07T02:07:25.675556Z  INFO {net="Main"}: zebrad::commands::start: spawning block gossip task
2023-11-07T02:07:25.675564Z  INFO {net="Main"}: zebrad::commands::start: spawning mempool queue checker task
2023-11-07T02:07:25.675568Z  INFO {net="Main"}: zebrad::commands::start: spawning mempool transaction gossip task
2023-11-07T02:07:25.675571Z  INFO {net="Main"}: zebrad::commands::start: spawning delete old databases task
2023-11-07T02:07:25.675574Z  INFO {net="Main"}: zebrad::commands::start: spawning progress logging task
2023-11-07T02:07:25.675572Z  INFO {net="Main"}: zebrad::components::mempool::queue_checker: initializing mempool queue checker task
2023-11-07T02:07:25.675572Z  INFO {net="Main"}: zebrad::components::sync::gossip: initializing block gossip task
2023-11-07T02:07:25.675583Z  INFO {net="Main"}: zebrad::commands::start: spawning end of support checking task
2023-11-07T02:07:25.675583Z  INFO {net="Main"}: zebrad::components::mempool::gossip: initializing transaction gossip task
2023-11-07T02:07:25.675587Z  INFO {net="Main"}: zebrad::commands::start: spawning mempool crawler task
2023-11-07T02:07:25.675590Z  INFO {net="Main"}: zebrad::commands::start: spawning syncer task
2023-11-07T02:07:25.675593Z  INFO {net="Main"}: zebrad::commands::start: spawned initial Zebra tasks
2023-11-07T02:07:25.675597Z  INFO {net="Main"}: zebra_state::config: checking for old database versions
2023-11-07T02:07:25.675619Z  INFO {net="Main"}: zebrad::components::sync::end_of_support: Starting end of support task
2023-11-07T02:07:25.675638Z  INFO {net="Main"}: zebrad::components::mempool::crawler: initializing mempool crawler task
2023-11-07T02:07:25.675654Z  INFO {net="Main"}: zebra_state::config: finished old database version cleanup task
2023-11-07T02:07:25.683220Z  INFO {net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(906888))
2023-11-07T02:07:25.751915Z  INFO {net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: state is not fully synced yet, remaining checkpoints will be verified during syncing
2023-11-07T02:07:25.751930Z  INFO {net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: finished state checkpoint validation
2023-11-07T02:07:27.641272Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_attempt: Some(Instant { tv_sec: 20056348, tv_nsec: 400784629 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to peer error=Os { code: 111, kind: ConnectionRefused, message: "Connection refused" } candidate.addr=v4redacted:8233
2023-11-07T02:07:28.127550Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_
attempt: Some(Instant { tv_sec: 20056348, tv_nsec: 502002926 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to p
eer error=Serialization(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) candidate.addr=v4redacted:8233
2023-11-07T02:07:30.025374Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_attempt: Some(Instant { tv_sec: 20056347, tv_nsec: 996885794 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to peer error=Elapsed(()) candidate.addr=v4redacted:8233
2023-11-07T02:07:30.394198Z  INFO {net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=499 extra_hashes=0 lookahead_limit=1000 st
ate_tip=Some(Height(906888))
2023-11-07T02:07:30.834088Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_attempt: Some(Instant { tv_sec: 20056348, tv_nsec: 804865328 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to peer error=Elapsed(()) candidate.addr=v4redacted:8233
2023-11-07T02:07:30.986898Z  INFO {net="Main"}:sync:try_to_sync:obtain_tips:checkpoint: zebra_consensus::checkpoint: verified checkpoint range block_count=361 current_range=(Excluded(Height(906888)), Included(Height(907249)))
2023-11-07T02:07:30.987788Z  INFO zebra_state::service::write: committing a block to the finalized state failed, resetting state queue error=CloneError { source: InvalidBlockCommitment(InvalidChainHistoryRoot { expected: [160, 116, 16, 229, 140, 235, 54, 202, 75, 21, 105, 110, 209, 37, 215, 125, 49, 25, 115, 18, 177, 49, 205, 198, 154, 185, 194, 110, 157, 70, 168, 38], actual: [173, 220, 236, 66, 148, 7
5, 229, 131, 172, 231, 193, 95, 187, 98, 186, 250, 118, 159, 142, 11, 8, 16, 227, 146, 234, 76, 210, 2, 251, 75, 12, 213] }) } last_valid_height=Some(Height(906888)) last_valid_hash=Some(block::Hash("000
00000009a21b1ba2e9e649cdbc8397ced98c6dcd1cdd336f2176532c20028"))
2023-11-07T02:07:31.681666Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_attempt: Some(Instant { tv_sec: 20056350, tv_nsec: 826457443 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to peer error=Serialization(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) candidate.addr=v4redacted:8233
2023-11-07T02:07:33.258533Z  INFO {net="Main"}:crawl_and_dial{new_peer_interval=61s}:dial{candidate=MetaAddr { addr: v4redacted:8233, services: None, untrusted_last_seen: None, last_response: None, last_
attempt: Some(Instant { tv_sec: 20056351, tv_nsec: 230622659 }), last_failure: None, last_connection_state: AttemptPending }}: zebra_network::peer_set::initialize: failed to make outbound connection to p
eer error=Elapsed(()) candidate.addr=v4redacted:8233
2023-11-07T02:07:34.084284Z  WARN {net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: error downloading and verifying block e=Invalid { error: Checkpoint { source: 
CommitCheckpointVerified(CloneError { source: InvalidBlockCommitment(InvalidChainHistoryRoot { expected: [160, 116, 16, 229, 140, 235, 54, 202, 75, 21, 105, 110, 209, 37, 215, 125, 49, 25, 115, 18, 177, 
49, 205, 198, 154, 185, 194, 110, 157, 70, 168, 38], actual: [173, 220, 236, 66, 148, 75, 229, 131, 172, 231, 193, 95, 187, 98, 186, 250, 118, 159, 142, 11, 8, 16, 227, 146, 234, 76, 210, 2, 251, 75, 12,
 213] }) }) }, height: Height(906889), hash: block::Hash("000000000153edfc9bda9d1b27fcb0c7b86b2dab4d15da4ce310211ce45ed56f") }
2023-11-07T02:07:34.276173Z  INFO {net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(906888))
2023-11-07T02:07:35.676684Z  INFO {net="Main"}: zebrad::components::sync::end_of_support: Checking if Zebra release is inside support range ...
2023-11-07T02:07:35.676710Z  INFO {net="Main"}: zebrad::components::sync::end_of_support: Zebra release is supported until block 2419024, please report bugs at https://github.com/ZcashFoundation/zebra/is
sues
2023-11-07T02:07:45.671335Z  INFO {net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=73 peer_cache_file="/home/dev/.cache/zebra/network/mainnet.peers
2023-11-07T02:08:25.692188Z  INFO {net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=39.510% current_height=Height(906888) network_upgrade=Heartwood remaining_
sync_blocks=1388442 time_since_last_state_block=0s
2023-11-07T02:08:41.277377Z  INFO {net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(906888))
2023-11-07T02:08:44.555171Z  INFO {net="Main"}:sync:try_to_sync:obtain_tips:checkpoint: zebra_consensus::checkpoint: verified checkpoint range block_count=761 current_range=(Excluded(Height(906888)), Inc
luded(Height(907649)))
2023-11-07T02:08:44.555582Z  INFO zebra_state::service::write: committing a block to the finalized state failed, resetting state queue error=CloneError { source: InvalidBlockCommitment(InvalidChainHistor
yRoot { expected: [160, 116, 16, 229, 140, 235, 54, 202, 75, 21, 105, 110, 209, 37, 215, 125, 49, 25, 115, 18, 177, 49, 205, 198, 154, 185, 194, 110, 157, 70, 168, 38], actual: [173, 220, 236, 66, 148, 7
5, 229, 131, 172, 231, 193, 95, 187, 98, 186, 250, 118, 159, 142, 11, 8, 16, 227, 146, 234, 76, 210, 2, 251, 75, 12, 213] }) } last_valid_height=Some(Height(906888)) last_valid_hash=Some(block::Hash("000
00000009a21b1ba2e9e649cdbc8397ced98c6dcd1cdd336f2176532c20028"))
2023-11-07T02:08:44.559803Z  WARN {net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: error downloading and verifying block e=Invalid { error: Checkpoint { source: 
CommitCheckpointVerified(CloneError { source: InvalidBlockCommitment(InvalidChainHistoryRoot { expected: [160, 116, 16, 229, 140, 235, 54, 202, 75, 21, 105, 110, 209, 37, 215, 125, 49, 25, 115, 18, 177, 
49, 205, 198, 154, 185, 194, 110, 157, 70, 168, 38], actual: [173, 220, 236, 66, 148, 75, 229, 131, 172, 231, 193, 95, 187, 98, 186, 250, 118, 159, 142, 11, 8, 16, 227, 146, 234, 76, 210, 2, 251, 75, 12,
 213] }) }) }, height: Height(906889), hash: block::Hash("000000000153edfc9bda9d1b27fcb0c7b86b2dab4d15da4ce310211ce45ed56f") }
2023-11-07T02:08:44.560256Z  INFO {net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(906888))
^C2023-11-07T02:08:47.293332Z  INFO {net="Main"}:sig{kind=SignalKind(2) name="SIGINT"}: zebrad::signal: received SIGINT, starting shutdown
2023-11-07T02:08:47.293407Z  INFO {net="Main"}: zebrad::components::tokio: waiting for async tokio tasks to shut down TOKIO_SHUTDOWN_TIMEOUT=20s
2023-11-07T02:08:47.293685Z  INFO zebra_state::service::write: StateService closed the block reset channel. Is Zebra shutting down?
2023-11-07T02:08:47.294853Z  INFO {net="Main"}: zebra_network::address_book_updater: stopping address book updater error=Err(AllAddressBookUpdaterSendersClosed)
2023-11-07T02:08:47.300357Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: checking new blocks were written in current database format running_version=25.3.0
2023-11-07T02:08:47.300367Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: checking database format produced by new blocks in this instance is valid running_version=25.3.0
2023-11-07T02:08:49.005330Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=25.3.0 inital_disk_version=25.3.0
2023-11-07T02:08:49.005714Z  INFO zebra_state::service: waiting for the block write task to finish
2023-11-07T02:08:49.005770Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: checking new blocks were written in current database format running_version=25.3.0
2023-11-07T02:08:49.005774Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: checking database format produced by new blocks in this instance is valid running_version=25.3.0
2023-11-07T02:08:50.634425Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=25.3.0 inital_disk_version=25.3.0
2023-11-07T02:08:50.641930Z  INFO {net="Main"}: zebrad::components::tokio: shutting down Zebra
2023-11-07T02:08:50.641946Z  INFO {net="Main"}: zebrad::commands::start: stopping zebrad

@teor2345
Copy link
Contributor Author

teor2345 commented Nov 7, 2023

All my other Zebra instances are running fine on 1.4.0, including a fully synced state, and starting from genesis, on both mainnet and testnet. So this looks like a corrupt state.

@teor2345
Copy link
Contributor Author

teor2345 commented Nov 7, 2023

(Once the corrupt state was deleted, the same command ran fine.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-release Area: Zebra releases and release management C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Publish next Zebra release: 1.4.0
2 participants