Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return status code after testing #2069

Merged
merged 3 commits into from
Apr 27, 2014
Merged

Return status code after testing #2069

merged 3 commits into from
Apr 27, 2014

Conversation

AlfonZ42
Copy link
Contributor

Currently, even if some tests fail, status code returned is 0, thus CI doesn't find out that there was failure and build doesn't fail. This changes it.

Current behavior:

  • Non-Windows, tests pass - process finishes, status code is zero
  • Non-Windows, tests fail - process finishes, status code is zero
  • Windows, tests pass - process keeps running
  • Windows, tests fail - process keeps running

Keeping the process running under Windows is desired behavior - see 9ad7e38.

New behavior:

  • Non-Windows, tests pass - process finishes, status code is zero
  • Non-Windows, tests fail - process finishes, status code is non-zero (CI reports failed build)
  • Windows, tests pass - process keeps running
  • Windows, tests fail - process keeps running

I have disabled verbose mode for now, even though 9ad7e38 intended to have it enabled.
Due to bug (which this change fixes) it wasn't taken into account. Enabling it would report passed tests as well, making failed tests less visible, see below.

Non-verbose output (with fake failed test added):

-- Starting suite "test/test_maps", 4 test(s)
  .F..
-- Starting suite "main", 87 test(s)
  ...................................................................
  ....................
---- Testing finished in 4.00 s, with 911 assertion(s) ----
  90 passed, 1 failed, 0 error(s), 0 skipped.
FAIL: test/test_maps.test_fakefail (0.00ms): (Failed) - Fake failed test. (286)

Verbose output:

Starting tests, 16 suite(s)
-- Starting suite "test/test_maps", 4 test(s)
PASS: test_breakable_blocks (0.00ms)
FAIL: test_fakefail (0.00ms): (Failed) - Fake failed test. (286)
PASS: test_music (140.63ms)
PASS: test_doors (15.63ms)
    Finished suite "test/test_maps", +3 -1 E0 s0
-- Starting suite "main", 87 test(s)
PASS: test_fsm_cancel_onbefore_event (0.00ms)
PASS: test_load_current (46.88ms)
PASS: test_tween_handle_anything (0.00ms)
PASS: test_get_action (0.00ms)
PASS: test_queue_push_events (0.00ms)
PASS: test_split_string (0.00ms)
PASS: test_sparkle_same_versions (0.00ms)
PASS: test_fsm_state_transition (0.00ms)
PASS: test_give_gcc_key (0.00ms)
PASS: test_transition_new (0.00ms)
PASS: test_dialog_whole_message (0.00ms)
PASS: test_sparkle_osx_unzip_unknown_file (0.00ms)
PASS: test_dialog_update_cursor (0.00ms)
PASS: test_round_down (0.00ms)
PASS: test_randomness (0.00ms)
PASS: test_find_related_costume (0.00ms)
PASS: test_fsm_fire_onwarn (0.00ms)
PASS: test_round_up (0.00ms)
PASS: test_track_row (0.00ms)
PASS: test_init_no_args (0.00ms)
PASS: test_transition_time (0.00ms)
PASS: test_find_unrelated_costume (15.63ms)
PASS: test_queue_new (0.00ms)
PASS: test_add_scroll_to_inventory_and_select (0.00ms)
PASS: test_remove_all_items (0.00ms)
PASS: test_fsm_no_yellow (0.00ms)
PASS: test_sparkle_lower_minor_version (0.00ms)
PASS: test_give_taco_meat (0.00ms)
PASS: test_distinct_id_len (15.63ms)
PASS: test_pick_unknown_costume (0.00ms)
PASS: test_sparkle_higher_minor_version (0.00ms)
PASS: test_remap_persistence (0.00ms)
PASS: test_fsm_change_handler (0.00ms)
PASS: test_transition_go_forward (0.00ms)
PASS: test_updater_create (0.00ms)
PASS: test_keyinuse (0.00ms)
PASS: test_updater_progress_not_started (0.00ms)
PASS: test_get_key (0.00ms)
PASS: test_keymap (0.00ms)
PASS: test_osx_get_application_path (0.00ms)
PASS: test_fsm_mixins (0.00ms)
PASS: test_sparkle_remove_directory (15.63ms)
PASS: test_sparkle_lower_versions (0.00ms)
PASS: test_load_unknown_character (0.00ms)
PASS: test_fsm_can (0.00ms)
PASS: test_fsm_cannot (0.00ms)
PASS: test_toggle_jump_high (0.00ms)
PASS: test_sparkle_higher_bug_versions (0.00ms)
PASS: test_give_money (0.00ms)
PASS: test_transition_go_backward (0.00ms)
PASS: test_queue_overwrite (0.00ms)
PASS: test_sparkle_unsupported_versions (0.00ms)
PASS: test_fsm_canel_warn_event (0.00ms)
PASS: test_osx_short_path (0.00ms)
PASS: test_fsm_accept_arguments (0.00ms)
PASS: test_string_endswith (0.00ms)
PASS: test_pick_unknown_character (0.00ms)
PASS: test_fsm_start_green (0.00ms)
PASS: test_remove_first_args (0.00ms)
PASS: test_get (0.00ms)
PASS: test_distinct_id (15.63ms)
PASS: test_string_startswith (0.00ms)
PASS: test_fsm_check_states (0.00ms)
PASS: test_distinct_id_source (0.00ms)
PASS: test_queue_events_arguments (0.00ms)
PASS: test_updater_no_thread_started (0.00ms)
PASS: test_sign_negvative (0.00ms)
PASS: test_pick_known_combination (0.00ms)
PASS: test_load_abed (0.00ms)
PASS: test_flush_row (15.63ms)
PASS: test_sparkle_lower_major_version (0.00ms)
PASS: test_random_id (0.00ms)
PASS: test_osx_no_root_path (0.00ms)
PASS: test_remove_duplicate_args (0.00ms)
PASS: test_sparkle_windows_basename (0.00ms)
PASS: test_sign_zero (0.00ms)
PASS: test_unlock_levels (0.00ms)
PASS: test_tween_hanlde_nil (0.00ms)
PASS: test_queue_validate_arguments (0.00ms)
PASS: test_sparkle_higher_major_version (0.00ms)
PASS: test_sign_positive (0.00ms)
PASS: test_sparkle_remove_path_no_exist (0.00ms)
PASS: test_use_respect (31.25ms)
PASS: test_dialog_message (0.00ms)
PASS: test_filename (0.00ms)
PASS: test_tween_negative_value (0.00ms)
PASS: test_tween_zero_value (0.00ms)
    Finished suite "main", +87 -0 E0 s0

---- Testing finished in 3.98 s, with 911 assertion(s) ----
  90 passed, 1 failed, 0 error(s), 0 skipped.

More importantly, there is a bug in lunatest that causes not reporting suite_setup's error in verbose mode - see silentbicycle/lunatest#14.

danhp added a commit that referenced this pull request Apr 27, 2014
@danhp
Merge pull request #2069 from AlfonZ42/test-status_code
561c3b8 
Return status code after testing
@danhp danhp merged commit 561c3b8 into hawkthorne:master Apr 27, 2014
@AlfonZ42 AlfonZ42 deleted the test-status_code branch April 27, 2014 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AlfonZ42 @danhp