Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return status code after testing #2069

Merged
merged 3 commits into from
Apr 27, 2014
Merged

Conversation

AlfonZ42
Copy link
Contributor

Currently, even if some tests fail, status code returned is 0, thus CI doesn't find out that there was failure and build doesn't fail. This changes it.

Current behavior:

  • Non-Windows, tests pass - process finishes, status code is zero
  • Non-Windows, tests fail - process finishes, status code is zero
  • Windows, tests pass - process keeps running
  • Windows, tests fail - process keeps running

Keeping the process running under Windows is desired behavior - see 9ad7e38.

New behavior:

  • Non-Windows, tests pass - process finishes, status code is zero
  • Non-Windows, tests fail - process finishes, status code is non-zero (CI reports failed build)
  • Windows, tests pass - process keeps running
  • Windows, tests fail - process keeps running

I have disabled verbose mode for now, even though 9ad7e38 intended to have it enabled.
Due to bug (which this change fixes) it wasn't taken into account. Enabling it would report passed tests as well, making failed tests less visible, see below.

Non-verbose output (with fake failed test added):

-- Starting suite "test/test_maps", 4 test(s)
  .F..
-- Starting suite "main", 87 test(s)
  ...................................................................
  ....................
---- Testing finished in 4.00 s, with 911 assertion(s) ----
  90 passed, 1 failed, 0 error(s), 0 skipped.
FAIL: test/test_maps.test_fakefail (0.00ms): (Failed) - Fake failed test. (286)

Verbose output:

Starting tests, 16 suite(s)
-- Starting suite "test/test_maps", 4 test(s)
PASS: test_breakable_blocks (0.00ms)
FAIL: test_fakefail (0.00ms): (Failed) - Fake failed test. (286)
PASS: test_music (140.63ms)
PASS: test_doors (15.63ms)
    Finished suite "test/test_maps", +3 -1 E0 s0
-- Starting suite "main", 87 test(s)
PASS: test_fsm_cancel_onbefore_event (0.00ms)
PASS: test_load_current (46.88ms)
PASS: test_tween_handle_anything (0.00ms)
PASS: test_get_action (0.00ms)
PASS: test_queue_push_events (0.00ms)
PASS: test_split_string (0.00ms)
PASS: test_sparkle_same_versions (0.00ms)
PASS: test_fsm_state_transition (0.00ms)
PASS: test_give_gcc_key (0.00ms)
PASS: test_transition_new (0.00ms)
PASS: test_dialog_whole_message (0.00ms)
PASS: test_sparkle_osx_unzip_unknown_file (0.00ms)
PASS: test_dialog_update_cursor (0.00ms)
PASS: test_round_down (0.00ms)
PASS: test_randomness (0.00ms)
PASS: test_find_related_costume (0.00ms)
PASS: test_fsm_fire_onwarn (0.00ms)
PASS: test_round_up (0.00ms)
PASS: test_track_row (0.00ms)
PASS: test_init_no_args (0.00ms)
PASS: test_transition_time (0.00ms)
PASS: test_find_unrelated_costume (15.63ms)
PASS: test_queue_new (0.00ms)
PASS: test_add_scroll_to_inventory_and_select (0.00ms)
PASS: test_remove_all_items (0.00ms)
PASS: test_fsm_no_yellow (0.00ms)
PASS: test_sparkle_lower_minor_version (0.00ms)
PASS: test_give_taco_meat (0.00ms)
PASS: test_distinct_id_len (15.63ms)
PASS: test_pick_unknown_costume (0.00ms)
PASS: test_sparkle_higher_minor_version (0.00ms)
PASS: test_remap_persistence (0.00ms)
PASS: test_fsm_change_handler (0.00ms)
PASS: test_transition_go_forward (0.00ms)
PASS: test_updater_create (0.00ms)
PASS: test_keyinuse (0.00ms)
PASS: test_updater_progress_not_started (0.00ms)
PASS: test_get_key (0.00ms)
PASS: test_keymap (0.00ms)
PASS: test_osx_get_application_path (0.00ms)
PASS: test_fsm_mixins (0.00ms)
PASS: test_sparkle_remove_directory (15.63ms)
PASS: test_sparkle_lower_versions (0.00ms)
PASS: test_load_unknown_character (0.00ms)
PASS: test_fsm_can (0.00ms)
PASS: test_fsm_cannot (0.00ms)
PASS: test_toggle_jump_high (0.00ms)
PASS: test_sparkle_higher_bug_versions (0.00ms)
PASS: test_give_money (0.00ms)
PASS: test_transition_go_backward (0.00ms)
PASS: test_queue_overwrite (0.00ms)
PASS: test_sparkle_unsupported_versions (0.00ms)
PASS: test_fsm_canel_warn_event (0.00ms)
PASS: test_osx_short_path (0.00ms)
PASS: test_fsm_accept_arguments (0.00ms)
PASS: test_string_endswith (0.00ms)
PASS: test_pick_unknown_character (0.00ms)
PASS: test_fsm_start_green (0.00ms)
PASS: test_remove_first_args (0.00ms)
PASS: test_get (0.00ms)
PASS: test_distinct_id (15.63ms)
PASS: test_string_startswith (0.00ms)
PASS: test_fsm_check_states (0.00ms)
PASS: test_distinct_id_source (0.00ms)
PASS: test_queue_events_arguments (0.00ms)
PASS: test_updater_no_thread_started (0.00ms)
PASS: test_sign_negvative (0.00ms)
PASS: test_pick_known_combination (0.00ms)
PASS: test_load_abed (0.00ms)
PASS: test_flush_row (15.63ms)
PASS: test_sparkle_lower_major_version (0.00ms)
PASS: test_random_id (0.00ms)
PASS: test_osx_no_root_path (0.00ms)
PASS: test_remove_duplicate_args (0.00ms)
PASS: test_sparkle_windows_basename (0.00ms)
PASS: test_sign_zero (0.00ms)
PASS: test_unlock_levels (0.00ms)
PASS: test_tween_hanlde_nil (0.00ms)
PASS: test_queue_validate_arguments (0.00ms)
PASS: test_sparkle_higher_major_version (0.00ms)
PASS: test_sign_positive (0.00ms)
PASS: test_sparkle_remove_path_no_exist (0.00ms)
PASS: test_use_respect (31.25ms)
PASS: test_dialog_message (0.00ms)
PASS: test_filename (0.00ms)
PASS: test_tween_negative_value (0.00ms)
PASS: test_tween_zero_value (0.00ms)
    Finished suite "main", +87 -0 E0 s0

---- Testing finished in 3.98 s, with 911 assertion(s) ----
  90 passed, 1 failed, 0 error(s), 0 skipped.

More importantly, there is a bug in lunatest that causes not reporting suite_setup's error in verbose mode - see silentbicycle/lunatest#14.

danhp added a commit that referenced this pull request Apr 27, 2014
Return status code after testing
@danhp danhp merged commit 561c3b8 into hawkthorne:master Apr 27, 2014
@AlfonZ42 AlfonZ42 deleted the test-status_code branch April 27, 2014 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants