Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Mutation testing appears to not be correctly running the test suite against mutants #1316

Open
samlev opened this issue Nov 18, 2024 · 0 comments
Labels

Comments

@samlev
Copy link

samlev commented Nov 18, 2024

What Happened

I am attempting to run mutation tests against my application, and it is frequently reporting untested methods which are definitely explicitly tested. If I attempt to make the same mutations to the original classes, my test suites will correctly catch the changes and fail, but pest simply reports the classes as untested:

   UNTESTED  src/Data/Frame.php  > Line 21: EmptyStringToNotEmpty - ID: 202027099d87aea4

    */
   readonly class Frame extends BaseData
   {
  -    final public function __construct(public string $file = '', public string $class = '', public string $function = '', public int $line = 0, public object|null $object = null)
  +    final public function __construct(public string $file = 'PEST Mutator was here!', public string $class = '', public string $function = '', public int $line = 0, public object|null $object = null)
       {
           //
       }

This should be caught with this test which explicitly tests the "defaults" of a new class:

use RecursionGuard\Data\Frame;

covers(Frame::class);

it('makes a new frame that is empty', function () {
    $frame = new Frame();

    expect($frame->file)->toBe('')
        ->and($frame->class)->toBe('')
        ->and($frame->function)->toBe('')
        ->and($frame->line)->toBe(0)
        ->and($frame->object)->toBeNull()
        ->and($frame->empty())->toBeTrue();
});

And when I make the same modification to the Frame class, I (correctly) get a test failure:

readonly class Frame extends BaseData
{
    final public function __construct(
        public string $file = 'PEST Mutator was here!',
        public string $class = '',
        public string $function = '',
        public int $line = 0,
        public object|null $object = null,
    ) {
        //
    }
    
    // ...
}

With test results:

 FAILED  Tests\Unit\Data\FrameTest > it makes a new frame that is empty                                                                                                                                                                    
  Failed asserting that two strings are identical.
  -''
  +'PEST Mutator was here!'
  

  at tests/Unit/Data/FrameTest.php:14
     10▕ it('makes a new frame that is empty', function () {
     11▕     $frame = new Frame();
     12▕ 
     13▕     expect($frame->empty())->toBeTrue()
  ➜  14▕         ->and($frame->file)->toBe('')
     15▕         ->and($frame['file'])->toBe('')
     16▕         ->and($frame->class)->toBe('')
     17▕         ->and($frame['class'])->toBe('')
     18▕         ->and($frame->function)->toBe('')

  1   tests/Unit/Data/FrameTest.php:14

How to Reproduce

I'm not entirely sure how to force a reproduction here. It seems to happen on the same files each time, but not on others. There's no common theme that I can discover - some classes are abstract, some are readonly, some methods are final, but then there's another class right next to each one with the same properties that doesn't have the issue.

Code coverage doesn't seem to be an issue, either:

 ./vendor/bin/pest --coverage --parallel
 
  Tests:    409 passed (2247 assertions)
  Duration: 0.91s
  Parallel: 32 processes

  Data/BaseData ............................................. 100.0%  
  Data/Frame ................................................ 100.0%  
  Data/RecursionContext ..................................... 100.0%  
  Data/Trace ................................................ 100.0%  
  Exception/InvalidContextException ......................... 100.0%  
  Exception/InvalidTraceException ........................... 100.0%  
  Exception/RecursionException .............................. 100.0%  
  Factory ................................................... 100.0%  
  Recursable ................................................ 100.0%  
  Recurser .................................................. 100.0%  
  Support/ArrayWritesForbidden .............................. 100.0%  
  Support/WithRecursable .................................... 100.0%  
  ────────────────────────────────────────────────────────────────── 
                                                      Total: 100.0 % 

Sample Repository

No response

Pest Version

3.5.1

PHP Version

8.3.13

Operation System

Linux

Notes

I noticed this in an earlier version of Pest, and mentioned it to @nunomaduro on twitter. At that time it was only reporting a handful of failures, but the latest version of pest seems to be giving me significantly higher numbers of failures.

@samlev samlev added the bug label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant