Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fine-tune coverage execution #77

Open
1 of 3 tasks
Zearin opened this issue Feb 12, 2013 · 3 comments
Open
1 of 3 tasks

Fine-tune coverage execution #77

Zearin opened this issue Feb 12, 2013 · 3 comments

Comments

@Zearin
Copy link
Collaborator

Zearin commented Feb 12, 2013

Problem

Part I

The documentation for coverage states that GEvent can confuse coverage’s results:

gevent, which is based on greenlet, and is similar to eventlet. All of these manipulate the C stack, and therefore confuse coverage.py. Issue 149 has some pointers to more information.

Part II

Should we be using --timid?

If your coverage results seem to be overlooking code that you know has been executed, try running coverage again with the --timid flag. This uses a simpler but slower trace method.

Source: http://nedbatchelder.com/code/coverage/cmd.html#cmd-execution

Part III

Should we be using --parallel-mode?

If you are measuring coverage in a multi-process program, or across a number of machines, you’ll want the --parallel-mode switch to keep the data separate during measurement. See Combining data files below.

Source: http://nedbatchelder.com/code/coverage/cmd.html#cmd-execution


Summary

  • Investigate coverage + GEvent
  • Investigate --timid
  • Investigate --parallel-mode
@heynemann
Copy link
Owner

Have you tried any of those?

It seems to me that it would be useful to have tests around these issues before fixing them. Can you write the tests that prove the issue?

Thanks!!!

@Zearin
Copy link
Collaborator Author

Zearin commented Feb 16, 2013

Makes sense. ☺ Will try to test this shortly.

@Zearin
Copy link
Collaborator Author

Zearin commented Feb 16, 2013

coverage + GEvent

Confirmed.

On coverage’s gevent issue, Ned Batchelder said:

Here's a fix, though it sacrifices other behavior: https://github.com/newbrough/coverage

Here is coverage as pyvows currently stands:

=============
 Code Coverage
 =============

 ✗ pyvows/__init__                    0.0%                                                     2, 36, 37, and 4 more
 ✗ pyvows/__main__                    0.0%                                                     3, 5, 7, and 2 more
 ✗ pyvows/assertions/__init__         0.0%                                                     3, 25, 26, and 11 more
 ✗ pyvows/color                       0.0%                                                     2, 16, 17, and 35 more
 ✗ pyvows/reporting/__init__          0.0%                                                     11, 12, 13, and 4 more
 ✗ pyvows/reporting/profile           0.0%                                                     2, 11, 13, and 31 more
 ✗ pyvows/version                     0.0%                                                     2, 13, 16, and 1 more
 ✗ pyvows/reporting/common            •• 4.3%                                                  2, 12, 14, and 63 more
 ✗ pyvows/cli                         •• 5.5%                                                  3, 14, 16, and 101 more
 ✗ pyvows/reporting/coverage          •••••• 13.2%                                             2, 12, 14, and 63 more
 ✗ pyvows/assertions/types/numeric    •••••••••• 20.0%                                         3, 14, 16, and 1 more
 ✗ pyvows/assertions/emptiness        •••••••••••• 25.0%                                       3, 14, 17
 ✗ pyvows/assertions/equality         •••••••••••• 25.0%                                       3, 14, 17
 ✗ pyvows/assertions/inclusion        •••••••••••• 25.0%                                       3, 14, 17
 ✗ pyvows/reporting/test              ••••••••••••••• 30.6%                                    2, 11, 13, and 47 more
 ✗ pyvows/errors                      ••••••••••••••••• 35.0%                                  2, 13, 18, and 10 more
 ✗ pyvows/result                      •••••••••••••••••••• 41.5%                               2, 15, 23, and 21 more
 ✗ pyvows/assertions/types/function   •••••••••••••••••••••• 44.4%                             3, 14, 16, and 2 more
 ✗ pyvows/assertions/types/regexp     •••••••••••••••••••••• 44.4%                             3, 14, 16, and 2 more
 ✗ pyvows/assertions/types/errors     •••••••••••••••••••••• 45.5%                             3, 14, 16, and 3 more
 ✗ pyvows/async_topic                 •••••••••••••••••••••••• 48.5%                           2, 15, 18, and 14 more
 ✗ pyvows/reporting/xunit             •••••••••••••••••••••••• 49.1%                           2, 12, 13, and 26 more
 ✗ pyvows/assertions/length           ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows/assertions/types/boolean    ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows/assertions/types/classes    ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows/assertions/types/nullable   ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows/core                        ••••••••••••••••••••••••• 51.1%                          2, 12, 13, and 62 more
 ✗ pyvows/assertions/types/file       •••••••••••••••••••••••••••••••• 65.0%                   3, 18, 19, and 4 more
 ✗ pyvows/assertions/like             •••••••••••••••••••••••••••••••••• 68.6%                 3, 19, 21, and 13 more
 ✗ pyvows/runner                      ••••••••••••••••••••••••••••••••••• 70.6%                2, 15, 16, and 50 more

 ✗ OVERALL                            •••••••••••••••••• 36.7%


 ============
 Vows Results
 ============

  ✓ OK » 497 honored • 0 broken (0.959065s)


Next, I uninstalled coverage, then re-installed it directly from newbrough/coverage.

The difference is not huge, but it’s there:

=============
 Code Coverage
 =============

 ✗ pyvows.__init__                    0.0%                                                     2, 36, 37, and 4 more
 ✗ pyvows.__main__                    0.0%                                                     3, 5, 7, and 2 more
 ✗ pyvows.color                       0.0%                                                     2, 16, 17, and 35 more
 ✗ pyvows.version                     0.0%                                                     2, 13, 16, and 1 more
 ✗ pyvows.assertions.__init__         0.0%                                                     3, 25, 26, and 11 more
 ✗ pyvows.reporting.__init__          0.0%                                                     11, 12, 13, and 4 more
 ✗ pyvows.reporting.profile           0.0%                                                     2, 11, 13, and 31 more
 ✗ pyvows.reporting.common            •• 4.3%                                                  2, 12, 14, and 63 more
 ✗ pyvows.cli                         ••• 7.3%                                                 3, 14, 16, and 99 more
 ✗ pyvows.reporting.coverage          •••••• 13.2%                                             2, 12, 14, and 63 more
 ✗ pyvows.assertions.types.numeric    •••••••••• 20.0%                                         3, 14, 16, and 1 more
 ✗ pyvows.assertions.emptiness        •••••••••••• 25.0%                                       3, 14, 17
 ✗ pyvows.assertions.equality         •••••••••••• 25.0%                                       3, 14, 17
 ✗ pyvows.assertions.inclusion        •••••••••••• 25.0%                                       3, 14, 17
 ✗ pyvows.reporting.test              ••••••••••••••• 30.6%                                    2, 11, 13, and 47 more
 ✗ pyvows.errors                      ••••••••••••••••• 35.0%                                  2, 13, 18, and 10 more
 ✗ pyvows.result                      •••••••••••••••••••• 41.5%                               2, 15, 23, and 21 more
 ✗ pyvows.assertions.types.function   •••••••••••••••••••••• 44.4%                             3, 14, 16, and 2 more
 ✗ pyvows.assertions.types.regexp     •••••••••••••••••••••• 44.4%                             3, 14, 16, and 2 more
 ✗ pyvows.assertions.types.errors     •••••••••••••••••••••• 45.5%                             3, 14, 16, and 3 more
 ✗ pyvows.async_topic                 •••••••••••••••••••••••• 48.5%                           2, 15, 18, and 14 more
 ✗ pyvows.reporting.xunit             •••••••••••••••••••••••• 49.1%                           2, 12, 13, and 26 more
 ✗ pyvows.assertions.length           ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows.assertions.types.boolean    ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows.assertions.types.classes    ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows.assertions.types.nullable   ••••••••••••••••••••••••• 50.0%                          3, 14, 17, and 1 more
 ✗ pyvows.core                        ••••••••••••••••••••••••• 51.1%                          2, 12, 13, and 62 more
 ✗ pyvows.assertions.types.file       •••••••••••••••••••••••••••••••• 65.0%                   3, 18, 19, and 4 more
 ✗ pyvows.assertions.like             •••••••••••••••••••••••••••••••••• 68.6%                 3, 19, 21, and 13 more
 ✗ pyvows.runner                      •••••••••••••••••••••••••••••••••••• 72.2%               2, 15, 16, and 47 more

 ✗ OVERALL                            •••••••••••••••••• 37.2%


 ============
 Vows Results
 ============

  ✓ OK » 497 honored • 0 broken (0.927781s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants