Does it make sense to run code coverage against different language versions?

Question

In a project I am involved with, our pipeline was designed to run coveralls in parallel on several different versions of python:

Github Action yaml

matrix:
        include:
          - python-version: 2.7
            tox-env: py27
          - python-version: 3.6
            tox-env: py36
          - python-version: 3.7
            tox-env: py37,docs,readme,black
          - python-version: 3.8
            tox-env: py38
          - python-version: 3.9
            tox-env: py39
          - python-version: pypy3
            tox-env: pypy3
...
    - name: Coverage format into lcov
      if: ${{ matrix.python-version != '2.7' }}
      run: |
        coverage-lcov --output_file_path lcov.info

I can appreciate why unit and functional tests would be run against several versions of the associated technology, however the code coverage, I can't truly come up with a good reason to keep running against all the different versions of python. I am finding that running this against multiple versions doesn't reveal anything useful besides what appears to be small differences in relevant line calculations causing meaningless failures in the pipeline.

Running coverallsapp(github actions) against 3.8 and greater, results in calculations that return more "relevant" lines than the same code against 3.7 and smaller. This, I believe, is causing a pipeline failure due to a report of decreased code coverage of -0.0%, which after reviewing the actual reports in detail is a meaningless result.

Just wanted to pose this question to determine if there was any real benefit to running the code coverage against so many versions of python.

MatthewMartin · Accepted Answer · 2022-01-03 20:08:02Z

3

No. In python I rarely, if ever, see code that branches on the python version, usually other techniques are used. Either way, coverage would be the same, e.g.

try:
   # do python 2 things
except:
   # do python 3 things instead.

You'd get about the same number of lines covered either way. I think you've answered your own question in that in some later versions of python the error messages and metadata for tooling gets better for better reporting of a problem and where it happened.

answered Jan 3, 2022 at 20:08

MatthewMartin

33.3k33 gold badges117 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

John Hanley · Accepted Answer · 2022-01-04 00:03:46Z

1

Your project should state the versions of Python that you support and then test against those versions.

Advice: drop Python 2. That has been dead for more than two years.

For Python support all versions that still have support unless you require features that are only available in newer versions. That means dropping 3.6 and supporting 3.7, 3.8, 3.9, and 3.10.

However, I prefer to only support the last two released versions. That is a decision for your project.

answered Jan 4, 2022 at 0:03

John Hanley

81.8k7 gold badges115 silver badges181 bronze badges

Collectives™ on Stack Overflow

Does it make sense to run code coverage against different language versions?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related