Description
Reports not showing hits and misses in source code
Repository
Versions
Mac OSX 10.11.6 with Firefox or Chrome
Additional Information
It does seem to work for us in other repos/codes,
Hi @apcraig
If you click the build tab and view the raw report for the job (the download link), does it show the line coverage in the uploaded report? Can you share the job number for that commit and I can check too?
Thanks @drazisil for your help. We have many tests and they are merged in the end. If I download one report, it does look like line coverage is included, I think that’s the “24” below, correct?
branch 7 taken 26%::
call 8 returned 99%::
24:250:
24:251:
call 0 returned 100%::
24:259:
call 0 returned 100%::
I’m not sure where I can find the job number. The github hash and other info is here,
https://codecov.io/gh/apcraig/Test_CICE_Icepack/commit/884953e95f7cc6ea06fe48268fa102136a46ee10
My apoligies, it looks like you don’t have build numbers.
When I look at one the reports, we we are looking to do is match the # path= lines with files in the network section at the top. It looks like the # path= lines have a different path, so you may want to look into Path Fixing to get them to match
Thanks again @drazisil. There are several things I don’t understand about how to use the codecov. We build and run the model in a different directory than where the source code lives. Then we copy the gcno,gcda/gcov files into some part of the source code directory before running the codecov uploader. Mostly this works, but in this case it seems it doesn’t. I do not understand the requirements for this tool. It’s possible that fixing-paths would help us, I have read thru almost all the codecov documentation in the last week or two, but because many of the basics are not explicitly stated, the benefit of things like fixing-paths is not entirely clear.
My sense is that most of your users are shooting for 100% coverage as part of their CI testing and that codecov addresses that need and is tightly linked into that workflow (git + CI). We are a very different kind of application. We use github, but we are a sea ice science code written mostly in Fortran, running on high performance computing platforms. We do use automated travisCI testing but only for a tiny set of testing when a PR is created. We run our full test suite on our HPC platforms with MPI and OpenMP and that requires some different strategies (like storing our code in one directory but building and running from a different directory). We also have to run many different cases, each with different input, to support our test coverage. We are also not expert python/SE tools types. So we’re struggling a bit to understand exactly what we need to do to get things to work.
What we want is to generate a code coverage report occasionally and as needed to see how we’re doing and where we can add new tests to improve coverage. It’s not going to be part of our standard model development workflow. So we are willing to deal with some work-arounds as needed. I have looked for other tools and codecov seems like just what we need, if we can get it to work reliably.
So far, we’re struggled with
- uploads timings out
- problems with stalled reporting (it seems others are seeing this as well)
- inability to show reports for source code associated with git submodules
- this issue where hits and misses are not reported in source code reports
I think we are making progress and hope that we get to the point where the tool is robust for us. I would be happy to provide feedback somewhere about our experience once everything is working to help improve documentation for other users like us. Any help we can get is appreciated. I will play with the fixing-paths next. Thanks!
The fixing-paths documentation is typical of the kinds of problems I’m running into when trying to understand what to do.
For Codecov to operate properly, all files must match the git/hg file structure.
What files are these? Are these the gcno and gcda files? So are those supposed to be dropped into the same directory structure that the source code exists in? Or are we talking about the .gcov files that start with #. When the bash uploader is invoked, where is it looking and what is it looking for? I assume it’s grabbing
- info about the git repository
- source code (it looks like the codecov reports remove the source code info from the gcov files?)
- gcov reports (does it also need to find the gcno and gcda files?)
and all of that needs to be in some specific “file structure” but it’s not clear quite what that means. Can that be clarified in the docs?
1 Like
@apcraig 100%.
I have a task on my backlog to clarify the docs.
The network is generated by doing a git-ls, so every file known to the git repo at that commit.
When matching coverage, the coverage file needs to list the filename and path in a way that we can match the results to a file in the repo. You shouldn’t need to worry about where you run the report or uploader, but if the report lists files in a relative or absolute path that doesn’t match what git sees, that’s where path fixing comes in.
Does that help some?
1 Like
Sorry, I’m still not clear. Sorry for being dense. If I look in one of my reports, I see
<<<<<< network
# path=cicecore/#p#home#apcraig#cice-consortium#test_cice_icepack#cicecore#shared#ice_constants.F90.gcov.reduced
-: 0:Source:/p/home/apcraig/cice-consortium/test_cice_icepack/cicecore/shared/ice_constants.F90
The git repo is “test_cice_icepack” and the source file is cicecore/shared/ice_constants.F90. Other than the leading directory path “/p/home/apcraig/cice-consortium”, the paths seem to be OK. Both the “source” value and the “path” value look reasonable. What am I missing?
Also, we do drop the gcov files into the cicecore directory of our repository, that is /p/home/apcraiga/cice-consortium/test_cice_icepack/cicecore/. I guess that is why the path has the leading “cicecore” string?
So, what files need to be reorganized or have fixed-paths done, just gcov files or also gcno and gcda? I believe the bash uploader is actually generating the gcov files, right? They are not generated by the model at runtime?
Hi @apcraig, you aren’t being dense, don’t worry.
The /p/home/apcraig/cice-consortium path needs to be stripped so both the full path in the coverage report, and the file in the network section (the git tree) match.
The uploader does generate gcov reports, if this is a problem you can disable it with the -X gcov flag.
Thanks @drazisil for all your help. I have run some additional tests. It’s odd because sometimes I get the source code hits and misses report and other times not. I am not changing the implementation, just changing how many tests I run and then rerunning them a few times. It seems to depend how many tests I run and which reports are uploaded. I have noticed that often not all reports are uploaded which is also a little concerning. It seems to be a bit random what is produced. If I run 5 tests, sometimes 3 reports are uploaded successfully, sometimes 4, sometimes 5. All I’m doing is rerunning the exact same thing. I think I need to take a step back and understand the bash uploader a bit better and even look into running the gcov part ourselves. Thanks again! Will keep you posted on problems and progress.
1 Like
We added the following to stop the upload failures:
bash <(curl --connect-timeout 10 --retry 10 --retry-max-time 0 https://codecov.io/bash) || (sleep 30 && bash <(curl --connect-timeout 10 --retry 10 --retry-max-time 0 https://codecov.io/bash))
Which is a bit ridiculous, but fixed about 1 upload failure per 5 travis builds. However, like you, we still frequently see reports that lack hit and miss highlighting.
I have a similar problem, and I’m puzzled by the fact that the directory view does show percentages correctly, so codecov figured the right file names at some point. Only per-line info in the file view doesn’t show up.
Now, I’ve tried many different things over the last few hours, and what appears to change things for me is the total number of builds. If I remove some builds, it starts working again. And it doesn’t matter which specific builds I remove.
EDIT: I solved my problem entirely by merging all the coverage data with coverage combine for python and gcov-tool merge for C before sending with codecov.
1 Like
I’m also not seeing lines hit/missed on our reports.
Take this file for example.
I’m unable to see any information on which lines were hit and which weren’t.
We do upload XML reports and based on some of the comments about path fixing, I confirmed that our XML report does in fact have relative paths(as seen on GH repo):
If it matters we upload several coverage reports since we run our test suite against multiple OS’es
When I go to the builds tab, the available reports to download are binary, so I can’t really confirm that what’s getting uploaded respects the expected path convention.
What else can we do?
Hi @s0undt3ch
The report isn’t actually binary, some browsers have trouble detecting it’s type, which we are investigating.
For now, the workaround is to add the extension .gz to the end of the .txt and decompress it. Hopefully that will help you view the uploaded reports and determine the issue.
Yeah, once I downloaded it on my laptop I also reached that conclusion…
My problem did seem to get fixed. Don’t know on which end though, ours when sending the reports or codedcov’s.
Anyway , all is well now.
1 Like