Code Coverage Visualization in GitLab

Continuous Integration (CI) / Continuous Development (CD) tools such as GitLab provide post-processing features for code coverage results. For example, with the help of GitLab CI/CD, you can collect the test coverage information of your favorite testing or coverage-analysis tool and visualize them inside the file diff view of your merge requests (MRs). This will allow you to see which lines are covered by tests, and which lines still require coverage before the MR is merged. GitLab requires Cobertura format as input to be able to generate code coverage analysis reports.

Cobertura report is an .xml file that contains information about the percentage of codes covered by test cases. It helps us find the parts of code that lacks the test coverage. This format was originally developed for Java, but most coverage analysis frameworks for other languages have plugins to add support for it, like:

simplecov-cobertura (Ruby)

gocover-cobertura (Golang)

Other coverage analysis frameworks support the format out of the box, for example:

Istanbul (JavaScript)

Coverage.py (Python)

Riviera-PRO generates the ACDB file which stands for "Aldec Code Coverage Data Base". So, Riviera-PRO users need to convert the ACDB report to Cobertura xml format to be able to utilize GitLab's visual coverage analysis feature. We have prepared a Python script to convert ACDB to Cobertura. This app note is prepared to provide more information on how to convert ACDB to Cobertura format.

How to convert ACDB into Cobertura XML

To run this app note, you will need to create a new project in GitLab followed by a gitlab.ci.yml file. We have covered how to create a project using GitLab and Riviera-PRO in another app note that can be found here.

GitLab uses CI/CD's artifacts reports feature to collect the coverage information. You can specify one or more coverage report to collect, including wildcard paths. This change should be applied into the .yml file. For the coverage analysis to work, you have to provide a properly formatted Cobertura XML report to artifacts:reports:cobertura. Below you can find the the .yml file content for this project.

stages:         #List of stages for jobs, and their order of execution 
   -build
build-job:      # This job runs in the build stage, which runs first. 
  stage: build
  tags:
  - ci
  artifacts:
  reports:
    cobertura: cobertura.xml
  script:
    - echo "Compiling the code..."
    - vsim -c -do sr c/runme_verilog.do
    - python3 ucdb2cobertura.py -i ucdb.xml -o cobertura.xml
    - echo "Compile complete."

The "vsim" command is called using the host machine shell environment to run "runme_verilog.do" file. You can use your own run.do file in this project. The code below shows the .do file used in this app note.

alib work
adel -all
alog -coverage sb -coverage_options count \
  src/verilog/board.v \
  src/verilog/tbscenario1.v src/verilog/tbscenario2.v \
  src/verilog/tbscenario3.v src/verilog/tbscenario4.v
asim -acdb scenario1
run -all 
acdb save -file cov/out1/out1.acdb
endsim
asim -acdb scenario2
run -all
acdb save -file cov/out2/out2.acdb
endsim
acdb merge -i cov/out1/out1.acdb -path /scenario1/UUT -i
cov/out2/out2.acdb -path /scenario2/UUT -o
cov/aggregate/aggregate.acdb
asim -acdb scenario3
run -all 
acdb save -file cov/out3/out3.acdb
endsim

acdb merge -i cov/out3/out3.acdb -path /scenario3/UUT -i 
cov/aggregate/aggregate.acdb -path / -o cov/aggregate/aggregate.acdb
asim -acdb scenario4
run -all
acdb save -file cov/out4/out4.acdb
endsim
acdb merge -i cov/out4/out4.acdb -path /scenario4/UUT -i
cov/aggregate/aggregate.acdb -path / -o cov/aggregate/aggregate.acdb
acdb2xml -i cov/aggregate/aggregate.acdb -o ucdb.xml
quit

After running the .do file, Riviera-PRO reads the project source files from the GitLab repository and does the compilation and simulation. The simulation runs four different scenarios and saves the .acdb reports. Then, the .acdb files are merged using "acdb merge" command. Then the acdb report is converted to .xml format using "acdb2xml" command that generates ucdb.xml format output that is going to be used in the python script.

Once the simulation is done, the ucdb2cobertura.py script starts running using python3 tool installed on the host machine. It's important to have the python3 path setup correctly on both user and system variables so it can be called. If for some reason this command is not recognized, you need to use the path to the exe file inside the installation folder:

C:\path-to-python3-installation\python ucdb2cobertura.py -i ucdb.xml -o cobertura.xml

How the Python converter works

The python script gets the ucdb.xml file as input, which is generated by Riviera-PRO. It parses the ucdb.xml file to generate the cobertura.xml format as an output. The image below shows a part of the ucdb2cobertura.py script. This script is free and can be requested from Aldec's support team.

Aldec has made the Python source file public. You can download it from Aldec's Github.

References: