Tiobe Integration #420

kadinsayani · 2024-10-09T20:23:07Z

This PR adds code coverage for unit tests and snap system tests.

tomponline

Let me know when you need secrets.TICS_AUTH_TOKEN setup

kadinsayani · 2024-10-10T21:34:57Z

For the system tests using the MicroCloud snap, it looks like the only requirements to generate the coverage files are:

GOCOVERDIR needs to be set in the snap confinement space - I'm creating a directory on the runner at /var/snap/microcloud/common/data/coverage; and
The side loaded binary should be compiled with the -cover flag.

@tomponline, once I've got it working here, we can utilize a similar approach for LXD.

This PR creates a GOCOVERDIR in the snap confinement space when using debug binaries. Note: required for canonical/microcloud#420.

kadinsayani · 2024-10-16T19:31:04Z

There appears to be a problem with the MicroCloud CI tests because of a regression in upstream Ceph versioning. As a result, I haven’t tested this yet.

test/main.sh

kadinsayani · 2024-10-18T19:44:10Z

Should be good to go, just waiting on https://github.com/kadinsayani/microcloud/actions/runs/11413516898/job/31763088034

kadinsayani · 2024-10-19T04:08:27Z

Good to go on this, here's an example successful run on my fork: https://github.com/kadinsayani/microcloud/actions/runs/11413516898/job/31763088034.

kadinsayani · 2024-10-19T19:20:00Z

We recently started using the go1.23.1 toolchain for MicroCloud. To use StaticCheck with Go 1.23, it needs to be built with Go 1.23. Hence why the Tics job throws errors like this one:

2024-10-19 04:00:23.592: [INFO 3012] Analyzing 30/58 /home/runner/work/microcloud/microcloud/cmd/microcloud/session.go
start analysis: 2024-10-19 04:00:23.592
  Initialization
  Initialization time: 0.001 sec
Error: [ERROR 5073] The following error(s) occurred while 'StaticCheck' was running: '/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.1.linux-amd64/src/maps/iter.go:51:20: cannot range over seq (variable of type iter.Seq2[K, V])'.

These errors are what caused the downtick in TQI scores across various Tiobe quality metrics. The analyzer runs are erroring out, so data isn't being sent to Tics for the coding standards runs.

For the security checks, I'm seeing the following error on every run:

[ERROR 5255] Command 'cov-build' failed with exit status 256, error: -fno-stack-protector ./cgo.go See log '/tmp/tics/11413516898_1_tics_tics-github-action/ticstmpdir/Coverity/cov-build-console.log'.

Tics uses Coverity for security scanning, here's something I found from their docs:

By default, cov-build returns the exit code from the native build.

Exit code 256 typically indicates that the system can't locate the binary to run the command, so I think this is also a dependency issue.

I propose that we bump the Go version in the Tics job to ensure we capture the right dependencies for Tics at the Install dependencies step. Testing now: https://github.com/kadinsayani/microcloud/actions/runs/11420168463.

tomponline · 2024-10-21T07:19:40Z

Yeah we should switch to go 1.23

kadinsayani · 2024-10-21T14:34:58Z

Good to go - all of the Tiobe runs are operational and the quality metrics are accurate.

https://github.com/kadinsayani/microcloud/actions/runs/11432414984/job/31830683250

roosterfish

Thanks for this! A few comments from my side, mostly for clarification.

roosterfish · 2024-10-22T10:31:30Z

test/includes/microcloud.sh

@@ -271,8 +271,8 @@ set_debug_binaries() {
    lxc exec "${name}" -- rm -f /var/snap/microcloud/common/microcloudd.debug
    lxc exec "${name}" -- rm -f /var/snap/microcloud/common/microcloud.debug

-    lxc file push --quiet "${MICROCLOUDD_DEBUG_PATH}" "${name}"/var/snap/microcloud/common/microcloudd.debug
-    lxc file push --quiet "${MICROCLOUD_DEBUG_PATH}" "${name}"/var/snap/microcloud/common/microcloud.debug
+    lxc file push --quiet "${MICROCLOUDD_DEBUG_PATH}" "${name}"/var/snap/microcloud/common/microcloudd.debug --mode 0755


Out of interest what is the reason for changing the permission for the debug binaries?

lxc file push sets the file mode to 0644 by default - we want to make sure the binaries are executable and maintain the file mode set by the Go compiler. We also check for the presence of these executables in commands/daemon.start and commands/microcloud in the MicroCloud snap. When they are present and executable, the GOCOVERDIR is created. Here's the relevant PR: canonical/microcloud-pkg-snap#15.

roosterfish · 2024-10-22T10:33:44Z

staticcheck.conf

@@ -0,0 +1,3 @@
+# Checks being ignored:


Why is this required?

We wanted to see if adding a staticcheck.conf with this rule would silence the Coding standards violations, but it looks like Tics isn't picking this up so our Coding standards TQI score remains the same. See https://github.com/kadinsayani/microcloud/actions/runs/11446739960/job/31905615699.

We can still keep this staticcheck rule since the violation doesn't apply to us.

roosterfish · 2024-10-22T10:35:49Z

test/main.sh

@@ -192,6 +192,8 @@ run_test() {
 	${TEST_CURRENT}
 	END_TIME="$(date +%s)"

+	collect_go_cover_files


Why is it not sufficient to do the collection here but also in reset_systems and restore_systems?

A few of the test suites use reset_systems during test cases. reset_systems may call restore_systems when SNAPSHOT_RESTORE is set to 1.

.github/workflows/tests.yml

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>

kadinsayani · 2024-10-24T06:31:33Z

I've rebased and resolved a naming issue with the coverage file directories. Test run: https://github.com/kadinsayani/microcloud/actions/runs/11493847667/job/31990262151.

kadinsayani · 2024-10-25T03:10:06Z

I've rebased and resolved a naming issue with the coverage file directories. Test run: https://github.com/kadinsayani/microcloud/actions/runs/11493847667/job/31990262151.

Looks like there's still an issue with the naming. I'll try and fix this some time today. Just need a unique identifier to append so there aren't naming conflicts.

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>

kadinsayani · 2024-10-25T11:28:38Z

Successful run: https://github.com/kadinsayani/microcloud/actions/runs/11516440151

kadinsayani requested review from tomponline and masnax October 9, 2024 20:23

kadinsayani force-pushed the tiobe-integration branch 2 times, most recently from 3c9ee1a to b78a357 Compare October 9, 2024 20:51

tomponline previously approved these changes Oct 10, 2024

View reviewed changes

kadinsayani changed the title ~~Unit Test Tiobe Integration~~ Tiobe Integration Oct 10, 2024

kadinsayani dismissed tomponline’s stale review via 9fec240 October 14, 2024 18:38

kadinsayani force-pushed the tiobe-integration branch 2 times, most recently from 5f8676c to 9a0ecdf Compare October 14, 2024 18:43

kadinsayani marked this pull request as draft October 14, 2024 18:44

kadinsayani mentioned this pull request Oct 14, 2024

snap: Add GOCOVERDIR to commands canonical/microcloud-pkg-snap#15

Merged

tomponline added a commit to canonical/microcloud-pkg-snap that referenced this pull request Oct 16, 2024

snap: Add GOCOVERDIR to commands (#15)

1289a99

This PR creates a GOCOVERDIR in the snap confinement space when using debug binaries. Note: required for canonical/microcloud#420.

kadinsayani force-pushed the tiobe-integration branch from 9a0ecdf to 606db39 Compare October 16, 2024 19:27

kadinsayani marked this pull request as ready for review October 16, 2024 19:28

kadinsayani force-pushed the tiobe-integration branch 2 times, most recently from 608198c to 519061b Compare October 18, 2024 17:31

github-advanced-security bot found potential problems Oct 18, 2024

View reviewed changes

test/main.sh Fixed Show fixed Hide fixed

kadinsayani force-pushed the tiobe-integration branch 2 times, most recently from ced5879 to d038f14 Compare October 18, 2024 19:22

kadinsayani force-pushed the tiobe-integration branch from d038f14 to 25f8594 Compare October 18, 2024 21:35

kadinsayani requested a review from tomponline October 19, 2024 04:08

tomponline previously approved these changes Oct 19, 2024

View reviewed changes

tomponline requested a review from roosterfish October 19, 2024 05:45

kadinsayani dismissed tomponline’s stale review via d9073dd October 21, 2024 14:33

kadinsayani force-pushed the tiobe-integration branch from 25f8594 to d9073dd Compare October 21, 2024 14:33

roosterfish requested changes Oct 22, 2024

View reviewed changes

kadinsayani force-pushed the tiobe-integration branch 5 times, most recently from 7b37e0f to 61a0ec4 Compare October 23, 2024 14:33

Makefile: Add -cover go build step for coverage

758416a

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>

kadinsayani force-pushed the tiobe-integration branch from 61a0ec4 to 9270c6d Compare October 24, 2024 06:23

kadinsayani added 2 commits October 25, 2024 02:35

github: Add tiobe tics job for system and unit test coverage

1578a8e

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>

staticcheck: Add configuration file to ignore check ST1005

ed6d74c

Signed-off-by: Kadin Sayani <kadin.sayani@canonical.com>

kadinsayani force-pushed the tiobe-integration branch from 9270c6d to ed6d74c Compare October 25, 2024 11:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tiobe Integration #420

Tiobe Integration #420

kadinsayani commented Oct 9, 2024 •

edited

Loading

tomponline left a comment

kadinsayani commented Oct 10, 2024 •

edited

Loading

kadinsayani commented Oct 16, 2024 •

edited

Loading

kadinsayani commented Oct 18, 2024 •

edited

Loading

kadinsayani commented Oct 19, 2024

kadinsayani commented Oct 19, 2024 •

edited

Loading

tomponline commented Oct 21, 2024

kadinsayani commented Oct 21, 2024 •

edited

Loading

roosterfish left a comment

roosterfish Oct 22, 2024

kadinsayani Oct 22, 2024

roosterfish Oct 22, 2024

kadinsayani Oct 22, 2024

kadinsayani Oct 22, 2024 •

edited

Loading

roosterfish Oct 22, 2024

kadinsayani Oct 22, 2024

kadinsayani commented Oct 24, 2024

kadinsayani commented Oct 25, 2024

kadinsayani commented Oct 25, 2024

Tiobe Integration #420

Are you sure you want to change the base?

Tiobe Integration #420

Conversation

kadinsayani commented Oct 9, 2024 • edited Loading

tomponline left a comment

Choose a reason for hiding this comment

kadinsayani commented Oct 10, 2024 • edited Loading

kadinsayani commented Oct 16, 2024 • edited Loading

kadinsayani commented Oct 18, 2024 • edited Loading

kadinsayani commented Oct 19, 2024

kadinsayani commented Oct 19, 2024 • edited Loading

tomponline commented Oct 21, 2024

kadinsayani commented Oct 21, 2024 • edited Loading

roosterfish left a comment

Choose a reason for hiding this comment

roosterfish Oct 22, 2024

Choose a reason for hiding this comment

kadinsayani Oct 22, 2024

Choose a reason for hiding this comment

roosterfish Oct 22, 2024

Choose a reason for hiding this comment

kadinsayani Oct 22, 2024

Choose a reason for hiding this comment

kadinsayani Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

roosterfish Oct 22, 2024

Choose a reason for hiding this comment

kadinsayani Oct 22, 2024

Choose a reason for hiding this comment

kadinsayani commented Oct 24, 2024

kadinsayani commented Oct 25, 2024

kadinsayani commented Oct 25, 2024

kadinsayani commented Oct 9, 2024 •

edited

Loading

kadinsayani commented Oct 10, 2024 •

edited

Loading

kadinsayani commented Oct 16, 2024 •

edited

Loading

kadinsayani commented Oct 18, 2024 •

edited

Loading

kadinsayani commented Oct 19, 2024 •

edited

Loading

kadinsayani commented Oct 21, 2024 •

edited

Loading

kadinsayani Oct 22, 2024 •

edited

Loading