Reduce default log verbosity for itests

[masih]

The log output for itests is so verbose that in cases GitHub Actions UI recommends downloading the log to look at it as it refuses to render it all on the page. They make seeing the cause of failures cumbersome.

• Change the default log levels for itests to print nothing.
• Add utilities to easily enable verbose logging manually for failing tests.

[rjan90]

2024-10-08:
From triage meeting:

• Ideally on failure get debug logs, else print nothing
• Dump all logs to a file as a test artifact that way if there is a failure it can still be seen but it doesn't clutter standard output in GitHub UI. (This was the recommended approach in the UI.)

[Abhay-2811]

@rjan90 @masih Can is there a way I can visualise the issue and pinpoint the places I need to make changes?

[masih]

a way I can visualise the issue and pinpoint the places I need to make changes?

Take a look at the output from any one of the itests; example. Or run go test ./itests locally.

There is already some machinery in place to quiet down the logging but the output is still verbose. More specifically, to see what assertion failed in a failing test one has to scroll through a lot of log lines to reach the assertion failure output, because a whole bunch of shut down logs continue to print after test ends.

Ideally, we want the best of both worlds:

• not be bothered by verbose logging if tests pass.
• have immediate access to it if the test fails without having to re-run it.

The solution discussed with the team and documented by Orjan is to:

1. redirect logging for each of the itests to a file.
2. attach the log file as build artefact to the corresponding build.

This way, when a test fails the immediate output is simply the assertion failure. One can then download the full logs from CI to dig further into what caused it.

places I need to make changes?

• Look at setting appropriate environment variables to redirect the logs to a file. You may need to change test kit logging to achieve this but ideally environment variables alone should be enough.
• modify the itest CI worlfow to archive log files as build artifact when the test fails.

[Abhay-2811]

Ok I'm on it, you can assign me the issue