JUnit report step used to fail for no XML input files #1431

reinhold.fuereder · 1 decade ago

We are using custom test specific notifications when there is a report available:


if (pubStep.errorMessage != null && pubStep.errorMessage.contains("report not found")) ...

However, since some time the JUnit report step seems to just log a warning instead of failing this step with the aforementioned error message ("[...] report not found [...]"):

21:55:17,934 [master>publish>publish-integrationtests>publish-report-tests-integration@LAB21:8811] INFO - Running step...
21:55:17,935 [master>publish>publish-integrationtests>publish-report-tests-integration@LAB21:8811] WARN - There is no input file need publish for 'JUnit Report', [Dir: c:\qb3\workspace\trunk\all\integrationtests\osgi.integrationtest.test\report, Pattern: **/TEST-*.xml]
21:55:17,935 [master>publish>publish-integrationtests>publish-report-tests-integration@LAB21:8811] INFO - Executing post-execute script...

While we could adapt our customizations with hopefully moderate effort, I am still unsure if not failing the step as in the past is the right thing?

replies 12
views 4479
stars 0

steveluo ADMIN · 1 decade ago

Hi,
We may enhance this so that you can define the step failure condition and step failure message yourself. A possible way is to define the step failure condition like below:


current.tests == 0 || current.errors > 0 || current.failures > 0

and define the step failure message as:


${current.tests == 0 ? "No tests found" : "Failed tests found"}

any idea?

I create an enhancement in our tracker and you may watch it to get notified:
http://track.pmease.com/browse/QB-917

steveluo ADMIN · 1 decade ago

Hi,
Please upgrade to QB-3.1.42, and define the failure condition and failure message like what I mentioned above.

reinhold.fuereder · 1 decade ago

Thanks Steve, we will have a look at it.

While this nonetheless means adapting at least 7 (different junit report steps) times 6 (different versions) steps = 42 steps (that 42 is the result comes by no surprise?), however, the bigger problem in this case is/was that this modification (step failing => step successful + logging warning) was silent and took us by surprise -- or have we missed a release note entry for it?

steveluo ADMIN · 1 decade ago

Hi,
I am not very sure I understand your questions, sorry. The modification in QB-3.1.42 is when there is no test report found, the publish JUnit step can use current.tests == 0 as a failure condition to fail the JUnit step and the failure condition is customized by user, so you need not detect the error messages from the JUnit publish step.

We did not change step failing to step successful + logging warning. The publish report steps use step failure condition to determine whether the step is failed or not as always.

Hope I answered your questions.

steveluo ADMIN · 1 decade ago

Another question about 42 steps defined, I do not know how you defined your publish step, you may consider use repeat parameters to reduce the steps. For example, you can define two parameters like:
PLATFORM: Linux, Leopard, WinXP, Win7
JDK: 1.4, 1.5, 1.6

and define the files to process and reportset as:
Files to process: targets/junit/${params.get("PLATFORM")}/${params.get("JDK")}/*.xml
Report set: ${params.get("PLATFORM")} - ${params.get("JDK")}

Now, this step can run 4 (platforms) times 3 (jdk versions) = 12 instead of defining 12 steps.

reinhold.fuereder · 1 decade ago

Hi Steve,

I must apologize for my wrong allegation, because the actual change (and thus our fault) was that we guarded the junit publish step with this scripted execution condition some time ago:


${(new java.io.File(configuration.workspaceDir, "report")).exists()}

Thanks again for your advice, we extended the check to skip our custom test specific notifications (so it is only one change per version):


if ((pubStep.errorMessage != null && pubStep.errorMessage.contains("report not found")) || pubStep.tests <= 0) { ...

Repeat parameter usage (which is IMHO a great QB feature) is therefore no more necessary (one change) and would not be appropriate anyhow: by 6 different versions I meant product versions configured in QB; by 7 different junit report steps I meant 7 different test types (plain JUnit vs. client vs. integration vs. ...) that are independent of each other.

reinhold.fuereder · 1 decade ago

Hi Steve,

unfortunately there seems to be a big problem: when we access the JUnitPublisher step in the notify-authors step (at the very end of the build), the number of tests seems to be 0 (at least when there are failing tests); note that the variable "pubSteb" is of type "JUnitPublisher" in the following "debug log":

14:41:36,183 [master>notify@lab14:8811] INFO - [notify-authors] Evaluating Execute Condition
14:41:36,184 [master>notify@lab14:8811] INFO - [notify-authors] Found publish report step
14:41:36,185 [master>notify@lab14:8811] INFO - pubStep.errorMessage: Found failed tests in JUnit report.
14:41:36,185 [master>notify@lab14:8811] INFO - pubStep.tests: 0
14:41:36,185 [master>notify@lab14:8811] INFO - [notify-authors] Skipping executing notify-authors

steveluo ADMIN · 1 decade ago

Hi,
You cannot use pubStep.tests in your notification step. The variables of tests, failures and errors are only visible in the JUnit publish step which may run on the node different with notification step. We will enhance this later so that other steps can also access those variables. But in your case, I think you can define your step like what I mentioned before, that is, in JUnit step, define the failure condition as:


current.tests == 0 || current.errors > 0 || current.failures > 0

define the failure message as:


current.tests == 0 ? "report not found" : "failed tests found"

and keep your notification logic like before:


if (pubStep.errorMessage != null && pubStep.errorMessage.contains("report not found")) { ...

If you do want to access the variables of JUnit step in notification step, let me know and we will add this to next patch release.

#10

reinhold.fuereder · 1 decade ago

Hi Steve,

In our case it is guaranteed that the notification step is run on the same node as the JUnit publish step, but seemingly that does not help.

I was wondering where one can see which step variables are available outside the step itself?

Accessing the variables outside the JUnit publish step would allow for a cleaner and less duplicate (DRY) implementation (cf. my recent postings) and would be therefore very appreciated. When might this patch be available?

Thanks,
Reinhold

#11

steveluo ADMIN · 1 decade ago

Hi Reinhold,

You are right. To expose the step variables to other steps will make the implementations more cleaner. The patch release will be released in days. You may watch below issue:
http://track.pmease.com/browse/QB-924

#12

reinhold.fuereder · 1 decade ago

Thanks Steve.

I dare to repeat my question that should be completely general and not specific to the report publish steps:

I was wondering where one can see which step variables are available outside the step itself?

#13

steveluo ADMIN · 1 decade ago

Hi Reinhold,

To access variables outside of the report publish step are not open-api before QB-3.1.42 (include 3.1.42). Because you can not find any help about them when you view the popup help of a scriptable field (we list all of the scriptable APIs in the popup help). And we will add all of the stats variables (e.g. failures/errors/tests for JUnit step) for the report publish step to next release. That means, you can see the variables help and access the variables outside safely in next version.