I am testing the test muting feature before making general use of it.
Is the result in step 4 the correct behaviour? I had expected TeamCity to forget about the previous "muting" on those tests once those tests had passed, and fail the build.
Also, the muted tests in Step 4 do not have the "muted" icon.
I am running TeamCity 8.0.3 Enterprise. Screenshot of Step 4 results attached.
I reproduced your scenario in the same TC version and it works as intended - recreated tests (step 4) failed with no mute flag set on. What tests do you use? Do you have some history builds? Can you please also check server and agent logs for some errors?
Thank-you for your response -- it confirms what I thought was the expected behaviour. I had been wondering if what I was seeing was a mechanism for dealing with transient failures, but no.
The server and agent logs show no errors. And no history builds involved.
When I re-ran the build this morning - the same content; absolutely no changes - it behaved as it should: it failed with 3 test failures. Please see the attached screen shot. The circled build (build 49) is the build referenced in Step 4 in my previous post.
When I came in this morning, I reran the build (no changes to content or configuration), and the result was as it should be: the build failed with 3 failed tests (build 50).
I then experimented over the day as follows, with builds 59, 67, 68 and onward having incorrect results:
build 50: after the build completed, I muted the 3 tests.
build 51 & 52: no changes; pass with 3 muted tests.
build 53: changed the tests to pass; the build passed, no muted tests.
build 54: no changes; the build passed, no muted tests.
build 55: changed the tests to fail; the build failed with 3 failed tests.
build 56: no changes; the build failed with 3 failed tests.
-- after the build, I muted the 3 tests.
build 57: no changes; pass with 3 muted tests.
build 58: changed the tests to pass; the build passed, no muted tests.
build 59: changed the tests to fail; the build passed with 3 muted tests!!!!
build 60: no changes; the build now fails (but it was a different agent...).
build 61: no changes; reran the build on the same agent as build 59: fail (so not the agent).
build 62: changed the tests to pass; build passes, no muted tests.
build 63: changed the tests to fail; build fails with 3 failed tests.
build 64 & 65: oops - At this point, I realized I forgot to mute the tests on build 61.
-- at this point, I muted the 3 failed tests on build 63.
build 66: changed the tests to pass; build passes, no muted tests.
build 67: changed the tests to fail; build passes with 3 muted tests!!!!
build 68 & onward: no changes; failed with 2 failed tests but 1 muted!
During the day, I noticed "mute has been automatically removed" entries in the server log. By the end of the day, there were these 3:
[2013-10-11 12:05:02,938] INFO - jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 3 tests in Dev :: RBird
[2013-10-11 14:50:20,754] INFO - jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 3 tests in Dev :: RBird
[2013-10-11 19:19:43,116] INFO - jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 2 tests in Dev :: RBird
The first message occurs between builds 54 & 55.
The second between builds 59 & 60.
The last after build 67 and during build 68 through 70.
My hypothesis: after a build successfully runs failed tests, there is an approximately 30 minute wait time before TeamCity "clears" the mute for the project. Except: why did it clear only 2 of the 3 tests in the last go-round?
Now that I have some notion of what is going on, this is just a minor issue that I can work around.
You asked about the tests I was running: JUnit and CUnit. The build has 5 steps, 2 of which are disabled. The JUnit and CUnit tests are run using ant in build step 4. I have a custom ant task that parses the output from the CUnit tests as they occur and sends services messages to TeamCity. I've rigged the ant script so that it will not fail if there are JUnit or CUnit test failures. Of the 3 tests that I have been using in the above, one is a JUnit test and the other two are one CUnit test that gets run in two different situations.
So, I reproduced the problem and checked the Muted Problems tab - see attached pic "AllTestsPassed2Muted". I included the clock to remind myself that this was the state after a build had been run where all tests passed successfully after having been muted before running the build. None of the muted tests are "active": clicking on them does not expand them to show more information. And then this tab becomes empty after 30 minutes or so.
I did check the Muted Problems tab before running the successful build (sorry, forgot to take a pic), and the muted tests appeared there and could be expanded to show their failure details and the build they failed in, which was the build that they did fail in. No evidence that I could see of any other build configuration being involved.
I will follow your suggestion and upgrade to 8.0.4 this weekend.
I upgraded to 8.0.4, and the issue is still there.
To summarize, the problem manifests by following this sequence of steps:
1. run a build with 3 tests that I rigged to fail. (Build #0.25.l.22241.eval02.84 in attached pic MutedTests-allBuilds.png).
2. after the build, mute the 3 tests, to be unmuted when next successfully run.
3. run a build where the 3 tests are rigged to pass. (Build #...85)
4. run a build where the 3 tests are rigged to fail. (Build #...86) If the tests within the build are run (and fail) before about 30 minutes after the successful build, the failing tests will be muted (see attached MutedTests-3.png).
- As before, there was this message in the log, about 30 minutes after the successful build (step 3 above):
INFO - jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 3 tests in Dev :: RBird
- I checked the project's Muted Problems tab several times. Between steps 2 and 3 above, the muted problems tab shows all the details of the failed tests, including the build it failed in (see attached MutedTests-1.png). After step 3, the failed tests are still shown, but there are no details (see attached MutedTests-2.png). After step 4, after more the 30 minutes after the successful build, nothing is shown. I was running no other builds in my project for several days, and there was no indication that there were any other builds involved.
Again, this is just a minor issue for us, readily lived with.