Open: AllTestFailTest (4 tests) and DependencyTest (6 tests) have no other test that runs them. A skip is accepted only for a fixture that another test runs, so these 10 tests are still open.
All 35 tests are fixtures, and none has a defect to repair. They exist as data for other tests. MockTest class>>initialize sets ShouldIgnore to true so the TestRunner/RottenTestsFinder fixture classes are not run by the normal suite sweep, and RTFFakeTestSuperClassTest>>shouldIgnore defaults to true. The consumer tests call doNotIgnore in setUp and ignore in tearDown. ClyTestedClassMockTest>>testExpectedFailure is a fixture whose purpose is to be an expected failure; the Cly*Query tests look it up by its pragma. I made no change to any test or image method, so there is no fix file. Eight of the 35 tests (AllTestFailTest 4, DependencyTest 6 is separate: see tests list) have no consumer test at all, and LongTest's only consumer skips itself; both are noted under blockers.
| Test | Defect in | State | Cause |
|---|---|---|---|
MockTest subclasses: EmptyTest>>test1-4, GreenTest>>testPass/testPassAgain, RedTest>>testFail/testPass/testWithError, YellowTest>>testFail/testPass, FailThenPassTest>>testFailAtFirstRunning/testPass |
none-not-applicable | not applicable | Fixtures whose skip is the by-design ShouldIgnore flag (true after MockTest class>>initialize). Consumers: NewTestRunnerTest (26 tests; uses GreenTest, RedTest, YellowTest, FailThenPassTest), ShuffleSuitePreparatorTest (EmptyTest suite, 2 tests), SuitePreparatorTest, StdReporterTest, XmlReporterTest, NewCommandLineRunnerTest (GreenTest). Measured: NewTestRunnerTest 26/26 pass, ShuffleSuitePreparatorTest 2/2, SuitePreparatorTest 1/1, on both VMs. MockTest classPool ShouldIgnore = true on both VMs via evalfile. |
AllTestFailTest>>testFail1-4 |
none-not-applicable | not applicable | Fixture whose four tests assert false. With MockTest doNotIgnore, AllTestFailTest suite run gives 4 ran, 0 passed, 4 failures on both VMs. Failing is its purpose. No test or method in the image references AllTestFailTest (allReferencesTo: size 0), so no consumer exists to cite. |
DependencyTest>>testDataIs10..15ThenIncrement |
none-not-applicable | not applicable | Fixture for order dependence: only testDataIs10 resets the shared counter, so the other five pass only when run in selector order. TestSuite run order is shuffled; with the flag off, 3 runs of the six tests gave 1 or 2 passes on both VMs and the failing set varied from run to run on each VM (ours 2,1,.. passes; Cog 1,1,..). The pass count difference seen in one earlier run (ours 4 vs Cog 1) comes from the random order, not from the VM. No test in the image references DependencyTest, so no consumer exists to cite. |
LongTest>>testIsTakingTooLong |
none-not-applicable | not applicable | Fixture run only by RunnerLongDurationTest, whose two tests call self skip as their first statement. That is a different cluster (clusters/RunnerLongDurationTest.txt). LongTest itself is a 3 s Delay wait and an assertion that would pass. |
RTFFakeTestClassTest>>testCallingSuperHelper, testFailingButNotConsideredAsRotten, testHelperHelperAssertHelper, testWithAssertionInHelperRun, testWithAssertionInSuperHelperRun, testWithAssertionRun, testWithHelperHelper, testWithNoAssertionInHelperRun, testWithNoAssertionInSuperHelperRun, testWithoutAssertionRun |
none-not-applicable | not applicable | Fixture data for RottenTestsFinder. RTFFakeTestSuperClassTest>>shouldIgnore defaults to true. RottenTestsFinderTest (subclass of RTFMetaTest, which calls doNotIgnore) analyzes RTFFakeTestClassTest suite and asserts 3 rotten tests, so it needs the fixture tests to run. RTFSelfCallStackBuilderTest, RTFSelfCallVisitorInterpreterTest, RTFPrimitiveCallsCollectorTest and RTFLeadsToAssertPrimitiveCallCheckerTest also use the class. Measured on both VMs: RottenTestsFinderTest 1/1, RTFSelfCallStackBuilderTest 5/5, RTFSelfCallVisitorInterpreterTest 2/2, RTFPrimitiveCallsCollectorTest 1/1, RTFLeadsToAssertPrimitiveCallCheckerTest 1/1 pass. RTFMetaTest itself has 0 tests (abstract). |
ClyTestedClassMockTest>>testExpectedFailure |
none-not-applicable | not applicable | Fixture whose body is assert: false with the expectedFailure pragma; it must stay an expected failure. ClyExpectedFailedTestMethodsQueryTest (35 pass) asserts that the query finds exactly #(testExpectedFailure) in this class, ClyExpectedFailureMethodGroupProviderTest (1 pass) and ClyTestedEnvironmentPluginTest (10 pass) use the class. Removing the mark or the failure would break these. The test fails by design on both VMs. |
Regression check: No image change was made, so nothing was compared with and without a fix. I ran the consumer classes on both VMs (ours and Cog, no fix applied): NewTestRunnerTest 26 pass, RunnerLongDurationTest 2 skip, ShuffleSuitePreparatorTest 2 pass, SuitePreparatorTest 1 pass, RottenTestsFinderTest 1, RTFSelfCallStackBuilderTest 5, RTFSelfCallVisitorInterpreterTest 2, RTFPrimitiveCallsCollectorTest 1, RTFLeadsToAssertPrimitiveCallCheckerTest 1, ClyExpectedFailedTestMethodsQueryTest 35, ClyExpectedFailureMethodGroupProviderTest 1, ClyTestedEnvironmentPluginTest 10 pass. Our VM and Cog gave identical counts in every class. The tests were run once per VM, not three times, since nothing was repaired.
Open points: Three gaps in the "fixture run by another test that passes" proof. (1) AllTestFailTest (4 tests) and DependencyTest (6 tests) have no referencing test in the image; I showed they are designed to fail or to depend on order, but no consumer passes for them. (2) LongTest>>testIsTakingTooLong is used only by RunnerLongDurationTest, whose tests both call self skip; those belong to another cluster (RunnerLongDurationTest), and when that cluster removes the skips, LongTest will run under it. (3) I did not run StdReporterTest, XmlReporterTest or NewCommandLineRunnerTest (GreenTest consumers). The summary text also contains a garbled count: the fixture tests are 35, with no overlap between the groups in the tests list.
Upstream note: No upstream change proposed. Observations only: AllTestFailTest and DependencyTest in TestRunner-FakeTests have no consumer test in the image, and RunnerLongDurationTest (consumer of LongTest) skips both of its tests with self skip.
Last full sweeps of 2026-11-17: iospharo (sweep14-new14) and stock Cog with the Cog changes (sweepcogp14-new10), with all image fixes. "Before" is the state of the test before the repairs. The Stock Cog column has a value only where the record measured stock Cog: "PASS 3 of 3 (repair run)" is the check of each repair on stock Cog v12.0.6 with the fix entry; "-" is not measured separately.
| Test | Before | State | iospharo VM | Patched Cog | Stock Cog |
|---|---|---|---|---|---|
AllTestFailTest>>testFail1 | SKIP | not applicable | SKIP | SKIP | - |
AllTestFailTest>>testFail2 | SKIP | not applicable | SKIP | SKIP | - |
AllTestFailTest>>testFail3 | SKIP | not applicable | SKIP | SKIP | - |
AllTestFailTest>>testFail4 | SKIP | not applicable | SKIP | SKIP | - |
ClyTestedClassMockTest>>testExpectedFailure | XFAIL | not applicable | XFAIL | XFAIL | - |
DependencyTest>>testDataIs10ThenIncrement | SKIP | not applicable | SKIP | SKIP | - |
DependencyTest>>testDataIs11ThenIncrement | SKIP | not applicable | SKIP | SKIP | - |
DependencyTest>>testDataIs12ThenIncrement | SKIP | not applicable | SKIP | SKIP | - |
DependencyTest>>testDataIs13ThenIncrement | SKIP | not applicable | SKIP | SKIP | - |
DependencyTest>>testDataIs14ThenIncrement | SKIP | not applicable | SKIP | SKIP | - |
DependencyTest>>testDataIs15ThenIncrement | SKIP | not applicable | SKIP | SKIP | - |
EmptyTest>>test1 | SKIP | not applicable | SKIP | SKIP | - |
EmptyTest>>test2 | SKIP | not applicable | SKIP | SKIP | - |
EmptyTest>>test3 | SKIP | not applicable | SKIP | SKIP | - |
EmptyTest>>test4 | SKIP | not applicable | SKIP | SKIP | - |
FailThenPassTest>>testFailAtFirstRunning | SKIP | not applicable | SKIP | SKIP | - |
FailThenPassTest>>testPass | SKIP | not applicable | SKIP | SKIP | - |
GreenTest>>testPass | SKIP | not applicable | SKIP | SKIP | - |
GreenTest>>testPassAgain | SKIP | not applicable | SKIP | SKIP | - |
LongTest>>testIsTakingTooLong | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testCallingSuperHelper | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testFailingButNotConsideredAsRotten | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testHelperHelperAssertHelper | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithAssertionInHelperRun | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithAssertionInSuperHelperRun | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithAssertionRun | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithHelperHelper | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithNoAssertionInHelperRun | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithNoAssertionInSuperHelperRun | SKIP | not applicable | SKIP | SKIP | - |
RTFFakeTestClassTest>>testWithoutAssertionRun | SKIP | not applicable | SKIP | SKIP | - |
RedTest>>testFail | SKIP | not applicable | SKIP | SKIP | - |
RedTest>>testPass | SKIP | not applicable | SKIP | SKIP | - |
RedTest>>testWithError | SKIP | not applicable | SKIP | SKIP | - |
YellowTest>>testFail | SKIP | not applicable | SKIP | SKIP | - |
YellowTest>>testPass | SKIP | not applicable | SKIP | SKIP | - |
Run the tests in a Pharo 14 image on macOS arm64: they skip (the fixtures and the platform tests skip by design).
"Run in a Playground of a Pharo 14 image (Do it and print each line)." ((AllTestFailTest selector: #testFail1) run) printString. ((AllTestFailTest selector: #testFail2) run) printString. ((AllTestFailTest selector: #testFail3) run) printString. ((AllTestFailTest selector: #testFail4) run) printString. ((ClyTestedClassMockTest selector: #testExpectedFailure) run) printString. ((DependencyTest selector: #testDataIs10ThenIncrement) run) printString. ((DependencyTest selector: #testDataIs11ThenIncrement) run) printString. ((DependencyTest selector: #testDataIs12ThenIncrement) run) printString. ((DependencyTest selector: #testDataIs13ThenIncrement) run) printString. ((DependencyTest selector: #testDataIs14ThenIncrement) run) printString. ((DependencyTest selector: #testDataIs15ThenIncrement) run) printString. ((EmptyTest selector: #test1) run) printString. ((EmptyTest selector: #test2) run) printString. ((EmptyTest selector: #test3) run) printString. ((EmptyTest selector: #test4) run) printString. ((FailThenPassTest selector: #testFailAtFirstRunning) run) printString. ((FailThenPassTest selector: #testPass) run) printString. ((GreenTest selector: #testPass) run) printString. ((GreenTest selector: #testPassAgain) run) printString. ((LongTest selector: #testIsTakingTooLong) run) printString. ((RTFFakeTestClassTest selector: #testCallingSuperHelper) run) printString. ((RTFFakeTestClassTest selector: #testFailingButNotConsideredAsRotten) run) printString. ((RTFFakeTestClassTest selector: #testHelperHelperAssertHelper) run) printString. ((RTFFakeTestClassTest selector: #testWithAssertionInHelperRun) run) printString. ((RTFFakeTestClassTest selector: #testWithAssertionInSuperHelperRun) run) printString. ((RTFFakeTestClassTest selector: #testWithAssertionRun) run) printString. ((RTFFakeTestClassTest selector: #testWithHelperHelper) run) printString. ((RTFFakeTestClassTest selector: #testWithNoAssertionInHelperRun) run) printString. ((RTFFakeTestClassTest selector: #testWithNoAssertionInSuperHelperRun) run) printString. ((RTFFakeTestClassTest selector: #testWithoutAssertionRun) run) printString. ((RedTest selector: #testFail) run) printString. ((RedTest selector: #testPass) run) printString. ((RedTest selector: #testWithError) run) printString. ((YellowTest selector: #testFail) run) printString. ((YellowTest selector: #testPass) run) printString.
testFail1 self assert: false
testFail2 self assert: false
testFail3 self assert: false
testFail4 self assert: false
testExpectedFailure <expectedFailure> self assert: false description: 'it is example test with expected failure'
testDataIs10ThenIncrement self resetData; assertCountThenIncrement: 0
testDataIs11ThenIncrement self assertCountThenIncrement: 1
testDataIs12ThenIncrement self assertCountThenIncrement: 2
testDataIs13ThenIncrement self assertCountThenIncrement: 3
testDataIs14ThenIncrement self assertCountThenIncrement: 4
testDataIs15ThenIncrement self assertCountThenIncrement: 5
test1
test2
test3
test4
testFailAtFirstRunning self assert: self class counter equals: 1
testPass self assert: true
testPass self assert: true
testPassAgain self assert: true
testIsTakingTooLong (Delay forSeconds: 3) wait. self assert: true
testCallingSuperHelper self superAssertHelper
testFailingButNotConsideredAsRotten self assert: false. self assert: true
testHelperHelperAssertHelper self helperHelper; assertHelper
testWithAssertionInHelperRun self assertHelper
testWithAssertionInSuperHelperRun self superAssertHelper
testWithAssertionRun self assert: true
testWithHelperHelper self helperHelper
testWithNoAssertionInHelperRun self rottenHelper
testWithNoAssertionInSuperHelperRun self superNotAssertHelper
testWithoutAssertionRun false ifTrue: [ self assert: true ]
testFail self assert: false
testPass self assert: true
testWithError self assert: nil
testFail self assert: false
testPass self assert: true.
No fix: the tests cannot apply on this machine (proof in the record above).
State of 2026-11-20. Made from the test records of the iospharo project (a second Pharo VM). The fixes are proposals for review.