Pharo test defects

NA-fixture-tests

State
Not applicable (proof incomplete for 10 tests)
Defect in
none: fixture classes that other tests run, skipped by design
Tests
35
Fix
none
Upstream repository
pharo-project/pharo

Open: AllTestFailTest (4 tests) and DependencyTest (6 tests) have no other test that runs them. A skip is accepted only for a fixture that another test runs, so these 10 tests are still open.

Record

All 35 tests are fixtures, and none has a defect to repair. They exist as data for other tests. MockTest class>>initialize sets ShouldIgnore to true so the TestRunner/RottenTestsFinder fixture classes are not run by the normal suite sweep, and RTFFakeTestSuperClassTest>>shouldIgnore defaults to true. The consumer tests call doNotIgnore in setUp and ignore in tearDown. ClyTestedClassMockTest>>testExpectedFailure is a fixture whose purpose is to be an expected failure; the Cly*Query tests look it up by its pragma. I made no change to any test or image method, so there is no fix file. Eight of the 35 tests (AllTestFailTest 4, DependencyTest 6 is separate: see tests list) have no consumer test at all, and LongTest's only consumer skips itself; both are noted under blockers.

Test Defect in State Cause
MockTest subclasses: EmptyTest>>test1-4, GreenTest>>testPass/testPassAgain, RedTest>>testFail/testPass/testWithError, YellowTest>>testFail/testPass, FailThenPassTest>>testFailAtFirstRunning/testPass none-not-applicable not applicable Fixtures whose skip is the by-design ShouldIgnore flag (true after MockTest class>>initialize). Consumers: NewTestRunnerTest (26 tests; uses GreenTest, RedTest, YellowTest, FailThenPassTest), ShuffleSuitePreparatorTest (EmptyTest suite, 2 tests), SuitePreparatorTest, StdReporterTest, XmlReporterTest, NewCommandLineRunnerTest (GreenTest). Measured: NewTestRunnerTest 26/26 pass, ShuffleSuitePreparatorTest 2/2, SuitePreparatorTest 1/1, on both VMs. MockTest classPool ShouldIgnore = true on both VMs via evalfile.
AllTestFailTest>>testFail1-4 none-not-applicable not applicable Fixture whose four tests assert false. With MockTest doNotIgnore, AllTestFailTest suite run gives 4 ran, 0 passed, 4 failures on both VMs. Failing is its purpose. No test or method in the image references AllTestFailTest (allReferencesTo: size 0), so no consumer exists to cite.
DependencyTest>>testDataIs10..15ThenIncrement none-not-applicable not applicable Fixture for order dependence: only testDataIs10 resets the shared counter, so the other five pass only when run in selector order. TestSuite run order is shuffled; with the flag off, 3 runs of the six tests gave 1 or 2 passes on both VMs and the failing set varied from run to run on each VM (ours 2,1,.. passes; Cog 1,1,..). The pass count difference seen in one earlier run (ours 4 vs Cog 1) comes from the random order, not from the VM. No test in the image references DependencyTest, so no consumer exists to cite.
LongTest>>testIsTakingTooLong none-not-applicable not applicable Fixture run only by RunnerLongDurationTest, whose two tests call self skip as their first statement. That is a different cluster (clusters/RunnerLongDurationTest.txt). LongTest itself is a 3 s Delay wait and an assertion that would pass.
RTFFakeTestClassTest>>testCallingSuperHelper, testFailingButNotConsideredAsRotten, testHelperHelperAssertHelper, testWithAssertionInHelperRun, testWithAssertionInSuperHelperRun, testWithAssertionRun, testWithHelperHelper, testWithNoAssertionInHelperRun, testWithNoAssertionInSuperHelperRun, testWithoutAssertionRun none-not-applicable not applicable Fixture data for RottenTestsFinder. RTFFakeTestSuperClassTest>>shouldIgnore defaults to true. RottenTestsFinderTest (subclass of RTFMetaTest, which calls doNotIgnore) analyzes RTFFakeTestClassTest suite and asserts 3 rotten tests, so it needs the fixture tests to run. RTFSelfCallStackBuilderTest, RTFSelfCallVisitorInterpreterTest, RTFPrimitiveCallsCollectorTest and RTFLeadsToAssertPrimitiveCallCheckerTest also use the class. Measured on both VMs: RottenTestsFinderTest 1/1, RTFSelfCallStackBuilderTest 5/5, RTFSelfCallVisitorInterpreterTest 2/2, RTFPrimitiveCallsCollectorTest 1/1, RTFLeadsToAssertPrimitiveCallCheckerTest 1/1 pass. RTFMetaTest itself has 0 tests (abstract).
ClyTestedClassMockTest>>testExpectedFailure none-not-applicable not applicable Fixture whose body is assert: false with the expectedFailure pragma; it must stay an expected failure. ClyExpectedFailedTestMethodsQueryTest (35 pass) asserts that the query finds exactly #(testExpectedFailure) in this class, ClyExpectedFailureMethodGroupProviderTest (1 pass) and ClyTestedEnvironmentPluginTest (10 pass) use the class. Removing the mark or the failure would break these. The test fails by design on both VMs.

Regression check: No image change was made, so nothing was compared with and without a fix. I ran the consumer classes on both VMs (ours and Cog, no fix applied): NewTestRunnerTest 26 pass, RunnerLongDurationTest 2 skip, ShuffleSuitePreparatorTest 2 pass, SuitePreparatorTest 1 pass, RottenTestsFinderTest 1, RTFSelfCallStackBuilderTest 5, RTFSelfCallVisitorInterpreterTest 2, RTFPrimitiveCallsCollectorTest 1, RTFLeadsToAssertPrimitiveCallCheckerTest 1, ClyExpectedFailedTestMethodsQueryTest 35, ClyExpectedFailureMethodGroupProviderTest 1, ClyTestedEnvironmentPluginTest 10 pass. Our VM and Cog gave identical counts in every class. The tests were run once per VM, not three times, since nothing was repaired.

Open points: Three gaps in the "fixture run by another test that passes" proof. (1) AllTestFailTest (4 tests) and DependencyTest (6 tests) have no referencing test in the image; I showed they are designed to fail or to depend on order, but no consumer passes for them. (2) LongTest>>testIsTakingTooLong is used only by RunnerLongDurationTest, whose tests both call self skip; those belong to another cluster (RunnerLongDurationTest), and when that cluster removes the skips, LongTest will run under it. (3) I did not run StdReporterTest, XmlReporterTest or NewCommandLineRunnerTest (GreenTest consumers). The summary text also contains a garbled count: the fixture tests are 35, with no overlap between the groups in the tests list.

Upstream note: No upstream change proposed. Observations only: AllTestFailTest and DependencyTest in TestRunner-FakeTests have no consumer test in the image, and RunnerLongDurationTest (consumer of LongTest) skips both of its tests with self skip.

Results

Last full sweeps of 2026-11-17: iospharo (sweep14-new14) and stock Cog with the Cog changes (sweepcogp14-new10), with all image fixes. "Before" is the state of the test before the repairs. The Stock Cog column has a value only where the record measured stock Cog: "PASS 3 of 3 (repair run)" is the check of each repair on stock Cog v12.0.6 with the fix entry; "-" is not measured separately.

TestBeforeStateiospharo VMPatched CogStock Cog
AllTestFailTest>>testFail1SKIPnot applicableSKIPSKIP-
AllTestFailTest>>testFail2SKIPnot applicableSKIPSKIP-
AllTestFailTest>>testFail3SKIPnot applicableSKIPSKIP-
AllTestFailTest>>testFail4SKIPnot applicableSKIPSKIP-
ClyTestedClassMockTest>>testExpectedFailureXFAILnot applicableXFAILXFAIL-
DependencyTest>>testDataIs10ThenIncrementSKIPnot applicableSKIPSKIP-
DependencyTest>>testDataIs11ThenIncrementSKIPnot applicableSKIPSKIP-
DependencyTest>>testDataIs12ThenIncrementSKIPnot applicableSKIPSKIP-
DependencyTest>>testDataIs13ThenIncrementSKIPnot applicableSKIPSKIP-
DependencyTest>>testDataIs14ThenIncrementSKIPnot applicableSKIPSKIP-
DependencyTest>>testDataIs15ThenIncrementSKIPnot applicableSKIPSKIP-
EmptyTest>>test1SKIPnot applicableSKIPSKIP-
EmptyTest>>test2SKIPnot applicableSKIPSKIP-
EmptyTest>>test3SKIPnot applicableSKIPSKIP-
EmptyTest>>test4SKIPnot applicableSKIPSKIP-
FailThenPassTest>>testFailAtFirstRunningSKIPnot applicableSKIPSKIP-
FailThenPassTest>>testPassSKIPnot applicableSKIPSKIP-
GreenTest>>testPassSKIPnot applicableSKIPSKIP-
GreenTest>>testPassAgainSKIPnot applicableSKIPSKIP-
LongTest>>testIsTakingTooLongSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testCallingSuperHelperSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testFailingButNotConsideredAsRottenSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testHelperHelperAssertHelperSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithAssertionInHelperRunSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithAssertionInSuperHelperRunSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithAssertionRunSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithHelperHelperSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithNoAssertionInHelperRunSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithNoAssertionInSuperHelperRunSKIPnot applicableSKIPSKIP-
RTFFakeTestClassTest>>testWithoutAssertionRunSKIPnot applicableSKIPSKIP-
RedTest>>testFailSKIPnot applicableSKIPSKIP-
RedTest>>testPassSKIPnot applicableSKIPSKIP-
RedTest>>testWithErrorSKIPnot applicableSKIPSKIP-
YellowTest>>testFailSKIPnot applicableSKIPSKIP-
YellowTest>>testPassSKIPnot applicableSKIPSKIP-

Test

Run the tests in a Pharo 14 image on macOS arm64: they skip (the fixtures and the platform tests skip by design).

"Run in a Playground of a Pharo 14 image (Do it and print each line)."
((AllTestFailTest selector: #testFail1) run) printString.
((AllTestFailTest selector: #testFail2) run) printString.
((AllTestFailTest selector: #testFail3) run) printString.
((AllTestFailTest selector: #testFail4) run) printString.
((ClyTestedClassMockTest selector: #testExpectedFailure) run) printString.
((DependencyTest selector: #testDataIs10ThenIncrement) run) printString.
((DependencyTest selector: #testDataIs11ThenIncrement) run) printString.
((DependencyTest selector: #testDataIs12ThenIncrement) run) printString.
((DependencyTest selector: #testDataIs13ThenIncrement) run) printString.
((DependencyTest selector: #testDataIs14ThenIncrement) run) printString.
((DependencyTest selector: #testDataIs15ThenIncrement) run) printString.
((EmptyTest selector: #test1) run) printString.
((EmptyTest selector: #test2) run) printString.
((EmptyTest selector: #test3) run) printString.
((EmptyTest selector: #test4) run) printString.
((FailThenPassTest selector: #testFailAtFirstRunning) run) printString.
((FailThenPassTest selector: #testPass) run) printString.
((GreenTest selector: #testPass) run) printString.
((GreenTest selector: #testPassAgain) run) printString.
((LongTest selector: #testIsTakingTooLong) run) printString.
((RTFFakeTestClassTest selector: #testCallingSuperHelper) run) printString.
((RTFFakeTestClassTest selector: #testFailingButNotConsideredAsRotten) run) printString.
((RTFFakeTestClassTest selector: #testHelperHelperAssertHelper) run) printString.
((RTFFakeTestClassTest selector: #testWithAssertionInHelperRun) run) printString.
((RTFFakeTestClassTest selector: #testWithAssertionInSuperHelperRun) run) printString.
((RTFFakeTestClassTest selector: #testWithAssertionRun) run) printString.
((RTFFakeTestClassTest selector: #testWithHelperHelper) run) printString.
((RTFFakeTestClassTest selector: #testWithNoAssertionInHelperRun) run) printString.
((RTFFakeTestClassTest selector: #testWithNoAssertionInSuperHelperRun) run) printString.
((RTFFakeTestClassTest selector: #testWithoutAssertionRun) run) printString.
((RedTest selector: #testFail) run) printString.
((RedTest selector: #testPass) run) printString.
((RedTest selector: #testWithError) run) printString.
((YellowTest selector: #testFail) run) printString.
((YellowTest selector: #testPass) run) printString.

Source of the tests before the fix

AllTestFailTest>>testFail1 before the fix (Pharo 14 build 791)
testFail1

	self assert: false
AllTestFailTest>>testFail2 before the fix (Pharo 14 build 791)
testFail2

	self assert: false
AllTestFailTest>>testFail3 before the fix (Pharo 14 build 791)
testFail3

	self assert: false
AllTestFailTest>>testFail4 before the fix (Pharo 14 build 791)
testFail4

	self assert: false
ClyTestedClassMockTest>>testExpectedFailure before the fix (Pharo 14 build 791)
testExpectedFailure
	<expectedFailure>
	self assert: false description: 'it is example test with expected failure'
DependencyTest>>testDataIs10ThenIncrement before the fix (Pharo 14 build 791)
testDataIs10ThenIncrement

	self 
		resetData;
		assertCountThenIncrement: 0
DependencyTest>>testDataIs11ThenIncrement before the fix (Pharo 14 build 791)
testDataIs11ThenIncrement

	self assertCountThenIncrement: 1
DependencyTest>>testDataIs12ThenIncrement before the fix (Pharo 14 build 791)
testDataIs12ThenIncrement

	self assertCountThenIncrement: 2
DependencyTest>>testDataIs13ThenIncrement before the fix (Pharo 14 build 791)
testDataIs13ThenIncrement

	self assertCountThenIncrement: 3
DependencyTest>>testDataIs14ThenIncrement before the fix (Pharo 14 build 791)
testDataIs14ThenIncrement

	self assertCountThenIncrement: 4
DependencyTest>>testDataIs15ThenIncrement before the fix (Pharo 14 build 791)
testDataIs15ThenIncrement

	self assertCountThenIncrement: 5
EmptyTest>>test1 before the fix (Pharo 14 build 791)
test1
EmptyTest>>test2 before the fix (Pharo 14 build 791)
test2
EmptyTest>>test3 before the fix (Pharo 14 build 791)
test3
EmptyTest>>test4 before the fix (Pharo 14 build 791)
test4
FailThenPassTest>>testFailAtFirstRunning before the fix (Pharo 14 build 791)
testFailAtFirstRunning

	self assert: self class counter equals: 1
FailThenPassTest>>testPass before the fix (Pharo 14 build 791)
testPass

	self assert: true
GreenTest>>testPass before the fix (Pharo 14 build 791)
testPass

	self assert: true
GreenTest>>testPassAgain before the fix (Pharo 14 build 791)
testPassAgain

	self assert: true
LongTest>>testIsTakingTooLong before the fix (Pharo 14 build 791)
testIsTakingTooLong

	(Delay forSeconds: 3) wait.
	 self assert: true
RTFFakeTestClassTest>>testCallingSuperHelper before the fix (Pharo 14 build 791)
testCallingSuperHelper
	self superAssertHelper
RTFFakeTestClassTest>>testFailingButNotConsideredAsRotten before the fix (Pharo 14 build 791)
testFailingButNotConsideredAsRotten

	self assert: false.

	self assert: true
RTFFakeTestClassTest>>testHelperHelperAssertHelper before the fix (Pharo 14 build 791)
testHelperHelperAssertHelper
	
	self helperHelper; assertHelper
RTFFakeTestClassTest>>testWithAssertionInHelperRun before the fix (Pharo 14 build 791)
testWithAssertionInHelperRun
	self assertHelper
RTFFakeTestClassTest>>testWithAssertionInSuperHelperRun before the fix (Pharo 14 build 791)
testWithAssertionInSuperHelperRun
	self superAssertHelper
RTFFakeTestClassTest>>testWithAssertionRun before the fix (Pharo 14 build 791)
testWithAssertionRun
	self assert: true
RTFFakeTestClassTest>>testWithHelperHelper before the fix (Pharo 14 build 791)
testWithHelperHelper
	self helperHelper
RTFFakeTestClassTest>>testWithNoAssertionInHelperRun before the fix (Pharo 14 build 791)
testWithNoAssertionInHelperRun
	self rottenHelper
RTFFakeTestClassTest>>testWithNoAssertionInSuperHelperRun before the fix (Pharo 14 build 791)
testWithNoAssertionInSuperHelperRun
	self superNotAssertHelper
RTFFakeTestClassTest>>testWithoutAssertionRun before the fix (Pharo 14 build 791)
testWithoutAssertionRun
	false
		ifTrue: [ self assert: true ]
RedTest>>testFail before the fix (Pharo 14 build 791)
testFail

	self assert: false
RedTest>>testPass before the fix (Pharo 14 build 791)
testPass

	self assert: true
RedTest>>testWithError before the fix (Pharo 14 build 791)
testWithError

	self assert: nil
YellowTest>>testFail before the fix (Pharo 14 build 791)
testFail

	self assert: false
YellowTest>>testPass before the fix (Pharo 14 build 791)
testPass
	self assert: true.

Fix

No fix: the tests cannot apply on this machine (proof in the record above).

State of 2026-11-20. Made from the test records of the iospharo project (a second Pharo VM). The fixes are proposals for review.