Pharo test defects

Cly-misc-queries

State
Fixed (image fix)
Defect in
the test
Tests
5
Fix
image fix 20-Cly-misc-queries.st
Upstream repository
pharo-project/pharo

Record

All five tests were Pharo tests marked as expected failures, and all five failed identically on our VM and Cog. In four of them the test override is a case where the inherited test does not apply to that query, and the author marked it as expected failure instead of testing the real contract. The fifth (ClyExtendingPackagesQueryTest) has an outdated expectation, from before traits-as-multiple-inheritance was removed. The fix replaces each of the five test methods and removes the mark. Constant query: with an empty method scope the result is still not empty (deny checkEmptyResult). Unknown query: scope and query are not bound to the environment (deny both). RestUntagged and Unimplemented queries: the empty scope is the package scope they support (ClyPackageScope emptyIn:), and checkEmptyResult is true. Extending packages: a class scope uses Behavior>>methods, which includes trait methods (ClyLocalClassScope>>methodsDo:). ClyClassWithTraits uses ClyTraitRoot1, which the PExtendingTrait package extends, so that package is the correct single result. Only test methods changed; no image code under test and no VM code changed.

Test Defect in State Cause
ClyConstantMethodQueryTest>>testIsEmptyFromEmptyMethodScope test repaired Override marked expectedFailure because a constant query ignores scope; no real check was made.
ClyExtendingPackagesQueryTest>>testFromClassWhichTraitIsExtendedButNotItself test repaired Expectation of 0 dates from before trait methods became part of Behavior>>methods; class scope now includes the trait extension method from package PExtendingTrait, so the result is that one package.
ClyRestUntaggedClassesQueryTest>>testIsEmptyFromEmptyClassScope test repaired Query supports only package scope; inherited test used a class scope (DNU packagesDo:). Now uses an empty package scope.
ClyUnimplementedMethodsQueryTest>>testIsEmptyFromEmptyMethodScope test repaired Query supports only class, package and global scopes; inherited test used a method scope (DNU classesDo:). Now uses an empty package scope.
ClyUnknownQueryTest>>testIsBoundToEnvironmentWhenScopeIs test repaired Inherited test asserts binding; unknown query is never bound by design. Override now asserts the opposite (deny) for scope and query.

Regression check: With the fix, the five owning test classes were run 3 times on each VM (ours and Cog): all tests pass, class counts 40, 20, 38, 44 and 36 passes, no fail, error or skip. Only test methods changed, so no other class covers a changed method. I did not run a no-fix baseline of the full classes. The first fix attempt, which ran the query in the Constant test, gave 39 of 41 on both VMs (Got 0 instead of 2); I removed that part.

Upstream note: Calypso tests: replace the five expectedFailure overrides in ClyConstantMethodQueryTest, ClyUnknownQueryTest, ClyRestUntaggedClassesQueryTest, ClyUnimplementedMethodsQueryTest and the Traits extension of ClyExtendingPackagesQueryTest with real checks. Each override marked "does not apply to this query" as expected failure. Each one now tests the real contract of the query: an empty supported scope gives an empty result, an unknown query is not bound, and a constant query is not empty from an empty scope. For the traits test, the class scope includes trait methods, so the extending package is expected. This is a Pharo (Calypso) change, not a VM change. Per the owner's rule I have not opened a PR.

Results

Last full sweeps of 2026-11-17: iospharo (sweep14-new14) and stock Cog with the Cog changes (sweepcogp14-new10), with all image fixes. "Before" is the state of the test before the repairs. The Stock Cog column has a value only where the record measured stock Cog: "PASS 3 of 3 (repair run)" is the check of each repair on stock Cog v12.0.6 with the fix entry; "-" is not measured separately.

TestBeforeStateiospharo VMPatched CogStock Cog
ClyConstantMethodQueryTest>>testIsEmptyFromEmptyMethodScopeXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
ClyExtendingPackagesQueryTest>>testFromClassWhichTraitIsExtendedButNotItselfXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
ClyRestUntaggedClassesQueryTest>>testIsEmptyFromEmptyClassScopeXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
ClyUnimplementedMethodsQueryTest>>testIsEmptyFromEmptyMethodScopeXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
ClyUnknownQueryTest>>testIsBoundToEnvironmentWhenScopeIsXFAILrepairedPASSPASSPASS 3 of 3 (repair run)

Test

Run the tests in a Pharo 14 image (build 791). Before the fix they skip, fail or carry the expected-failure mark.

"Run in a Playground of a Pharo 14 image (Do it and print each line)."
((ClyConstantMethodQueryTest selector: #testIsEmptyFromEmptyMethodScope) run) printString.
((ClyExtendingPackagesQueryTest selector: #testFromClassWhichTraitIsExtendedButNotItself) run) printString.
((ClyRestUntaggedClassesQueryTest selector: #testIsEmptyFromEmptyClassScope) run) printString.
((ClyUnimplementedMethodsQueryTest selector: #testIsEmptyFromEmptyMethodScope) run) printString.
((ClyUnknownQueryTest selector: #testIsBoundToEnvironmentWhenScopeIs) run) printString.

Source of the tests before the fix

ClyConstantMethodQueryTest>>testIsEmptyFromEmptyMethodScope before the fix (Pharo 14 build 791)
testIsEmptyFromEmptyMethodScope
	"Constant query do not depends on scope. So we redefine this method"
	<expectedFailure>
	super testIsEmptyFromEmptyMethodScope
ClyExtendingPackagesQueryTest>>testFromClassWhichTraitIsExtendedButNotItself before the fix (Pharo 14 build 791)
testFromClassWhichTraitIsExtendedButNotItself
	"I'm tagging this expected failure because I'm not sure now if it is appropriate to keep
	 or not with the removal of the 'traits-as-multiple-inheritance' stuff."
	<expectedFailure>

	self queryFromScope: ClyClassScope of: ClyClassWithTraits.
	self assert: resultItems size equals: 0
ClyRestUntaggedClassesQueryTest>>testIsEmptyFromEmptyClassScope before the fix (Pharo 14 build 791)
testIsEmptyFromEmptyClassScope
	"This query supports only package scope"
	<expectedFailure>

	super testIsEmptyFromEmptyClassScope
ClyUnimplementedMethodsQueryTest>>testIsEmptyFromEmptyMethodScope before the fix (Pharo 14 build 791)
testIsEmptyFromEmptyMethodScope
	"This query only supports class, package and global scopes
	because it analyses classes and only then methods"
	<expectedFailure>
	super testIsEmptyFromEmptyMethodScope
ClyUnknownQueryTest>>testIsBoundToEnvironmentWhenScopeIs before the fix (Pharo 14 build 791)
testIsBoundToEnvironmentWhenScopeIs
	<expectedFailure>
	"Unknown query is never bound to the environment"
	super testIsBoundToEnvironmentWhenScopeIs

Fix

How to apply an entries file: file in 00-header.st and the entries file as one chunk (the iospharo script scripts/gen-image-fixes.py makes that chunk). Each entry checks for the defect first and prints FIXED, ADDED or SKIP. Download 00-header.st.

20-Cly-misc-queries.st (download)
fix value: #ClyConstantMethodQueryTest value: #testIsEmptyFromEmptyMethodScope
	value: [ :s | s includesSubstring: '<expectedFailure>' ]
	value: 'testIsEmptyFromEmptyMethodScope
	"Constant query does not depend on the scope. The inherited test expects an empty result from an empty method scope; this query always returns its own methods, so the result is not empty."
	query scope: (ClyMethodScope emptyIn: environment).

	self deny: query checkEmptyResult'.

fix value: #ClyUnknownQueryTest value: #testIsBoundToEnvironmentWhenScopeIs
	value: [ :s | s includesSubstring: '<expectedFailure>' ]
	value: 'testIsBoundToEnvironmentWhenScopeIs
	"Unknown query is never bound to the environment"
	self deny: query scope isBoundToEnvironment.
	self deny: query isBoundToEnvironment'.

fix value: #ClyRestUntaggedClassesQueryTest value: #testIsEmptyFromEmptyClassScope
	value: [ :s | s includesSubstring: '<expectedFailure>' ]
	value: 'testIsEmptyFromEmptyClassScope
	"This query supports only package scope, so the empty scope is a package scope"
	query scope: (ClyPackageScope emptyIn: environment).

	self assert: query checkEmptyResult'.

fix value: #ClyUnimplementedMethodsQueryTest value: #testIsEmptyFromEmptyMethodScope
	value: [ :s | s includesSubstring: '<expectedFailure>' ]
	value: 'testIsEmptyFromEmptyMethodScope
	"This query only supports class, package and global scopes
	because it analyses classes and only then methods.
	So the empty scope is a package scope"
	query scope: (ClyPackageScope emptyIn: environment).

	self assert: query checkEmptyResult'.

fix value: #ClyExtendingPackagesQueryTest value: #testFromClassWhichTraitIsExtendedButNotItself
	value: [ :s | s includesSubstring: '<expectedFailure>' ]
	value: 'testFromClassWhichTraitIsExtendedButNotItself
	"The methods of a class include the methods that come from its traits (ClyLocalClassScope>>methodsDo: uses Behavior>>methods).
	ClyClassWithTraits uses ClyTraitRoot1, which another package extends, so that package extends the methods of the class."

	self queryFromScope: ClyClassScope of: ClyClassWithTraits.
	self assert: resultItems asSet equals: { (ClyTraitRoot1 >> #extensionMethodFromRoot1Trait) package } asSet'.

State of 2026-11-20. Made from the test records of the iospharo project (a second Pharo VM). The fixes are proposals for review.