Pharo test defects

Fuel-widestring-globals

State
Fixed (image fix)
Defect in
the Pharo image
Tests
15
Fix
image fix 20-Fuel-widestring-globals.st
Upstream repository
pharo-project/pharo

Record

Two defects. (1) Fuel (image code) wrote the name of a global (class name or global key) with encodeString:, which stores one byte per character, so the high bytes of a WideString/WideSymbol name were lost and the materializer looked up a wrong name (FLGlobalNotFound, FLClassNotFound). The fix writes global names as UTF-8 and reads them back as UTF-8, with a Latin-1 fallback for bytes that are not valid UTF-8; for ASCII names the bytes are identical to before, so existing files still load. (2) testWideStringGlobal was itself wrong: it used a WideString as a key of SystemEnvironment (which accepts only Symbols) and then asserted that the materialized value is identical to the KEY. The key is now a WideSymbol and the assertion compares with the global value that the key finds. The expectedFailures override in FLSerializationTest (the cause of the 'expected failure' mark) now answers #() because all three tests pass. Behaviour is the same on our VM and on Cog; no VM defect.

Test Defect in State Cause
FLBinaryFileStreamBasicSerializationTest>>testConsiderCustomWideSymbolGlobal image-code repaired FLGlobalCluster encoded global names one byte per character; wide names lost high bytes
FLBinaryFileStreamBasicSerializationTest>>testWideStringClassName image-code repaired same: class name written as bytes, FLClassNotFound
FLBinaryFileStreamBasicSerializationTest>>testWideStringGlobal test repaired WideString key rejected by SystemEnvironment>>at:put: (Symbols only); assertion compared value with the key. Fixed in the test plus the name encoding
FLByteArrayBasicSerializationTest>>testConsiderCustomWideSymbolGlobal image-code repaired same as above
FLByteArrayBasicSerializationTest>>testWideStringClassName image-code repaired same as above
FLByteArrayBasicSerializationTest>>testWideStringGlobal test repaired same as above
FLFileReferenceStreamBasicSerializationTest>>testConsiderCustomWideSymbolGlobal image-code repaired same as above
FLFileReferenceStreamBasicSerializationTest>>testWideStringClassName image-code repaired same as above
FLFileReferenceStreamBasicSerializationTest>>testWideStringGlobal test repaired same as above
FLFullBasicSerializationTest>>testConsiderCustomWideSymbolGlobal image-code repaired same as above
FLFullBasicSerializationTest>>testWideStringClassName image-code repaired same as above
FLFullBasicSerializationTest>>testWideStringGlobal test repaired same as above
FLGZippedBasicSerializationTest>>testConsiderCustomWideSymbolGlobal image-code repaired same as above
FLGZippedBasicSerializationTest>>testWideStringClassName image-code repaired same as above
FLGZippedBasicSerializationTest>>testWideStringGlobal test repaired same as above

Regression check: Whole classes run on both VMs, base image vs fixed image. Our VM and Cog gave the same counts. Base to fixed: FLBinaryFileStream, FLByteArray, FLFileReferenceStream BasicSerializationTest 75 pass + 2 fail + 1 error to 78 pass; FLGZippedBasicSerializationTest 78 pass + 2 fail + 1 error to 81 pass; FLFullBasicSerializationTest 82 pass + 2 fail + 1 error to 85 pass (a first base run of this class came back empty because of a parallel-run collision; I re-ran it alone to get the 82/2/1). Unchanged before and after: FLBlockClosureSerializationTest 18, FLCompiledMethodSerializationTest 6, FLContextSerializationTest 9, FLDecoderTest 8, FLEncoderDecoderTest 5, FLEncoderTest 16, FLFullBlockClosureSerializationTest 8, FLGlobalEnvironmentTest 12 pass + 1 skip, FLGlobalSendSerializationTest 2, FLMaterializerTest 12, FLSerializerTest 34. FLClassSerializationTest and FLTraitSerializationTest printed no result lines in any run, base or fixed, so they are not compared. I did not test the Latin-1 fallback for old files with bytes 128-255; I did not run the full suite.

Upstream note: Fuel: encode global names as UTF-8 (FLGlobalCluster). FLGlobalCluster wrote class names and global keys with FLEncoder>>encodeString:, one byte per character, so WideString/WideSymbol names lost their high bytes and materialization raised FLGlobalNotFound/FLClassNotFound. Names are now written as UTF-8 (identical bytes for ASCII) and decoded with utf8Decoded, with a Latin-1 fallback for older files. Also fix FLBasicSerializationTest>>testWideStringGlobal (WideString key in SystemEnvironment is rejected; the assertion compared the value to the key; use a WideSymbol key and compare with the global value) and remove the three entries of FLSerializationTest>>expectedFailures. Project: Fuel (pharo-project/fuel, in the image as FL*). Written for the owner to review; no PR opened.

Results

Last full sweeps of 2026-11-17: iospharo (sweep14-new14) and stock Cog with the Cog changes (sweepcogp14-new10), with all image fixes. "Before" is the state of the test before the repairs. The Stock Cog column has a value only where the record measured stock Cog: "PASS 3 of 3 (repair run)" is the check of each repair on stock Cog v12.0.6 with the fix entry; "-" is not measured separately.

TestBeforeStateiospharo VMPatched CogStock Cog
FLBinaryFileStreamBasicSerializationTest>>testConsiderCustomWideSymbolGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLBinaryFileStreamBasicSerializationTest>>testWideStringClassNameXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLBinaryFileStreamBasicSerializationTest>>testWideStringGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLByteArrayBasicSerializationTest>>testConsiderCustomWideSymbolGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLByteArrayBasicSerializationTest>>testWideStringClassNameXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLByteArrayBasicSerializationTest>>testWideStringGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLFileReferenceStreamBasicSerializationTest>>testConsiderCustomWideSymbolGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLFileReferenceStreamBasicSerializationTest>>testWideStringClassNameXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLFileReferenceStreamBasicSerializationTest>>testWideStringGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLFullBasicSerializationTest>>testConsiderCustomWideSymbolGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLFullBasicSerializationTest>>testWideStringClassNameXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLFullBasicSerializationTest>>testWideStringGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLGZippedBasicSerializationTest>>testConsiderCustomWideSymbolGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLGZippedBasicSerializationTest>>testWideStringClassNameXFAILrepairedPASSPASSPASS 3 of 3 (repair run)
FLGZippedBasicSerializationTest>>testWideStringGlobalXFAILrepairedPASSPASSPASS 3 of 3 (repair run)

Test

Run the tests in a Pharo 14 image (build 791). Before the fix they skip, fail or carry the expected-failure mark.

"Run in a Playground of a Pharo 14 image (Do it and print each line)."
((FLBinaryFileStreamBasicSerializationTest selector: #testConsiderCustomWideSymbolGlobal) run) printString.
((FLBinaryFileStreamBasicSerializationTest selector: #testWideStringClassName) run) printString.
((FLBinaryFileStreamBasicSerializationTest selector: #testWideStringGlobal) run) printString.
((FLByteArrayBasicSerializationTest selector: #testConsiderCustomWideSymbolGlobal) run) printString.
((FLByteArrayBasicSerializationTest selector: #testWideStringClassName) run) printString.
((FLByteArrayBasicSerializationTest selector: #testWideStringGlobal) run) printString.
((FLFileReferenceStreamBasicSerializationTest selector: #testConsiderCustomWideSymbolGlobal) run) printString.
((FLFileReferenceStreamBasicSerializationTest selector: #testWideStringClassName) run) printString.
((FLFileReferenceStreamBasicSerializationTest selector: #testWideStringGlobal) run) printString.
((FLFullBasicSerializationTest selector: #testConsiderCustomWideSymbolGlobal) run) printString.
((FLFullBasicSerializationTest selector: #testWideStringClassName) run) printString.
((FLFullBasicSerializationTest selector: #testWideStringGlobal) run) printString.
((FLGZippedBasicSerializationTest selector: #testConsiderCustomWideSymbolGlobal) run) printString.
((FLGZippedBasicSerializationTest selector: #testWideStringClassName) run) printString.
((FLGZippedBasicSerializationTest selector: #testWideStringGlobal) run) printString.

Source of the tests before the fix

FLBasicSerializationTest>>testConsiderCustomWideSymbolGlobal before the fix (Pharo 14 build 791)
testConsiderCustomWideSymbolGlobal
	"The same than #testConsiderCustomGlobal but with a WideSymbol."
	
	| aWideSymbol aPerson |	
	aWideSymbol := (WideString streamContents: [ :stream |
		256 to: 280 do: [ :code | stream nextPut: code asCharacter ] ]) asSymbol.

	self serializer lookUpInGlobals: aWideSymbol.

	aPerson := FLPerson new.	
	self environmentOfTest
		at: aWideSymbol
		put: aPerson.
	addedGlobals add: aWideSymbol.
	
	self
		shouldnt: [ self resultOfSerializeAndMaterialize: aPerson ]
		raise: FLGlobalNotFound
FLBasicSerializationTest>>testWideStringClassName before the fix (Pharo 14 build 791)
testWideStringClassName

	| class className |
	className := 'Foox' asWideString
		             at: 4 put: (Character value: 265);
		             yourself.
	class := self classFactory silentlyMake: [ :builder | builder name: className ].
	self environmentOfTest at: class name put: class.

	self shouldnt: [ self resultOfSerializeAndMaterialize: class ] raise: FLClassNotFound
FLBasicSerializationTest>>testWideStringGlobal before the fix (Pharo 14 build 791)
testWideStringGlobal
	| global globalValue |
	global := 'Foox' asWideString
		at: 4 put: (Character value: 265);
		yourself.
	globalValue := global , 'value'.
	self environmentOfTest
		at: global
		put: globalValue.
	addedGlobals add: global.
	self serializer lookUpInGlobals: global.

	self shouldnt: [ self serialize: 'bar' -> globalValue ] raise: Error.
	self assert: self materialized class equals: Association.
	self assert: self materialized value identicalTo: global

Fix

How to apply an entries file: file in 00-header.st and the entries file as one chunk (the iospharo script scripts/gen-image-fixes.py makes that chunk). Each entry checks for the defect first and prints FIXED, ADDED or SKIP. Download 00-header.st.

20-Fuel-widestring-globals.st (download)
"Fuel-widestring-globals.  Fuel wrote the name of a global (a class name, a
 global key) with encodeString:, which stores one byte per character.  A name
 with a character above 255 (WideString, WideSymbol) lost its high bytes, so
 the materializer looked up a wrong name (FLGlobalNotFound, FLClassNotFound).
 Global names are now written as UTF-8 and read back as UTF-8.  For an ASCII
 name the bytes are the same as before, so old files still load.  A name with
 bytes 128-255 that is not valid UTF-8 (a file from before this change) is
 read as Latin-1, as before."
add value: #FLGlobalCluster value: #encodeGlobalName:on: value: 'serialization'
	value:
'encodeGlobalName: aName on: anEncoder
	"Write the name of a global as UTF-8.  Same bytes as encodeString: for an ASCII name."

	anEncoder encodeString: aName asString utf8Encoded asString'.
add value: #FLGlobalCluster value: #nextGlobalNameFrom: value: 'serialization'
	value:
'nextGlobalNameFrom: aDecoder
	"Read a global name written by #encodeGlobalName:on:.  Bytes that are not valid UTF-8 come from an older file and are Latin-1."

	| string |
	string := aDecoder nextEncodedString.
	^ ([ string asByteArray utf8Decoded ]
		on: Error
		do: [ :error | string ]) asSymbol'.
fix value: #FLGlobalCluster value: #serializeGlobalClass:on:
	value: [ :source | source includesSubstring: 'anEncoder encodeString: aClass name' ]
	value:
'serializeGlobalClass: aClass on: anEncoder
	self encodeGlobalName: aClass name on: anEncoder'.
fix value: #FLGlobalCluster value: #serializeGlobalAssociationKeyed:on:
	value: [ :source | source includesSubstring: 'anEncoder encodeString: aSymbol' ]
	value:
'serializeGlobalAssociationKeyed: aSymbol on: anEncoder

	self encodeGlobalName: aSymbol on: anEncoder'.
fix value: #FLGlobalCluster value: #materializeGlobalAssociationFrom:
	value: [ :source | source includesSubstring: 'nextEncodedString asSymbol' ]
	value:
'materializeGlobalAssociationFrom: aDecoder

	| globalName |
	globalName := self nextGlobalNameFrom: aDecoder.
	^ aDecoder globalEnvironment
		associationAt: globalName
		ifAbsent: [ FLGlobalNotFound signalWithName: globalName ]'.
fix value: #FLGlobalCluster value: #materializeGlobalClassFrom:
	value: [ :source | source includesSubstring: 'nextEncodedString asSymbol' ]
	value:
'materializeGlobalClassFrom: aDecoder

	| className |
	className := self nextGlobalNameFrom: aDecoder.
	^ aDecoder classNamed: className'.
"testWideStringGlobal put a WideString into a SystemEnvironment as key, which
 only accepts Symbols (SystemEnvironment>>at:put:), and then expected the
 materialized VALUE to be the KEY.  The key is now a WideSymbol and the value
 expected is the global value, the object found by that key."
fix value: #FLBasicSerializationTest value: #testWideStringGlobal
	value: [ :source | source includesSubstring: 'yourself.' ]
	value:
'testWideStringGlobal
	| global globalValue |
	global := (''Foox'' asWideString
		at: 4 put: (Character value: 265);
		yourself) asSymbol.
	globalValue := global asString , ''value''.
	self environmentOfTest
		at: global
		put: globalValue.
	addedGlobals add: global.
	self serializer lookUpInGlobals: global.

	self shouldnt: [ self serialize: ''bar'' -> globalValue ] raise: Error.
	self assert: self materialized class equals: Association.
	self assert: self materialized value identicalTo: globalValue'.
"The three tests pass now: no expected failures remain."
fix value: #FLSerializationTest value: #expectedFailures
	value: [ :source | source includesSubstring: 'testWideStringGlobal' ]
	value:
'expectedFailures
	^ #()'.

State of 2026-11-20. Made from the test records of the iospharo project (a second Pharo VM). The fixes are proposals for review.