Pharo test defects

TaskIt-worker-pool-race

State
Fixed (image fix)
Defect in
the Pharo image
Tests
2
Fix
image fix 25-taskit-worker-pool-race.st
Upstream repository
pharo-project/pharo

Record

privateRemoveIdleWorkerIfNeeded stops one worker that is free. A worker is free from the end of a task until executeTask: records the next task, so a free worker can be in the code that takes a task from the queue, or can hold a task. The pool then terminates a process that holds a task, and the future of the task never gets a value. The entry stops a worker only when its process waits on the semaphore of the task queue, and takes the process out of that wait in the same step, at the highest priority.

Test Defect in State Cause
TKTCommonQueueWorkerPoolTest>>testWorkerPoolDoesNotExceedPoolSizeWhenSchedulingTasksInParallel image-code repaired The pool stops a worker that holds a task (TKTTimeoutException).
TKTFutureTest>>testFutureFirstCompleteOfSuccessIfFirstSuccessFaster image-code and test repaired The same cause in the pool of the test profile, and the order of two callbacks (next group).

Measurements: docs/image_issues.md, section "TaskIt worker pool kills a busy worker". Loops: scripts/cogdiff/taskit_pool_stress.st and scripts/cogdiff/taskit_pool_diag.st.

Upstream note: Title: TaskIt: the common-queue worker pool stops a worker that holds a task. Project: pharo (TaskIt). The change is in the three classes that the entry names.

Results

Last full sweeps of 2026-11-17: iospharo (sweep14-new14) and stock Cog with the Cog changes (sweepcogp14-new10), with all image fixes. "Before" is the state of the test before the repairs. The Stock Cog column has a value only where the record measured stock Cog: "PASS 3 of 3 (repair run)" is the check of each repair on stock Cog v12.0.6 with the fix entry; "-" is not measured separately.

TestBeforeStateiospharo VMPatched CogStock Cog
TKTCommonQueueWorkerPoolTest>>testWorkerPoolDoesNotExceedPoolSizeWhenSchedulingTasksInParallelrepairedPASSPASS-
TKTFutureTest>>testFutureFirstCompleteOfSuccessIfFirstSuccessFasterrepairedPASSPASS-

Test

Run the tests in a Pharo 14 image (build 791). Some of these tests are new: the entries file adds them, so file in the entries file first, then run them.

"Run in a Playground of a Pharo 14 image (Do it and print each line)."
((TKTCommonQueueWorkerPoolTest selector: #testWorkerPoolDoesNotExceedPoolSizeWhenSchedulingTasksInParallel) run) printString.
((TKTFutureTest selector: #testFutureFirstCompleteOfSuccessIfFirstSuccessFaster) run) printString.

Source of the tests before the fix

TKTCommonQueueWorkerPoolTest>>testWorkerPoolDoesNotExceedPoolSizeWhenSchedulingTasksInParallel before the fix (Pharo 14 build 791)
testWorkerPoolDoesNotExceedPoolSizeWhenSchedulingTasksInParallel
	| pool totalTasks futures |
	pool := TKTCommonQueueWorkerPool new.
	pool poolMaxSize: 10.
	pool start.
	totalTasks := 0.
	futures := AtomicSharedQueue new.
	1 to: 10 do: [ :i | futures nextPut: (pool future: [ 10 timesRepeat: [ futures nextPut: (pool future: [ totalTasks := totalTasks + 1 ]) ] ]) ].
	[ futures isEmpty ] whileFalse: [ futures next waitForCompletion: 100 milliSeconds ].
	self assert: totalTasks equals: 100.
	self assert: pool size <= 10
TKTFutureTest>>testFutureFirstCompleteOfSuccessIfFirstSuccessFaster before the fix (Pharo 14 build 791)
testFutureFirstCompleteOfSuccessIfFirstSuccessFaster
	| future got secondFuture error |
	future := TKTFuture new.
	secondFuture := TKTFuture new.
	(future firstCompleteOf: secondFuture) onSuccessDo: [ :v | got := v ].
	future deploySuccess: 5.
	error := Error new.
	[ error signal ]
		on: Error
		do: [ :e | secondFuture deployFailure: e ].
	50 milliSecond wait.
	self assert: got equals: 5

Fix

How to apply an entries file: file in 00-header.st and the entries file as one chunk (the iospharo script scripts/gen-image-fixes.py makes that chunk). Each entry checks for the defect first and prints FIXED, ADDED or SKIP. Download 00-header.st.

25-taskit-worker-pool-race.st (download)
"TaskIt: the worker pool stops a worker that holds a task (2026-11-17).

TKTCommonQueueWorkerPool>>privateRemoveIdleWorkerIfNeeded stops one free
worker.  A worker is free (currentTaskExecution isNil) from the moment when
its task ends to the moment when TKTWorkerProcess>>executeTask: records the
next task.  In that time the worker can be in AtomicSharedQueue>>next, in the
code that takes an item, or can hold the task that it took.  The pool also
makes the list of the free workers some time before it stops one of them, and
the worker can take and start a task in that time.  The pool then terminates
a process that holds a task: the task never runs, and its future never gets a
value.  A process that is terminated in WaitfreeQueue>>nextIfNone: also
leaves the queue in the state that a reader holds the queue.

The processes of a pool have the priority of the process that made them, so a
test and its workers run at one priority.  The image runs with
preemptionYields: each time a process of a higher priority runs (a timer, the
UI cycle), the VM puts the active process behind the other processes of its
priority.  Stock Cog and our VM do the same, and the defect shows on the two.

Tests that fail from this cause, in scripts/cogdiff/taskit_pool_stress.st:
  TKTCommonQueueWorkerPoolTest>>testWorkerPoolDoesNotExceedPoolSizeWhenSchedulingTasksInParallel
      (TKTTimeoutException: 83 of 600 runs on our VM, 2 of 600 on stock Cog)

The repair: the pool stops a worker only when the process of the worker waits
on the semaphore of the task queue, and the pool takes the process out of
that wait in the same step.  A process in that wait is in
AtomicSharedQueue>>next and holds no item.  The test and the suspend run at
the highest priority (valueUnpreemptively), so the worker cannot leave the
wait between the two.  After the suspend, a signal of the queue goes to
another reader, so no wake-up is lost.

Two weaker repairs failed, and the event log of
scripts/cogdiff/taskit_pool_diag.st showed why:
- 'the worker is free' and then 'the process of the worker waits': a worker
  passed that took a task between the two reads, started the task, and then
  waited for the mutex of the pool (4 failures in 3000 runs).
- 'the process waits on the semaphore of the queue', then the stop: correct
  while each signal of the queue comes under the mutex of the pool (0
  failures in 3000 runs on our VM, 0 in 600 on stock Cog).  The repair of the
  queue in 27-atomic-shared-queue-wakeups.st makes a reader give a signal
  too, so the test and the suspend must be one step.

P1. AtomicSharedQueue>>isProcessWaiting:
P2, P3. isWaitingForTask of the worker process and of the worker (a first
test with no cost).
P4, P5. suspendIfWaitingForTask of the worker process and of the worker (the
test and the suspend in one step).
P6. The pool stops a worker only when suspendIfWaitingForTask answers true."

add value: #AtomicSharedQueue value: #isProcessWaiting: value: 'testing'
	value:
'isProcessWaiting: aProcess
	"Answer if aProcess waits in #next for an item.  A process in that state holds no item of the queue."

	^ aProcess suspendingList == availSema'.

add value: #TKTWorkerProcess value: #isWaitingForTask value: 'testing'
	value:
'isWaitingForTask
	"The process of the worker waits in the task queue for a task.  A worker that is free (isFree) can be in the code that takes a task from the queue, or can hold a task that executeTask: did not record yet."

	^ process isNotNil and: [ taskQueue isNotNil and: [ taskQueue isProcessWaiting: process ] ]'.

add value: #TKTWorker value: #isWaitingForTask value: 'testing'
	value:
'isWaitingForTask

	^ process ifNil: [ false ] ifNotNil: [ process isWaitingForTask ]'.

add value: #TKTWorkerProcess value: #suspendIfWaitingForTask value: 'start-stop'
	value:
'suspendIfWaitingForTask
	"If the process of the worker waits in the task queue for a task, take the process out of that wait and answer true.  The process holds no task then, and it cannot get one: a signal of the queue goes to another reader.  The test and the suspend run at the highest priority, so the worker cannot leave the wait between the two."

	self isWaitingForTask ifFalse: [ ^ false ].
	^ [ (taskQueue isProcessWaiting: process)
		    ifTrue: [ process suspend. true ]
		    ifFalse: [ false ] ] valueUnpreemptively'.

add value: #TKTWorker value: #suspendIfWaitingForTask value: 'starting'
	value:
'suspendIfWaitingForTask

	^ process ifNil: [ false ] ifNotNil: [ process suspendIfWaitingForTask ]'.

fix value: #TKTCommonQueueWorkerPool value: #privateRemoveIdleWorkerIfNeeded
	value: [ :source | (source includesSubstring: 'suspendIfWaitingForTask') not ]
	value:
'privateRemoveIdleWorkerIfNeeded
	"Stop one worker that waits for a task.  A free worker can hold a task that it did not record yet, and can start a task after the list of the free workers is made: the stop of such a worker loses the task.  suspendIfWaitingForTask takes a worker out of the task queue only when the worker waits there, and such a worker holds no task."

	| removing free |
	free := self privateFreeWorkers.
	free size < taskQueue size
		ifTrue: [ ^ self ].
	removing := workers detect: [ :each | each suspendIfWaitingForTask ] ifNone: [ ^ self ].
	workers remove: removing.
	removing stop'.

State of 2026-11-20. Made from the test records of the iospharo project (a second Pharo VM). The fixes are proposals for review.