What does SuspendThread really do?

Windows Research Kernel @ HPI

By Martin von Löwis

Recently, I asked myself (and all people around me): When you invoke SuspendCall, will it really suspend the thread immediately, or might it continue to run for some time before it gets suspended? (The same question can be asked for TerminateThread, and, as we will see, it has a similar answer)

In the case of a single-processor system, it is quite plausible that SuspendThread suspends the thread right away: the thread-to-be-suspended is clearly not running at this point. In the multi-processor case, things might be more difficult, as the thread might be running on a different processor - then, how long might it take to suspend it, and: when SuspendThread returns, will it already be suspended?

Let's look at the code: NtSuspendThread (in psspnd.c) first checks whether the thread to be suspended is the current thread, and then calls KeSuspendThread immediately. If it is a remote thread, it first acquires Thread->RundownProtect, to prevent the thread from being deleted while it is operated on. KiSuspendThread (in thredobj.c) checks Thread->SuspendCount to determine whether more than MAXIMUM_SUSPEND_COUNT suspend calls have been issued, and gives up in this case (STATUS_SUSPEND_COUNT_EXCEEDED). Otherwise, it suspends the thread with the code

02179     //
02180     // Don't suspend the thread if APC queuing is disabled. In this case the
02181     // thread is being deleted.
02182     //
02183
02184     if (Thread->ApcQueueable == TRUE) {
02185
02186         //
02187         // Increment the suspend count. If the thread was not previously
02188         // suspended, then queue the thread's suspend APC.
02189         //
02190         // N.B. The APC MUST be queued using the internal interface so
02191         //      the system argument fields of the APC do not get written.
02192         //
02193
02194         Thread->SuspendCount += 1;
02195         if ((OldCount == 0) && (Thread->FreezeCount == 0)) {
02196             if (Thread->SuspendApc.Inserted == TRUE) {
02197                 KiLockDispatcherDatabaseAtSynchLevel();
02198                 Thread->SuspendSemaphore.Header.SignalState -= 1;
02199                 KiUnlockDispatcherDatabaseFromSynchLevel();
02200
02201             } else {
02202                 Thread->SuspendApc.Inserted = TRUE;
02203                 KiInsertQueueApc(&Thread->SuspendApc, RESUME_INCREMENT);
02204             }
02205         }
02206     }

With that done, it just returns (first releasing the dispatcher database which it had
acquired on entry).

The surprising thing here is that SuspendThread just schedules an
Asynchronous Procedure Call (APC), instead of doing any real work. Normally, one would expect that the APC gets considered the next time the system dispatches the thread; in the case of a thread running on a different processor, that would be when the quantum of the thread ends.

But let's look further. What is the value of Thread-SuspendApc? It is initialized in KeInitThread, with the code

00178     //
00179     // Initialize the kernel mode suspend APC and the suspend semaphore object.
00180     // and the builtin wait timeout timer object.
00181     //
00182
00183     KeInitializeApc(&Thread->SuspendApc,
00184                     Thread,
00185                     OriginalApcEnvironment,
00186                     (PKKERNEL_ROUTINE)KiSuspendNop,
00187                     (PKRUNDOWN_ROUTINE)KiSuspendRundown,
00188                     KiSuspendThread,
00189                     KernelMode,
00190                     NULL);
00191
00192     KeInitializeSemaphore(&Thread->SuspendSemaphore, 0L, 2L);

So this registers three functions for the APC: the kernel routine KiSuspendNop (which does nothing), KiSuspendRundown (which clears the APC should the thread be terminated before the APC runs), KiSuspendThread, which does the actual suspension. This is defined as

01667 {
01668
01669     PKTHREAD Thread;
01670
01671     UNREFERENCED_PARAMETER(NormalContext);
01672     UNREFERENCED_PARAMETER(SystemArgument1);
01673     UNREFERENCED_PARAMETER(SystemArgument2);
01674
01675     //
01676     // Get the address of the current thread object and Wait nonalertable on
01677     // the thread's builtin suspend semaphore.
01678     //
01679
01680     Thread = KeGetCurrentThread();
01681     KeWaitForSingleObject(&Thread->SuspendSemaphore,
01682                           Suspended,
01683                           KernelMode,
01684                           FALSE,
01685                           NULL);
01686
01687     return;
01688 }

So this essentially just blocks on the thread's suspend semaphore. To support multiple interleaving calls to SuspendThread, a counter is provided, so that the first suspender will schedule the APC, and the last resumer will signal the semaphore.

Still, the question is: will this happen immediately, or only when APCs get processed? KiInsertQueueApc first does what one would expect: insert the APC into the thread's APC queue. However, then it goes on with code that looks like it tries to run the APC immediately. That starts with

00489     // If the APC index from the APC object matches the APC Index of
00490     // the thread, then check to determine if the APC should interrupt
00491     // thread execution or sequence the thread out of a wait state.
00492     //
00493
00494     if (Apc->ApcStateIndex == Thread->ApcStateIndex) {

What is the "APC state index", and what values will it have? I don't really know; for the APC, it will be OriginalApcEnvironment. This should also be the ApcStateIndex of the target thread (so the condition should be true normally), but I fail to understand the concept of APC environments (and couldn't find any reasonable explanation anywhere).

In any case, assuming the condition is true, it then goes on testing whether we schedule an APC for the current thread, and run an APC_LEVEL software interrupt right away if it is. Otherwise (i.e. scheduling a remote APC), we lock the dispatcher database, start with RequestInterrupt=FALSE, and do this complex piece of code

00531         if (ApcMode == KernelMode) {
00532
00533             //
00534             // Thread transitions from the standby state to the running
00535             // state can occur from the idle thread without holding the
00536             // dispatcher lock. Reading the thread state after setting
00537             // the kernel APC pending flag prevents the code from not
00538             // delivering the APC interrupt in this case.
00539             //
00540             // N.B. Transitions from gate wait to running are synchronized
00541             //      using the thread lock. Transitions from running to gate
00542             //      wait are synchronized using the APC queue lock.
00543             //
00544             // N.B. If the target thread is found to be in the running state,
00545             //      then the APC interrupt request can be safely deferred to
00546             //      after the dispatcher lock is released even if the thread
00547             //      were to be switched to another processor, i.e., the APC
00548             //      would be delivered by the context switch code.
00549             //
00550
00551             Thread->ApcState.KernelApcPending = TRUE;
00552             KeMemoryBarrier();
00553             ThreadState = Thread->State;
00554             if (ThreadState == Running) {
00555                 RequestInterrupt = TRUE;
00556
00557             } else if ((ThreadState == Waiting) &&
00558                        (Thread->WaitIrql == 0) &&
00559                        (Thread->SpecialApcDisable == 0) &&
00560                        ((Apc->NormalRoutine == NULL) ||
00561                         ((Thread->KernelApcDisable == 0) &&
00562                          (Thread->ApcState.KernelApcInProgress == FALSE)))) {
00563
00564                 KiUnwaitThread(Thread, STATUS_KERNEL_APC, Increment);
00565
00566             } else if (Thread->State == GateWait) {
00567                 KiAcquireThreadLock(Thread);
00568                 if ((Thread->State == GateWait) &&
00569                     (Thread->WaitIrql == 0) &&
00570                     (Thread->SpecialApcDisable == 0) &&
00571                     ((Apc->NormalRoutine == NULL) ||
00572                      ((Thread->KernelApcDisable == 0) &&
00573                       (Thread->ApcState.KernelApcInProgress == FALSE)))) {
00574
00575                     GateObject = Thread->GateObject;
00576                     KiAcquireKobjectLock(GateObject);
00577                     RemoveEntryList(&Thread->WaitBlock[0].WaitListEntry);
00578                     KiReleaseKobjectLock(GateObject);
00579                     if ((Queue = Thread->Queue) != NULL) {
00580                         Queue->CurrentCount += 1;
00581                     }
00582
00583                     Thread->WaitStatus = STATUS_KERNEL_APC;
00584                     KiInsertDeferredReadyList(Thread);
00585                 }
00586
00587                 KiReleaseThreadLock(Thread);
00588             }
00589
00590         } else if ((Thread->State == Waiting) &&
00591                   (Thread->WaitMode == UserMode) &&
00592                   (Thread->Alertable || Thread->ApcState.UserApcPending)) {
00593
00594             Thread->ApcState.UserApcPending = TRUE;
00595             KiUnwaitThread(Thread, STATUS_USER_APC, Increment);
00596         }

So multiple conditions get evaluated; let's see which one are true:

  • ApcMode == Kernel? Yes, the suspend APC was created as a kernel APC
  • ThreadState == Running? Might be, if the thread runs on a different processor. We set RequestInterrupt to true in this case.
  • ThreadState == Waiting? Might be if the thread-to-be-suspended is blocked in a system call.
  • ThreadState == GateWait? No clue what this is.
  • Thread->SpecialApcDisable==0? It looks like this should normally be the case, i.e. threads normally allow such APCs to happen.
  • Apc->NormalRoutine == NULL? No, we do have a NormalRoutine (KiSuspendThread)
  • the other conditions (WaitIrql, KernelApcDisable, KernelApcInProgress)? Might evaluate to false if the target thread is doing something critical in kernel mode right now.

So if the target thread is in kernel mode, it may continue to do its stuff for a moment; if it is really just blocked, it will awake to run its APCs.

The code then releases the dispatcher lock, and runs

00604         if (RequestInterrupt == TRUE) {
00605             KiRequestApcInterrupt(Thread->NextProcessor);
00606         }

KiRequestApcInterrupt then checks whether it is the same processor (which it can't be in our case), and send an APC_LEVEL inter-processor interrupt to the processor running the thread to be suspended (through KiIpiSend, which invokes the HAL). This will cause the remote processor to enter kernel mode, and find that APCs should be run for the target thread.

So, in summary:

  • If the target thread is blocked in kernel mode, it may continue to do its kernel activity for a while. Before returning in user mode, the APC queue will be processed and thus the thread suspended.
  • If the target thread is running on a different processor, an IPI will interrupt it, and it becomes suspended.

In either case, the SuspendThread call itself will return before the target thread is actually suspended - either because the thread just gets "unwaited", and still needs to process its APC queue, or because just the IPI has been issued, but the end of the interrupt processing is not waited for.

As I indicated at the beginning: TerminateThread works quite similarly; except that different APC routines are used.

January 19, 2009 | Filed Under Uncategorized

Comments

Comments are closed.