1b422914d5  [JSC] Move FTL stack overflow check to prologue

The bug at a glance

A JIT soundness/robustness fix: the FTL stack-overflow check ran as a B3 patchpoint in the function body after the prologue rather than in the Air prologue before frame allocation, and the patchpoint path did not correctly restore the stack pointer. Getting the stack-overflow check ordering or the sp/callee-save restore wrong on the overflow path is the kind of defect that can leave the stack in an inconsistent state at the moment an exception is thrown deep in recursion. Medium/0.6: the change is structural and the commit notes a real sp-restore bug, but there is no demonstrated corruption primitive and no test (too slow to trigger).

FTL emitted the soft-stack-limit check as a PatchpointValue in the body, after emitFunctionPrologue, and its overflow late-path had to manually emitRestore the callee-saves before jumping to the throw-stack-overflow thunk. That design was fragile (the FIXME in the old code says the check belongs in the Air prologue) and the patchpoint version mis-restored sp. The fix moves the check into a custom Air prologue that runs before frame allocation, when SP == FP and no callee-saves are saved yet.

Root cause

An FTL-compiled function must verify it has enough stack before it commits to a frame, otherwise deep recursion or a large frame can run the native stack past its limit. Previously this check was implemented in LowerDFGToB3::lower() as a B3 PatchpointValue (stackOverflowHandler) placed in the function body. Because a patchpoint runs after the function prologue has already executed, emitFunctionPrologue() had pushed the frame-pointer linkage and the code was positioned amid register allocation; the patchpoint had to appendSomeRegister the call frame and vm, reserve a scratch, and clobber the macro-clobbered GPRs. On the overflow branch it added a late path that, before jumping to CommonJITThunkID::ThrowStackOverflowAtPrologue, had to jit.emitRestore(params.proc().calleeSaveRegisterAtOffsetList()) – an explicit FIXME (bug 172456) noted this would be unnecessary if the check were part of the Air prologue, since then callee-saves could not yet have been clobbered.

The old patchpoint scheme is the one the commit says was buggy: it did not correctly restore the stack pointer on the overflow path. Throwing the stack-overflow exception with sp not pointing where the unwinder/thunk expects is a stack-state corruption at the exact moment control transfers to the throw machinery.

The fix installs a custom prologue generator (mainPrologueGenerator) via m_proc.code().setPrologueForEntrypoint(0, …). The generator runs emitFunctionPrologue(), then, at a point where SP == FP and no callee-saves have been saved, computes maxFrameSize = max(exitFrameSize, ftlFrameSize) where exitFrameSize = m_graph.requiredRegisterCountForExit() * sizeof(Register) and ftlFrameSize = code.frameSize(). It asserts the CodeBlock/JITType with jitAssertCodeBlockOnCallFrameWithType, computes scratch = fp - maxFrameSize, and branches to the ThrowStackOverflowAtPrologue thunk when addressOfSoftStackLimit() > scratch. Only after the check passes does it allocate the frame (subPtr ftlFrameSize from the stack pointer) and then emitSave(code.calleeSaveRegisterAtOffsetList()).

Because the check now precedes both frame allocation and the callee-save store, the overflow path no longer needs to undo any of that: nothing has been pushed, sp still equals fp, and the jump to the throw thunk happens from a clean state. This removes the fragile emitRestore late-path and the sp-restore bug, and it simplifies the overflow logic exactly as the old FIXME anticipated.

Key code

FTLLowerDFGToB3.cpp: stack-overflow check in the Air prologue before frame allocation

                [=](CCallHelpers& jit, B3::Air::Code& code) {
                    jit.emitFunctionPrologue();

                    AllowMacroScratchRegisterUsage allowScratch(jit);

                    // Stack overflow check before frame allocation.
                    // At this point SP == FP; no callee-saves have been saved.
                    const unsigned ftlFrameSize = code.frameSize();
                    const unsigned maxFrameSize = std::max(exitFrameSize, ftlFrameSize);

                    GPRReg scratch = CCallHelpers::selectScratchGPR(GPRInfo::callFrameRegister);

                    jit.jitAssertCodeBlockOnCallFrameWithType(scratch, JITType::FTLJIT);

                    jit.addPtr(CCallHelpers::TrustedImm32(-maxFrameSize), GPRInfo::callFrameRegister, scratch);
                    auto stackOverflow = jit.branchPtr(CCallHelpers::GreaterThan, CCallHelpers::AbsoluteAddress(vm->addressOfSoftStackLimit()), scratch);
                    stackOverflow.linkThunk(CodeLocationLabel(vm->getCTIStub(CommonJITThunkID::ThrowStackOverflowAtPrologue).retaggedCode<NoPtrTag>()), &jit);

                    if (ftlFrameSize)
                        jit.subPtr(CCallHelpers::TrustedImm32(ftlFrameSize), CCallHelpers::stackPointerRegister);

                    jit.emitSave(code.calleeSaveRegisterAtOffsetList());
                });

Patch walkthrough

• Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp — Two coordinated edits in LowerDFGToB3::lower(). Added: a custom Air prologue (mainPrologueGenerator) registered with m_proc.code().setPrologueForEntrypoint(0, …). It calls emitFunctionPrologue(), then – with SP == FP and no callee-saves saved – performs the soft-stack-limit check against fp - max(exitFrameSize, ftlFrameSize), jumping to ThrowStackOverflowAtPrologue on overflow; only then subtracts ftlFrameSize to allocate the frame and emitSave()s the callee-saves. Removed: the old in-body PatchpointValue stackOverflowHandler that ran after the prologue, appended the call frame/vm registers, and used a late path calling emitRestore(calleeSaveRegisterAtOffsetList()) before jumping to the same thunk – the path that carried the sp-restore bug and the FIXME referencing bug 172456.

Background

FTL prologue / Air PrologueGenerator — FTL lowers DFG IR to B3 then to Air. code().setPrologueForEntrypoint() installs a generator that emits the function’s entry sequence: emitFunctionPrologue() (frame-pointer linkage), frame allocation (subtracting the frame size from sp), and saving callee-save registers.

Soft stack limit — VM::addressOfSoftStackLimit() holds a limit the JIT compares the prospective new stack top against; if the frame would cross it, control jumps to the ThrowStackOverflowAtPrologue thunk to raise a RangeError rather than run off the real stack.

B3 PatchpointValue vs Air prologue — A patchpoint injects hand-written assembly into the optimized body, running after the prologue amid allocated registers/frame; an Air prologue runs at entry before frame allocation. The old check was a body patchpoint, so its overflow path had to restore callee-saves/sp manually.

requiredRegisterCountForExit / exitFrameSize — OSR exit may need a larger frame than the FTL body; the check uses max(exitFrameSize, ftlFrameSize) so the reserved stack accommodates the worst case before the frame is committed.

Vulnerability window

1. Old design — FTL emitted the stack-overflow check as an in-body B3 patchpoint after emitFunctionPrologue, with a late path that emitRestore’d callee-saves before jumping to the throw thunk (FIXME bug 172456 noted this was avoidable).
2. Latent bug — The patchpoint version did not correctly restore the stack pointer on the overflow path, leaving stack state inconsistent when the stack-overflow exception was thrown.
3. Redesign — Move the check into a custom Air prologue that runs before frame allocation and before callee-saves are stored, when SP == FP.
4. Report — Tracked as webkit.org/b/172456, rdar://172371127; reviewed by Yusuke Suzuki.
5. Fix — setPrologueForEntrypoint installs the prologue check; the fragile patchpoint and its sp/callee-save restore path are deleted (canonical 314187@main).

Exploitation

1. Reach the check — Deeply recurse into an FTL-compiled function whose frame plus OSR-exit reservation crosses the soft stack limit, forcing the stack-overflow branch.
2. Inconsistent stack on throw — With the old patchpoint the sp was not correctly restored when jumping to the throw thunk, so the exception unwinds from a stack state the unwinder does not expect. Whether that inconsistency is controllable enough to corrupt memory is not demonstrated. INFERRED.
3. Post-fix — The prologue check throws before any frame allocation or callee-save store, so no restore is needed and the fragile window is closed.

Detection & hunting

For defenders and SOC / detection engineers:

• Stack-overflow crashes in FTL prologue/exit —
• jitAssert / register-state asserts on overflow —

Audit directions

• Other JIT stack-overflow checks —
• Custom prologue generators —
• OSR-exit frame sizing —