Difference between Linux and Windows

Foreign Linux uses a dynamic binary translator to run the application binary. This is because it can not run application binaries directly due to some incompatibilities between Linux and Windows. This article describes them.

Segment register handling (x86 and x64)

On x86 architecture, the fs and gs segment register are commonly used to implement thread local storage (TLS) as they provide an additional displacement base for addresses. But their usage between Linux and Windows are different and incompatible.

On x86, GDT or LDT must be set up properly before the use of a segment register. In Linux, upon a call to set_thread_area(), the kernel will set the application's segment descriptors in GDT and maintain these upon context-switch. Glibc then uses gs segment register to access the TLS storage. On Windows, these steps are handled inside the kernel, and applications are automatically provided with a usable fs segment register for TLS access. Although the registers do not collide, it is impossible to set up a custom GDT or LDT entry in user space. Historically Windows provides the SetLdtEntries() call in 32bit versions. But it suffers severe limitations as the segment limit cannot exceed 0x7FFFFFFF.

On x64, segmentation is completely removed. It provides several MSRs for accessing the base address of fs and gs registers. But they can only be accessed in Ring 0. In Linux, the kernel provides arch_prctl() call for applications to read/write the registers. Glibc uses fs register as TLS base. But on Windows, again, the set up is completely done in kernel space. No interface for accessing these segment registers are exposed to user space.

I have also noted some interesting additions to recent CPUs and operating systems related to this case. One is the FSGSBASE extension instructions starting with Intel Sandy Bridge and AMD Steamroller. This allows a Ring 3 code to directly access the fs and gs registers. Another is the User Mode Scheduling added mechanism since Windows 7. This allows pure user mode thread scheduling.

I was once excited to try utilizing these mechanisms, but eventually failed. The Windows kernel does not seem to properly save/restore the fs register upon context switch. AFAIK it just clears fs out for a pure 64bit application. For gs register, if I set it to an arbitrary value, at some point the operating system (or maybe just the VS debugger) will complain about that the register value is wrong, and the application is in a bad condition.

One possible way to mitigate the TLS mechanism is to leave fs and gs register as invalid, and depends on exception to intercept access to TLS storage. This is painfully slow and not always work.

syscall instruction (x64)

The x64 Glibc uses the `syscall' instruction for entry to kernel mode. Unfortunately Windows uses this too. This makes intercepting system calls virtually impossible without explicit application patching.

Redzone (x64)

One big issue for x64 is the redzone calling convention. In x64 Linux, 128-byte stack space beyond the RSP register is considered `redzone' and is guaranteed to not be modified by signal handlers or interrupt handlers. But on Windows, anything beyond RSP is considered volatile. For example, when an exception occurs, the kernel will directly write context information to the stack beyond RSP and delegate execution to application exception handler. This one single mismatch makes running x64 Linux applications directly under Windows impossible.