4 Evaluation

In this Section, we evaluate three aspects of Sem86. First, we construct a toy malware sample and confirm Sem86 is able to identify where this sample uses undefined behavior to detect whether it is running on an emulated CPU. Second, we boot several real-world operating systems to evaluate hardware support. Finally, we evaluate the performance of Sem86 by comparing it with QEMU and Bochs.

4.1 CPU-specific semantics

CPU-specific behavior has not been well-researched. To the best of our knowledge, there exists no comprehensive overview of differences in undefined behavior, nor analysis of real-world usage of such differences for emulator detection. Because of this, it is difficult to find real malware samples (ab)using CPU-specific behavior.

Instead, we evaluate Sem86′s ability to use different semantics by constructing a toy malware sample that relies on CPU-specific behavior. Our toy malware sample determines whether it is running inside an emulator by executing the IMUL instruction, and checking the value that is stored in the zero flag (ZF) afterwards. The source code of our toy malware sample is shown in Figure 3.

int APIENTRY WinMain(...) 
    int result = 0;
    int a = 15;
    int b = 0;

    __asm {
        xor result, 1
        mov eax, a
        mov ecx, b
        imul eax, ecx
        jnz done
        mov result, 0
done:
    };

    if (result) {
        MessageBox(NULL, "Deleting C drive", "", 0);
    } else {
        MessageBox(NULL, "Analysis attempt detected.", "", 0);
    }
    
    return 0;
}

The ZF is undefined for the IMUL instruction, and is often implemented differently both in actual CPUs as well as emulators. Bochs currently implements the ZF by setting it to 1 if the result was zero, and setting it to 0 otherwise. Although modern CPUs implement various different behaviors, for the purposes of this experiment we assume that a real CPU would not modify ZF, which appears to have been common at the time when this technique was detected [5].

We generated two different semantics: one where IMUL sets the ZF to the normal result, which behaves like Bochs, and one where IMUL does not modify the ZF. Starting Sem86 with the semantics where IMUL behaves like Bochs, shows the “Analysis attempt detected” message, while starting it with the semantics where IMUL does not behave like Bochs correctly shows the “Deleting C drive” message.

In order to evaluate bisection, we set the toy malware sample to automatically start upon boot, and then ran a bisection on the display output of the emulator after 400 million executed instructions. For real malware, other characteristics such as file system contents or network access attempts could be used instead of the display output. The bisection was able to automatically identify the IMUL instruction as being the instruction that determines which message box is shown.

4.2 Booting real-world operating systems

Sem86 implements all hardware required to boot x86 operating systems. We evaluated this by booting Windows 98, Windows XP and Windows 7. We were able to boot all of these operating systems, as depicted in Figure 4.

The ES1370 sound card and the NE2000 networking card cannot be used under Windows 7. While these work on Windows 98 and XP, no drivers are available for Windows 7. In order to enable sound and networking for Windows 7 and newer, more modern hardware needs to be implemented.

To demonstrate that Sem86′s implementation is portable, we have ported it to Android. We tested this on an Android phone that has 8 GiB of RAM. While it is fully functional, it can sometimes, crash due to the memory usage of LLVM, which can reach 4-5 GiB during JIT compilation of some instruction sequences.

4.3 Performance

We have evaluated the performance of Sem86 using CPUMark’99 running on Windows XP. This is a single-threaded benchmark that evaluates integer performance of CPUs, which was commonly used to evaluate CPU performance in the 90s. After completing the benchmark, it displays a single score that summarizes CPU performance. The results are presented in Table 2. We also list how many million instructions were executed per second (MIPS) during the benchmark.

We compare against QEMU and Bochs, because they are at opposite ends of the optimization spectrum: Bochs is purely an interpreter, and does not use any JIT optimization techniques for portability reasons. On the other hand, QEMU uses bespoke JIT code generation to translate guest instruction traces into native code, and has been heavily optimized for performance.

We believe the difference in performance between Sem86 and QEMU is mostly due to JIT code generation quality. QEMU uses a bespoke JIT, while we rely on off-the-shelf LLVM. LLVM is a general-purpose compiler library, which is not geared specifically to emulators. The code generated by LLVM is likely of lower quality than that generated by QEMU. For example, because we do not have full control over code generation, LLVM sometimes generates larger function preludes that preserve more registers on the stack. While this would likely benefit normal programs, it degrades performance when functions are called hundreds of millions of times per second.

Additionally, QEMU performs basic block linking, where jumps to the next basic block are inlined directly into the block. This avoids the overhead of returning to the execution loop, and looking up the next basic block in the cache data structures. While Sem86 also does this for basic blocks within the same page, we have not implemented this for jumps between different pages.