Decompilation—the process of turning low-level machine code back into human-readable source code—is a vital tool for cybersecurity experts hunting vulnerabilities and analyzing malware. Traditionally, decompilers like Ghidra and Hex-Rays generate code that often includes visible placeholders where the original intent is unclear, resulting in output that might not compile or run correctly. Recently, large language model (LLM)-based decompilers have emerged, producing polished, idiomatic C code that can be recompiled and executed, leading many to judge their success primarily by whether the output builds and passes existing tests.
However, a newly published research paper by Chang Liu, Edward Raff, and Kristopher Micinski reveals that relying on recompilability and passing pre-existing tests can be misleading. Their findings show that even when LLM-based decompiled code compiles and passes all shipped tests, it can still behave differently on other valid inputs or silently remove known vulnerabilities. This discrepancy poses serious risks for security applications that depend on accurate code recovery.
Key Takeaways
- LLM decompilers produce cleaner, re-executable code but can alter program behavior in subtle ways not caught by standard test suites.
- In experiments across multiple systems and configurations, about 4.9% of functions that passed all tests still behaved differently when exposed to a broader range of inputs; in some cases, divergence reached 13%.
- On real-world code from GitHub and known vulnerable functions, improved recompilation rates sometimes coincided with decreased behavioral fidelity—meaning the code looked correct but acted differently.
- Up to 10% of known vulnerabilities disappeared from the decompiled output without any visible indication, potentially masking critical security flaws.
To investigate these issues, the researchers developed a new approach called Decompile-Diverge. Instead of relying on fixed test suites, which only check behavior on a limited set of inputs, Decompile-Diverge automatically generates a wide variety of inputs for each function by synthesizing a test driver and using fuzzing techniques. Fuzzing is a method that feeds random or semi-random data into a program to explore unexpected behaviors or edge cases. By running both the original and the decompiled code on these inputs, the approach detects any divergence in their behavior.
This method revealed that many LLM decompilers, while improving on traditional tools in terms of producing buildable code, sometimes introduce changes such as new data fields, altered types, or additional checks (called guards) that replace the unknown placeholders typical in older tools. These changes can cause the decompiled code to behave differently, even if it looks cleaner and compiles successfully. Crucially, this includes cases where vulnerabilities present in the original code vanish silently from the decompiled version, which could mislead analysts and automated security tools.
The study’s comprehensive evaluation spanned eight systems and nine configurations, using established LLM decompilation benchmarks as well as real-world libraries and functions with documented security vulnerabilities. It showed that improving recompilability alone is not a sufficient measure of decompiler quality, especially for security-critical applications.
Looking ahead, this research highlights the importance of developing more robust evaluation methods for decompilers that go beyond simply checking if the output builds and passes existing tests. Behavioral equivalence—ensuring that the decompiled code acts the same as the original under a wide range of conditions—is essential for trustworthy reverse engineering. Tools like Decompile-Diverge offer promising steps toward this goal by automatically generating diverse inputs and detecting subtle discrepancies.
Ultimately, as LLM-based decompilers become more common, security professionals and developers should be cautious about assuming that clean, compilable code automatically means accurate recovery. Ongoing research and improved testing frameworks will be key to safely leveraging these powerful new technologies in vulnerability analysis and malware research.
Based on research published on arXiv by Chang Liu, Edward Raff, Kristopher Micinski.