What is LLVM?
The LLVM project started in 2000 and gained popularity in the early 2010s. The abbreviation means “low level virtual machine” (although currently it doesn’t reflect the essence of the project). LLVM is an open-source framework used to create compilers. On the basis of LLVM, you can build a compiler for your own programming language or ‘improve’ an existing one.
In terms of architecture, LLVM is divided into three parts: frontend, optimization, and backend. The frontend converts the source code into an intermediate representation: universal intermediate code that is used at subsequent stages to optimize the code and build the file. The optimization removes unused code, transforms arithmetic calculations into ready-made constants, and replaces inefficient constructs with faster ones. The backend processes IR and generates machine code.
If you are developing a new language, all you have to do is write a frontend that generates IR — and LLVM will perform the rest of the work. In the same way, you can work with code (originally written in any supported language) at the IR level and adjust it to suit your needs at the optimization stage. Alternatively, you can write your own backend to compile old code for an unknown processor architecture.
You don’t necessarily have to write a frontend from scratch — it might be sufficient to modify the LLVM source code. Optimization is a more flexible tool; it supports loadable plugins that add passes: special functions that process IR at the module, function, loop, or basic block level. A bit later, I will show how to use them for code obfuscation.
Building LLVM for Windows
First of all, let’s install CMake and place the ninja.exe utility to C:\ so that it’s visible to CMake during the building.
mkdir C:\LLVM && cd C:\LLVM
“C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat”
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build && cd build
cmake -G “Ninja” ^
-DLLVM_ENABLE_PROJECTS=”clang” ^
-DLLVM_ENABLE_ASSERTIONS=ON ^
-DCMAKE_BUILD_TYPE=Release ^
-DLLVM_BUILD_LLVM_DYLIB=ON ^
-DLLVM_ENABLE_RTTI=ON ^
-DCMAKE_INSTALL_PREFIX=C:/llvm/custom ^
../llvm
ninja
ninja install
I create a folder in the root directory of the disk and run vcvars64.. It will set up environment variables required for the compilation (e.g. path to the compiler and MSVC include files). Next, I clone the LLVM source code and run the build. It takes about an hour, and new files appear in C:\. They include utilities, compilers, and .lib files. These files will be required when you start creating your own optimization passes.
Building passes
Create a project in Visual Studio and configure its settings as follows:
Configuration Type = Dynamic Library (.dll)
Additional Include Directories = C:\LLVM\include
Additional Library Directories = C:\LLVM\lib
C++ Langauge Standart = ISO C++17 Standard (/std:c++17)
Code Generation → RuntimeLibrary = /MT
Add the .lib list to Additional :
LLVMCore.lib
LLVMSupport.lib
LLVMBitReader.lib
LLVMIRReader.lib
LLVMAnalysis.lib
LLVMPasses.lib
LLVMFrontendOpenMP.lib
LLVMTargetParser.lib
LLVMRemarks.lib
LLVMProfileData.lib
LLVMBinaryFormat.lib
LLVMDemangle.lib
LLVMBitstreamReader.lib
This is how all your passes are built.
Coverage analysis pass
Let’s create a simple plugin for the optimizer. The first thing that comes to mind is to add logging to the input of each function. AFL uses a similar approach to determine coverage: it adds its code to the analyzed source code to track the control flow.
#pragma warning(disable : 4146)#pragma comment(linker, "/export:llvmGetPassPluginInfo")#define _CRT_SECURE_NO_WARNINGS#include "llvm/Passes/PassBuilder.h"#include "llvm/Passes/PassPlugin.h"#include "llvm/IR/PassManager.h"#include "llvm/Pass.h"#include "llvm/IR/Function.h"#include "llvm/IR/Module.h"#include "llvm/IR/IRBuilder.h"#include "llvm/IR/Type.h"#include "llvm/IR/Instructions.h"#include "llvm/IR/Constants.h"#include "llvm/IR/DerivedTypes.h"#include "llvm/Support/raw_ostream.h"using namespace llvm;struct DebugTracePass : PassInfoMixin<DebugTracePass> { PreservedAnalyses run(Module& M, ModuleAnalysisManager&) { LLVMContext& Ctx = M.getContext(); // Get i8* type (char*) Type* i8Ty = Type::getInt8Ty(Ctx); PointerType* i8PtrTy = PointerType::get(i8Ty, 0); // Function type: void OutputDebugStringA(char*) FunctionType* debugFnTy = FunctionType::get(Type::getVoidTy(Ctx), { i8PtrTy }, false); FunctionCallee debugFn = M.getOrInsertFunction("OutputDebugStringA", debugFnTy); for (Function& F : M) { // Skip if this is function declaration if (F.isDeclaration()) continue; // Build IR before first instruction Instruction* insertPt = &*F.getEntryBlock().getFirstInsertionPt(); IRBuilder<> builder(insertPt); // Form string std::string msg = "Enter: " + F.getName().str(); Value* msgStr = builder.CreateGlobalString(msg); // Insert call to OutputDebugStringA builder.CreateCall(debugFn, msgStr); } return PreservedAnalyses::none(); }};extern "C"llvm::PassPluginLibraryInfo llvmGetPassPluginInfo() { return { LLVM_PLUGIN_API_VERSION, // Version of plugin API "InjectFunctionCallPass", // Plugin name "v0.1", // Version [](llvm::PassBuilder& PB) { PB.registerPipelineParsingCallback( [](llvm::StringRef Name, llvm::ModulePassManager& MPM, llvm::ArrayRef<llvm::PassBuilder::PipelineElement>) { if (Name == "debug-trace") { MPM.addPass(DebugTracePass()); return true; } return false; }); } };}First, I disable the annoying warning about invalid typing inside LLVM code. Then I ask to export the llvmGetPassPluginInfo function. It should be in the plugin export to provide its name when it’s loading and add a new debug-trace pass to the list of available passes. This pass is located in the run method of the DebugTracePass class.
Inside the pass, I get a reference to the OutputDebugStringA function (its prototype has to be specified beforehand). Next, I iterate through all functions inside the module that have a body (not just a definition). The call to getEntryBlock returns the first basic block in the function body; while getFirstInsertionPt returns the iterator. Then I convert it into a reference to the first instruction. Next, an IRBuilder object is created; it will generate instructions at a given point (i.e. before the first instruction). I form a global string with the function name; it will be located in the data section. After that, I create a call to the OutputDebugStringA function whose argument is the string I just defined. At the end, PreservedAnalyses explicitly tells LLVM that the module code has changed and the old module analysis is no longer relevant.
Running the pass
For test purposes, let’s create some simple source code:
#include <Windows.h>int main(){ return 0;}And build intermediate code from it:
clang.exe -S -emit-llvm test.cpp -o test.ll
Time to apply the new pass:
opt.exe -load-pass-plugin llvm_pass.dll -passes=debug-trace -S test.ll -o output.ll
The intermediate code created after the pass looks as follows:
@0 = private unnamed_addr constant [12 x i8] c"Enter: main\00", align 1
; Function Attrs: mustprogress noinline norecurse nounwind optnone uwtable
define dso_local noundef i32 @main() #0 {
entry:
call void @OutputDebugStringA(ptr @0)
%retval = alloca i32, align 4
store i32 0, ptr %retval, align 4
ret i32 0
}
declare void @OutputDebugStringA(ptr)
As expected, a call to OutputDebugStringA appeared at the beginning of the main function. Let’s try to compile this IR into an executable file.
clang.exe output.ll -o output.exe
I execute the resulting EXE and see the [ string in DbgView.
Pass to hide string
Now let’s examine a more interesting example. Let’s say you want to prevent detection of your file by string constants. To ensure this, you have to encrypt all strings at the compilation stage and decrypt them into a local buffer when the file is in use. Any symmetric algorithm will suite for encryption, but, for simplicity, let’s use the standard XOR with a constant.
class EncryptStringsPass : public PassInfoMixin<EncryptStringsPass> { LLVMContext* Ctx; Type* i8Ty; PointerType* i8PtrTy; Type* i32Ty; FunctionCallee DecryptFunc; std::string encryptString(const std::string& Original) { std::string Encrypted = Original; for (char& c : Encrypted) c ^= 0xAA; return Encrypted; } void createDecryptionBuffer(IRBuilder<>& B, Value* TargetPtr, unsigned Len, Value*& OutBufPtr) { auto* Buf = B.CreateAlloca(ArrayType::get(i8Ty, Len + 1)); auto* BufCast = B.CreatePointerCast(Buf, i8PtrTy); auto* PtrCast = B.CreatePointerCast(TargetPtr, i8PtrTy); B.CreateCall(DecryptFunc, { BufCast, PtrCast, B.getInt32(Len) }); OutBufPtr = BufCast; } void handleConstantExprUser(ConstantExpr* CE, const std::string& Original) { std::vector<User*> CEUsers(CE->users().begin(), CE->users().end()); for (User* Use : CEUsers) { if (auto* I = dyn_cast<Instruction>(Use)) { IRBuilder<> B(I); Value* BufPtr = nullptr; createDecryptionBuffer(B, CE, Original.size(), BufPtr); I->replaceUsesOfWith(CE, BufPtr); } } } void handleRegularUser(Value* Ptr, Instruction* InsertPt, const std::string& Original) { IRBuilder<> B(InsertPt); Value* BufPtr = nullptr; createDecryptionBuffer(B, Ptr, Original.size(), BufPtr); Ptr->replaceUsesWithIf(BufPtr, [&](Use& U) { return dyn_cast<Instruction>(U.getUser()) == InsertPt; }); }public: PreservedAnalyses run(Module& M, ModuleAnalysisManager&) { Ctx = &M.getContext(); i8Ty = Type::getInt8Ty(*Ctx); i8PtrTy = PointerType::get(i8Ty, 0); i32Ty = Type::getInt32Ty(*Ctx); DecryptFunc = M.getOrInsertFunction( "decryptStringInto", FunctionType::get(Type::getVoidTy(*Ctx), { i8PtrTy, i8PtrTy, i32Ty }, false) ); for (auto& GV : M.globals()) { if (!GV.hasInitializer() || !GV.isConstant()) continue; auto* CA = dyn_cast<ConstantDataArray>(GV.getInitializer()); if (!CA || !CA->isString()) continue; std::string Original = CA->getAsString().str(); std::string Encrypted = encryptString(Original); auto* ArrayTy = cast<ArrayType>(GV.getValueType()); if (Encrypted.size() < ArrayTy->getNumElements()) Encrypted.resize(ArrayTy->getNumElements(), '\0'); std::vector<uint8_t> Bytes(Encrypted.begin(), Encrypted.end()); GV.setInitializer(ConstantDataArray::get(*Ctx, Bytes)); GV.setConstant(false); std::vector<User*> Users(GV.users().begin(), GV.users().end()); for (User* U : Users) { if (auto* CE = dyn_cast<ConstantExpr>(U)) { if (CE->getOpcode() == Instruction::GetElementPtr) { handleConstantExprUser(CE, Original); } continue; } Instruction* InsertPt = nullptr; Value* Ptr = nullptr; if (auto* GEP = dyn_cast<GetElementPtrInst>(U)) { Ptr = GEP; if (!GEP->user_empty()) InsertPt = dyn_cast<Instruction>(*GEP->user_begin()); } else if (auto* I = dyn_cast<Instruction>(U)) { Ptr = &GV; InsertPt = I; } if (Ptr && InsertPt) handleRegularUser(Ptr, InsertPt, Original); } } return PreservedAnalyses::none(); }};Let’s examine the run method. The code iterates through all global variables and discards those that have no value and those that aren’t constants. If the constant is a string, I get its value and encrypt it. Next, I replace the original string with encrypted bytes. Then I copy the list of users of this constant to the Users array. I go through the list and call handlers for different types of ‘users’. In both cases, I insert a local buffer and a call to the decryption procedure before the instruction. Finally, I replace the reference to the original string with a reference to the newly-created buffer.
#include <windows.h>#include <stdint.h>#include <stdlib.h>extern "C" void decryptStringInto(char* out, const char* enc, int len) { char key = 0xAA; for (int i = 0; i < len; ++i) { out[i] = enc[i] ^ key; } out[len] = '\0';}int main(){ DWORD written; const char* message = "Xakep.ru\n"; WriteConsoleA(GetStdHandle(STD_OUTPUT_HANDLE), message, (DWORD)lstrlenA(message), &written, NULL); return 0;}I use simple test code that outputs a string to the console. The easiest way is to insert decryptStringInto called by the obfuscator into the original code.
clang.exe -S -emit-llvm test_str.cpp -o test_str.ll
opt.exe -load-pass-plugin llvm_pass.dll -passes=encrypt-strings -S test_str.ll -o output.ll
clang.exe output.ll -o output.exe
Let’s build the source code and see what’s inside it after decompiling:
int __fastcall main(int argc, const char **argv, const char **envp){ HANDLE StdHandle; LPCSTR lpBuffer; int nNumberOfCharsToWrite; CHAR v7[9]; LPCSTR lpString; DWORD NumberOfCharsWritten[2]; NumberOfCharsWritten[1] = 0; sub_140001000(v7, &unk_14001A000, 8); lpString = v7; nNumberOfCharsToWrite = lstrlenA(v7); lpBuffer = lpString; StdHandle = GetStdHandle(0xFFFFFFF5); WriteConsoleA(StdHandle, lpBuffer, nNumberOfCharsToWrite, NumberOfCharsWritten, 0); return 0;}Run the file in the console, and you’ll see that everything works properly: the string is output without obfuscation, although it’s not stored in plain text.
OLLVM-16
Time to discuss some serious obfuscators. The obfuscator-llvm (OLLVM) project was launched in 2010 as a university project on code protection. In fact, it’s a fork of LLVM that embeds obfuscation passes into the compiler. It operates with intermediate code; so, theoretically, it’s compatible with any output language and architecture supported by LLVM. The latest version of this obfuscator was released for LLVM-4 (the current version is LLVM-20). But fortunately, there is a modern port OLLVM-16, and I am going to use it.
Building
First, let’s install in Visual Studio 2022 a couple of packages required to work with LLVM:
Compilers, build tools, and runtimes
☑ C++ Clang Compiler for Windows (19.1.5)
☑ MSBuild support for LLVM (clang-cl) toolset
Now I can start building.
git clone -b llvmorg-16.0.6 --depth=1 https://github.com/llvm/llvm-project.git llvmorg-16.0.6
git clone https://github.com/wwh1004/ollvm-16
robocopy ollvm-16/Obfuscation llvmorg-16.0.6/llvm/lib/Obfuscation
I edit the file llvmorg-16. and add add_subdirectory( to it.
cmake -S C:\llvm-16.0.6\llvm -B C:\llvm-16.0.6-build ^
-G "Visual Studio 17 2022" -A x64 -Thost=x64 ^
-DLLVM_ENABLE_PROJECTS="clang;lld" -DCMAKE_BUILD_TYPE=Release ^
-DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_OBFUSCATION_LINK_INTO_TOOLS=ON ^
-DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF
I run CMake and get the LLVM. file; from it, I am going to build the Release version. The building process takes slightly more than an hour.
Integration into Visual Studio
By default, Visual Studio uses LLVM-19. I must tell the compiler that I want to use a different version. To do this, I create the Directory. file in the root folder of the project:
<Project><PropertyGroup><LLVMInstallDir>C:\llvm-16.0.6-build</LLVMInstallDir><LLVMToolsVersion>16.0.6</LLVMToolsVersion></PropertyGroup></Project>This changes the environment variables; IDE will configure the rest of the settings automatically; and I just have to choose the proper toolset:
Platform Toolset = LLVM (clang-cl)
Now any code will be compiled using the required version of clang-cl with a built-in obfuscator.
Overview of obfuscation techniques
Obfuscation control in the compiler is implemented using special keys. Add the following strings to Command :
-mllvm -sub -mllvm -sub_loop=3 -mllvm -split -mllvm -split_num=3 -mllvm -fla -mllvm -bcf -mllvm -bcf_loop=3 -mllvm -bcf_prob=40
Let’s find out what each of the keys does.
Bogus control flow
This method inserts fake code sections (that will never be executed) into the basic block. Settings:
-
-bcf— enables this obfuscation type; -
-bcf_loop=N— specifies how many times to apply BCF to the same function; and -
-bcf_prob=P— specifies the probability (in percent, from 0 to 100) of adding a bogus block.
Let’s see what the code looks like before and after obfuscation.
AcceleratorsW = LoadAcceleratorsW(hInstance, (LPCWSTR)0x6D);On the basis of this code block, a loop with a condition that is always False is inserted nearby:
AcceleratorsW = LoadAcceleratorsW(a1, (LPCWSTR)0x6D);v9 = x;for ( i = y; y >= 10 && (((_BYTE)x * ((_BYTE)x + 1)) & 1) != 0; i = y ){ LoadAcceleratorsW(a1, (LPCWSTR)0x6D); AcceleratorsW = LoadAcceleratorsW(a1, (LPCWSTR)0x6D); v9 = x;}Unlike traditional garbage (i.e. insignificant or ‘dead’ code) injections, it becomes much more difficult to find the original code by unique WinAPIs. In addition, bogus blocks complicate the execution graph by dividing the basic block into several parts.
Instruction substitution
This obfuscation technique replaces simple instructions (e.g. addition, subtraction, or flag checking) with more complex, but semantically equivalent constructs. Its main principle is “make the simple complicated” in order to make it difficult to analyze and recognize standard patterns in the code. Settings:
-
-sub— enables this obfuscation type; and -
-sub_loop=N— specifies how many times to loop through the function.
Below is what it was:
EndDialog(a1, a3);And here’s the code after obfuscation:
v34 = (~v32 & 0xCE296CF528B2527EuLL | v32 & 0x31D6930AD74DAD81LL) ^ (~v33 & 0xCE296CF528B2527EuLL | v33 & 0x31D6930AD74DAD81LL);v35 = ((v34 | ~(~v32 | ~v33)) ^ 0x45B110EDBA343CE6LL) & (v34 | ~(~v32 | ~v33) | 0x45B110EDBA343CE6LL);EndDialog(hDlg, ~(~(~(v26 ^ ~v35) & (v35 | v26) ^ 0x7B085DB5B30B2D66LL) ^ 0x7B085DB5B30B2D66LL | (~v35 & 0x861C6ABB45D8AB39uLL | v35 & 0x79E39544BA2754C6LL) ^ 0x79E39544BA2754C6LL));Simple computations (like X+Y) turn into a long chain of operations. Only suitable sections of code are processed. Sometimes, the code of this obfuscation method leaks plenty of memory; as a result, the compiler freezes the system. After all, 20 gigabytes of RAM seems to be a bit excessive to compile 5 kilobytes of code. As far as I remember, the first ‘serious’ release in the early 2010s was already marred by this problem.
Control flow flattening
This technique ‘flattens’ the control flow, thus, turning an easily readable graph into something like a virtual machine. For example, below is the code prior to obfuscation:
if (x > 0) do_positive();else do_negative();do_final();And after obfuscation:
int label = 0;while (1) { switch (label) { case 0: if (x > 0) label = 1; else label = 2; break; case 1: do_positive(); label = 3; break; case 2: do_negative(); label = 3; break; case 3: do_final(); return; }}The control flow logic is no longer readable by eye. To figure out what the code is doing, tracing is required.
Settings:
-
-fla— enables this obfuscation type; -
-split— enables splitting basic blocks into parts; and -
-split_num=N— specifies how many times to split a basic block.
The split key splits a basic block into several smaller blocks connected by transitions; this option complements Flattening making the graph even more confusing.
Conclusions
Obfuscators make lives of reverse engineers much more difficult. To remove the obfuscation layer, one has to write a deobfuscator focused on a specific version of protection. Just add your own passes or slightly alter the existing ones — and the number of man-hours required to analyze a protected algorithm will increase by an order of magnitude. And if you combine obfuscation with anti-debugging techniques, you can rest assured: analysts will be grateful to you for long unforgettable hours spent in the decompiler.
Good luck!