Reverse Engineering Asked by JoshuaS3 on September 30, 2021
I’m disassembling a working-as-expected compiled C program (MS/Windows x64, MinGW gcc 10.1.0). In the main
function, I have the following call which passes 6 arguments to the function DotProduct
:
...
// void** m3 [rbp-0x50] = DotProduct(ptr, ptr2, 3, 2, 2, 3);
401b9f: 48 8b 55 b8 mov rdx,QWORD PTR [rbp-0x48] // arg2 = m2
401ba3: 48 8b 45 c0 mov rax,QWORD PTR [rbp-0x40] // arg1 = m1
401ba7: c7 44 24 28 03 00 00 mov DWORD PTR [rsp+0x28],0x3 // arg6 = 3 (+40) <- Δ main rsp
401bae: 00
401baf: c7 44 24 20 02 00 00 mov DWORD PTR [rsp+0x20],0x2 // arg5 = 2 (+32)
401bb6: 00
401bb7: 41 b9 02 00 00 00 mov r9d,0x2 // arg4 = 2
401bbd: 41 b8 03 00 00 00 mov r8d,0x3 // arg3 = 3
401bc3: 48 89 c1 mov rcx,rax // arg1 = m1
401bc6: e8 34 fc ff ff call 4017ff <DotProduct> // DotProduct(m1, m2, 3, 2, 2, 3);
401bcb: 48 89 45 b0 mov QWORD PTR [rbp-0x50],rax // void** m3 [rbp-0x50] = returned
...
Note that arg5
and arg6
are placed into [rsp+0x20]
and [rsp+0x28]
.
See what happens when DotProduct is called:
00000000004017ff <DotProduct>:
4017ff: 55 push rbp
401800: 53 push rbx
401801: 48 83 ec 48 sub rsp,0x48 // (-72)
401805: 48 8d ac 24 80 00 00 lea rbp,[rsp+0x80] // (-72+128) => (+56)
The rbp
should be at -0x48+0x80
relative to main
‘s rsp
, which simplifies to +0x38
(+56, Δmain
rsp). The stack pointer decreased 72 bytes, and the base pointer was placed 128 bytes above that (so 56 bytes above the previous stack pointer location). Next, DotProduct
loads the first four arguments from their registers:
40180d: 48 89 4d e0 mov QWORD PTR [rbp-0x20],rcx
401811: 48 89 55 e8 mov QWORD PTR [rbp-0x18],rdx // (rbp-24) => (+56-24) => (+32) !
401815: 44 89 45 f0 mov DWORD PTR [rbp-0x10],r8d // (rbp-16) => (+56-16) => (+40) !
401819: 44 89 4d f8 mov DWORD PTR [rbp-0x8],r9d
Wait! How are rdx
and r8d
loaded into these memory locations? Aren’t they occupied by arguments 5 and 6? What happened?
Right after that, we have this:
40181d: 8b 45 00 mov eax,DWORD PTR [rbp+0x0]
What’s going on here? [rbp+0x0]
hasn’t been initialized in this function, so what’s in this memory location? Where did arguments 5 and 6 go?
when you are here 4017ff: 55 push rbp
your 5th argument will be available at [rsp+28]
(8 bytes for return address and 20 bytes for HOMEPARAMS (space for saving the 4 args passed via register)
two pushes and one subtract will make your argument no 5 available at 0x28 + 0x8 +0x8 +0x48 = 0x80
so rbp+0 will hold the address of 5th argument after the LEA operation
Answered by blabb on September 30, 2021
I've spent many hours (too many for me to admit) trying to figure this out and I just realized what it is, right after hitting submit. push
instructions implicitly decrease the stack pointer. The call to the function DotProduct
calls push
three times; once in the call
instruction itself, where the return address is implicitly pushed, and twice in the function's prolog, where two quadword registers are explicitly pushed with the push
instruction. This all automatically decreases rsp
by 24 bytes, leaving arg5 and arg6 at [rbp]
and [rbp+0x8]
. The arguments aren't overwritten.
Answered by JoshuaS3 on September 30, 2021
Get help from others!
Recent Answers
Recent Questions
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP