how to use class member function defined in a exe within a dll

This is an interesting problem where some issues have to be solved.

For the first part of your question: How to run an exe from memory, you might want to consult the following link: https://stackoverflow.com/questions/305203/createprocess-from-memory-buffer

In the following, I will try to show you how to tackle your second question, how to call a class method of that exe being present in memory.

The following example (keep it as simple as possible) does not assume a DLL as a caller, but another exe. The mechanics should be exactly the same, however.

Part 1 : Called exe

Let us start with a very simple C++ Windows (VS2013) exe serving as the one to be called from another exe (or DLL in your case). I named this exe "ExeToBeCalled".

#include <iostream>

class myClass
{
public: 
    int myVar;
    void myFunc(int a, int b)
    {
        myVar = a - b;
    }
};

int main(int argc, char* argv[])
{
    myClass* pmc = new myClass();  //parameterless constructor
    myClass* pmc2 = new myClass(); //parameterless constructor
    pmc->myFunc(123, 23);
    pmc2->myFunc(456, pmc->myVar);
    std::cout << pmc2->myVar;
    return 0;
}

It contains a class with one method, called myFunc, which subtracts its two parameters and stores it into a public member variable, called myVar. The subtraction instead of e.g. an addition has been chosen to be easily able to decide whether the correct parameter calling sequence is used.

Two instances of this class are created, and the result is output to the console. Sloppy programming, as the delete has been omitted.

The interesting part is the fact that the class instances (created with "new" on the heap) contain only the myVar on the heap, because each instance must have its own myVar, whereas the myFunc is common for all instances and as a consequence its address is statically assigned by the compiler. This address has to be found out, e.g. by inspection in Ida. From your posting I assumed that you solved this problem already. The problem becomes more difficult for virtual methods, not considered in our simple example.

Part 2 : Caller exe

This of course is the interesting part. I also created an example (VS2013), like so:

#include <Windows.h> // necessary for the "LoadLibrary"
#include <iostream>

// class method, statically assigned by the compiler
// The address has to be found out, e.g. by inspection in Ida
// You solved this part of the problem already.
// As we call the function from assembler, the __cdecl is irrelevant.
// It would matter though if we called from C++.
// Note that the callee needs a valid ecx for the function to succeed.
typedef void(*__cdecl pmyFunc)(int a, int b);

    // assumed myClass layout, being a guess of the real layout of the class.
    // We only need the member variable
    // This "class" is for problem illustration only, we do NOT need it in our program.
    class myClass
    {
    public:
        int myResult;
    };

int main(int argc, char* argv[])
{
    HANDLE pExe = LoadLibrary("ExeToBeCalled.exe");

    // Address e.g. obtained by investigation of the callee in Ida.
    pmyFunc myFunc = (pmyFunc)((byte*)pExe + 0x4320); 

    // let us get some memory from the heap for the class
    // As we do not know the "real" layout of the class, we allocate one kB, hoping it will be enough
    // We do NOT refer to the "myClass" above
    void* pmyClass = new byte[1000];

    // Set it all to zero.
    for (int i = 0; i < 1000; i++)
        ((byte*)pmyClass)[i] = 0;

    // let us switch to assembler for the call
    _asm
    {
        // We have to provide a valid memory in ecx where the real myclass can store its result
        // If we provide here for testing a zero into ecx, the app crashes.
        mov ecx, pmyClass;
        push 333;
        push 222;
        call myFunc; //result should be 222-333 = -111
    }
    // By inspection of the pmyClass memory in the debugger after the call to myFunc,
    //    we saw that the real myClass just stored the result into the first four bytes.
    // Of course, in reality this is not known beforehand, thus - in a real situation -
    //    the whole 1kB could have been searched for the expected result, determining
    //    the offset of myResult from the ecx value in reality.
    int result = *(((int*)pmyClass));
    std::cout << result;
    return 0;
}

It has to be noted that our test exe from part 1 has to be compiled in Debug instead of Release mode. It is too simple for the compiler which optimizes it and does not even bother to call the myFunc, but directly calculates the result.

To summarize, for the solution of your problem the following must be assumed or known:

1. The address of the class method. We assumed a static address which need not to be the case in general. "Static Address" here means a constant offset (RVA) of the exe's start address obtained by LoadLibrary, which usually differs on every load (ASLR).

2. The way the compiler of the exe used to call a method of a class. Here we assumed the passing of the "this" pointer in the ecx register.

3. The memory location where the result of the method call is stored. This also can be arbitrarily complicated in general, in our demo the simplest possible memory layout turned out to be correct.

In one of the comments the question about 64Bit has been asked. Of course, there the situation might be different, the method to investigate the problem would be rather similar.

I hope this little example could enlighten one method how to attack your problem.