Trace over configuration basis

I'll give it a shot. Since this is a many particle configuration, we have that $[q_i, q_j] = 0$ and $[p_i, p_j]=0$ and $[q_i, p_j] = i delta_{ij}$ where the indices label the particles. Therefore, the integral will become separable. Namely, we have that

begin{align*} &tilde{mathrm{tr}}left{f(mathbf {hat q}, mathbf {hat p})right}equivint_{mathbb{R}^N} mathrm dmathbf {hat q} leftlanglemathbf qmiddle|fleft(mathbf {hat q} ,-ifrac{partial}{partial {mathbf {hat q} }}right) middle|mathbf qrightrangle &=int dq_1cdotsint d q_N langle q_1|langle q_2| cdotslangle q_N| left(hat q_1^{n_1}cdots hat q_N^{n_N}left(-ifrac{partial}{partial hat q_1}right)^{m_1}cdots left(-ifrac{partial}{partial hat q_N}right)^{m_N} | q_1 rangle |q_2rangle cdots |q_Nrangle right) end{align*}

where I have made use of the definition of the direct product on a hilbert space for $N$ particles.

Given our commutation relations, these integrals are separable. That is,

begin{align*} &tilde{mathrm{tr}}left{f(mathbf {hat q}, mathbf {hat p})right}=prod_{i=1}^N int dq_i langle q_i| hat{q}_i^{n_i}hat{p}_i^{m_i}|q_irangle end{align*}

Now this we can deal with. first recall that $langle q_i| hat{q}_i = langle q_i| q_i$ so that

begin{align*} &tilde{mathrm{tr}}left{f(mathbf {hat q}, mathbf {hat p})right}=prod_{i=1}^N int dq_i int dp_i langle q_i| q_i^{n_i}hat{p}_i^{m_i}|q_irangle end{align*}

To take care of the momentum we insert unity resolved in the momentum basis of particle $i$ so that

begin{align*} &tilde{mathrm{tr}}left{f(mathbf {hat q}, mathbf {hat p})right}=prod_{i=1}^N int dq_i int dp_i langle q_i| q_i^{n_i}hat{p}_i^{m_i}|p_i rangle underbrace{langle p_i|q_irangle}_{frac{e^{-iq_ip_i}}{sqrt{2pi}}} &=frac{1}{(2pi)^{frac{N}{2}}}prod_{i=1}^N int dq_i int dp_i q_i^{n_i} p_i^{m_i} langle q_i|p_i rangle e^{-ip_i q_i} &=frac{1}{(2pi)^{frac{N}{2}}}prod_{i=1}^N int dq_i int dp_i q_i^{n_i} p_i^{m_i} e^{ip_i q_i} e^{-ip_i q_i} &=frac{1}{(2pi)^{frac{N}{2}}}prod_{i=1}^N left(int_{mathbb{R}} dq_i q_i^{n_i}right)left( int_{mathbb{R}} dp_i p_i^{m_i} right) end{align*}

so indeed it appears that unless we impose a momentum cutoff and restrict ourself to a finite region of space then what we have is just a big product of divergences, and hence is not a well defined map from $mathcal{H}to mathbb{R}$.

As for your question regarding the weighting factor of the hamiltonian I believe that the answer ought to depend on what the actual hamiltonian is, but if you think that's incorrect I can reconsider and try to approach that for general $hat{H}$.