Maximum number of "almost orthogonal" vectors one can embed in Hilbert space

This seems like it should be a known mathematical property of Hilbert spaces, but I can't immediately lay my hand on any such result. In lieu of that, this is very far from an answer to your question, but it perhaps indicates the difficulty of (some of) what you're asking...

First, perhaps we can clarify your problem statement. I assume you mean $$ |langle V_i|V_jrangle|leqepsilon,ineq j? $$ This would be consistent with your statement that $N(1/sqrt{2},2)=6$ because here you're be talking about the eigenvectors of the X,Y and Z Pauli operators. But Since you have both $|0rangle$ and $|1rangle$, for example, their overlaps are 0, not $1/sqrt{2}$.

Assuming this is the statement that you wanted...

There is the concept of Mutually Unbiased Bases. Essentially, this is attempting to evaluate $N(1/sqrt{d},d)$. (Strictly, it is a lower bound.) In the case that $d$ is the power of a prime, then there are $d+1$ mutually unbiased bases, meaning a set of $d(d+1)$ vectors. However, if $d$ cannot be written in this form, there are only bounds on the number of bases. For example, $d=6$, an answer is suspected but not known.

Of course, this doesn't rule out the possibility of getting the sort of limits you're after, but suggests that exact evaluation for arbitrary parameter values might be problematic. That said, you might want to clarify what you mean by the small $epsilon$ limit - you've already given the answer for $epsilon=0$, so that must be the small $epsilon$ limit.

You might be able to get some mileage out of this paper. It's related to mutually unbiased bases, but gives some other bounds as well which might be adapted to your purposes.