Norm estimates if adjoints can't be computed

Assume that you have two linear maps $A$ and $V$. For a given $x$ (of appropriate dimension) you can compute $Ax$ numerically, and for any $y$ (of appropriate dimension) you can calculate $V^Ty$ numerically. But you do not have access to both $A^Ty$ and $Vx$ on your computer. Hence, you can compute things like $$langle Ax,yranglequadtext{and}quad langle x,V^Tyrangle$$
and you may even be able to estimate
$$sup_{x,y}|langle Ax,yrangle – langle x,V^Tyrangle|.$$

My question is:

Is there a way to reliably compute $|A-V|$? (Any norm would be fine, spectral norm desirable. An upper bound would also be ok.)

I’d like to add, that this situation is somehow always the case in floating point arithmetic, because each time you compute A*x or A'*y, different rounding errors occur, hence the results of (A*x)'*y and (x')*(A'*y) are different, too. (Btw, does anybody know an estimate for this quantity?) In double precision, this is probably not much of a problem, but in single or half precision, the errors seem to become non-neglectable.