How to calculate the brightness of an object in space dependent on its distance from earth?

I'm not sure if this is the normal astronomer's way of calculating things, but i think this picture is quite intuitive:

Let's first assume the comet is radiating a constant total amount of light with a power $P$ into all directions evenly. We can now imagine that if we are a distance $R$ away from the comet, this power is distributed evenly across the surface A of the sphere (which is $4 pi R^{2}$), and at that point we have the power density $$p=frac{P}{A}=frac{P}{4 pi R^{2}}$$. Now, we can easily compare the power densities for two radii $R_1$ and $R_2$; the total power $P$ is independent of distance, thus all factors except $R_1$ and $R_2$ will drop out:

$$frac{p_1}{p_2}=frac{frac{P}{A_1}}{frac{P}{A_2}}=frac{A_2}{A_1}=frac{4 pi R_2^{2}}{4 pi R_1^{2}}=frac{R_2^{2}}{R_1^{2}}$$

With this, you can easily calculate the different power densities. In your case, plugging in $R_2=25AU=5*5AU=5*R_1$ and $R_1=5AU$ yields $frac{p_1}{p_2}=25$. So the number of photons hitting your eye will be 25 times larger for your given values and it would be 25 times brighter in astronomical terms. To relate this to the eyes' sensitivity is a little trickier, but I don't think this is of importance here and would also be subjective.

In astronomy you also call $P$ spectral luminosity $L$. $p$ would be flux density($F$ or $S$, depends on convention).