How to understand the "infinite distance" part of the definition of gravitational potential?

In the physical sense, "infinite distance" means the separation of masses beyond which there is exactly zero gravitational force. This is "zero" in the mathematical sense - no real-life distance will ever truly be "infinite distance", as, if you take Newton's Law of Gravitation to be true, there will always be some tiny force present.

The reason that you are in a position with a finite potential even though potential starts at zero an infinite distance away is because, mathematically, if you take the idea that work done, $W$, is the area under the Force-Displacement graph ($F$-$r$ graph), you find that the corresponding integral (mathematical expression of that area) evaluates to a finite potential, despite the $F$-$r$ graph literally going on forever up to $r = +infty%$. Essentially, a mathematical reason.

$$GPE(R) = int_{r=R}^{+infty} -frac{GMm}{r^2} ,dr$$

$$GPE(R) = left[ frac{GMm}{r} right]_{R}^{+infty}$$

$$GPE(R) = left( frac{GMm}{+infty} right) - left( frac{GMm}{R} right)$$

$$GPE(R) = 0 - left( frac{GMm}{R} right)$$

$$GPE(R) = -frac{GMm}{R}$$

An infinitely long region under a graph has a finite area - a very unintuitive mathematical truth. Also, this is incorrect:

the gravitational potential at my position would be some positive number since external energy has been put in

Gravitational potential energy at any distance except infinity is negative, because you need to do work against a gravitational field to add to that potential energy, such that you might bring it up to 0. This is because you want to satisfy the conservation of energy - as you come to earth from an infinite distance away, the kinetic energy you gain is taken away from the gravitational potential energy, as the gravitational field is like an agent using your gravitational potential energy as a reservoir of energy to do work on you and imbue you with some kinetic energy.

I get the idea that if, say, I want to leave Earth I would have to possess enough kinetic energy to overcome the gravitational pull

Indeed, you have kinetic energy - as you move away from the earth, the field does work on you with a force opposite to your escaping motion, converting that kinetic energy to gravitational potential energy. It follows, then, that you need enough kinetic energy in reserve to "top up" your GPE to zero for you to "escape" the field in the physical sense - even though you cannot "escape" a field in the mathematical sense.