Why is the completeness of UCS guaranteed only if the cost of every step exceeds some small positive constant?

Let's consider a problem where all edge costs are greater than zero, but not above some $epsilon$:

Image a problem where we have an infinite path where the first edge is cost $frac{1}{2}$, the next is $frac{1}{4}$, the following is $frac{1}{8}$, and so on forever. Every edge is greater than zero, meeting the condition being proposed in the question. However, this path overall has finite cost (1) even through there are an infinite number of states on that path. So, on this problem UCS will never reach paths with cost greater than 1. Thus, if the solution cost is 2, UCS will not find any solution to this problem, and thus it would not be a complete algorithm. So, all edges being greater than zero is not sufficient.

For most search algorithms to be complete, there must be a finite number of states with any given cost. (To be slightly more precise, there must exist some fixed $epsilon$ such that in each range size $epsilon$ there are a finite number of states.)