Breaking out of a neverending battle without leaving the match

If you did the maths and even an endless series of lucky critical hits would not be enough, there really isn't anything more you can do for that sort of situation. You would indeed be stuck in an endless loop. There is no AI programming that makes the opponent forfeit, nor are there usually ways of forfeiting the battle yourself without resetting the game. Your situation was very unfortunate and almost never happens. I've been playing Pokemon games for years and have never been stuck in a battle I couldn't win OR lose.

In the case of the opponent placing you into a situation where you cannot defeat them by battle, there are a few ways to eventually take them down indirectly. Such as:

• Perish Song: Both Pokemon in play will faint after 3 turns if they stay in play. If your Pokemon has the Soundproof ability, it will not faint from this move.
• Badly poisoning the opponent with a move like Toxic: As the damage this status condition inflicts increases each turn, it will eventually knock out the opponent in one turn, regardless of what recovery moves/items they have.
• Removing the opponent's healing item with a move like Knock Off. That way, the recoil damage from Struggle would eventually make them faint.

These solutions assume you have the Pokemon to pull them off. It could not help the situation you described but they could be cautionary means of reducing the chances of it happening again.