Futuristic agent stuck in past, runs off with own mother

Pretty sure this was a full novel from the 80s. Set in the future with spaceships, a special agent has a traumatic memory of someone taking away his mother and being abandoned.

Somehow on a mission he ends up in the past. I think there was an entire adventure plot with a villain, that I just don’t remember.

But the kicker is that he gets romantically involved with a woman and then tries to save her during a calamity, but doesn’t have time to get her son. He then realizes who she is, and that he caused his own trauma.