Unknown, though it's probable that Dumbledore was holding it for safekeeping.

There's no clearcut answer in canon as to how Hagrid ended up with Harry's key. However, we can venture an educated guess that he was given it by Dumbledore, based on the evidence of the books.

We can safely assume I think that the Potters' Gringotts key was in their house at Godric's Hollow during the attack by Voldemort. One does not leave one's banking identification lying around.

So someone must have collected the key from the rubble. The most obvious candidate would be Hagrid, since he was the one who picked up Harry in the aftermath of the attack. I think, however, that we can rule Hagrid out. Based on his own testimony, Hagrid had only just finished extracting Harry from the ruins when Sirius Black arrived.

"It was me what rescued Harry from Lily an' James's house after they was killed! Jus' got him outta the ruins, poor little thing, with a great slash across his forehead, an' Sirius Black turns up, on that flyin' motorbike he used ter ride."
_{(Prisoner of Azkaban, Chapter 10, The Marauder's Map).}

After a brief discussion with Black Hagrid took off. I don't think it's credible that Hagrid hung around in an attempt to root through the rubble for an old key. We know that Hagrid was keen to get going before the Muggle authorities showed up.

"No, sir - house was almost destroyed but I got him out all right before the Muggles started swarmin' around."
_{(Philosopher's Stone, Chapter 1, The Boy Who Lived).}

Now it's possible that somebody else, perhaps an Order member, salvaged the key from the wreckage. We know that Wormtail, for instance, visited the site shortly after Hagrid left. The most likely scenario, however, was that the Potters' surviving possessions were salvaged and repurposed according to the contents of their will.

It's almost certain that James and Lily would have bequeathed their entire estate to Harry, much as Sirius did.

"But first of all I must tell you that Sirius's will was discovered a week ago and that he left you everything he owned.
Over on the sofa, Uncle Vernon's head turned, but Harry did not look at him, nor could he think of anything to say except, "Oh. Right."
"This is, in the main, fairly straightforward," Dumbledore went on. "You add a reasonable amount of gold to your account at Gringotts and you inherit all of Sirius's personal possessions."
_{(Half-Blood Prince, Chapter 3, Will and Won't).}

It seems logical that he would've been bequeathed the key to his account, just as he was bequeathed the fortune within.

Yet Harry obviously didn't receive the key straightaway. Since Harry was a baby it seems likely someone served as guardian for his estate, and his Gringotts key, until he entered the wizarding world. The most obvious candidate for this role would be Dumbledore. We know that he performed a similar function with regards to Harry's Cloak.

Your father left this in my possession before he died.
It is time it was returned to you.
Use it well.
A very merry Christmas to you.
_{(Philosopher's Stone, Chapter 12, The Mirror of Erised).}

The main difference between the Cloak and the Gringotts key is that James gave the former to Dumbledore voluntarily before his death rather than through his will. Nevertheless, it shows that Dumbledore was willing and able to perform the role of guardian.

This fits with the prominent position that Dumbledore took in overseeing Harry's future. It was Dumbledore who decided to send Harry to the Dursleys, Dumbledore who sent Hagrid to pick him up and Dumbledore who oversaw Harry's placement on the steps of Privet Drive. He was the key figure who managed the fallout from the deaths of Lily and James. It's likely that Harry's parents would've entrusted Harry's fate to Dumbledore should anything happen to them.

Additionally, Hagrid has a letter from Dumbledore with him when he visits Gringotts. It would make sense if Dumbledore gave him the key and the letter together, since Hagrid was taking the trip on Dumbledore's orders, rather than Hagrid having to source the key separately.

"An' I've got a letter here from Professor Dumbledore," said Hagrid importantly, throwing out his chest.
_{(Philosopher's Stone, Chapter 5, Diagon Alley).}

This reasoning is necessarily speculative. However, I think it presents the most logical explanation as to how Hagrid came by the key.

Complicating matters further is the confusion about how one actually gains access to their vault at Gringotts.

These golden keys are never mentioned again. Hagrid gains access to Nicolas Flamel's vault with a letter (which isn't even written by the account holder!). Hermione gains access to the Lestrange vault by presenting Bellatrix's wand. Essentially, the keys don't seem to be essential for gaining access to an account. For instance, Harry takes money out of his vault in book three.

Once Harry had refilled his money bag with gold Galleons, silver Sickles and bronze Knuts from his vault at Gringotts, he needed to exercise a lot of self-control not to spend the whole lot at once.
_{(Prisoner of Azkaban, Chapter 4, The Leaky Cauldron).}

We're not told how Harry was able to do this. Did he have access to his own key by this point? Did he use his wand to gain entry? We don't know.

Additionally, Mrs Weasley accesses Harry's account and withdraws money on his behalf on more than one occasion. Did she have access to the key? Again, we don't know.

Nevertheless, I think we can reasonably conclude that Hagrid was given the key by someone who had access to the Potters' estate, and that the most likely person to fulfil this role was Dumbledore.

Maybe the following makes sense: after the death of lily and james potter the vault was locked up, since there was no active user. With the first appearance of harry at gringotts he was then established as the rightful owner of the estate and the goblins could recognize him as such. The vault was thereby reactivated and could be accessed by anyone for harrys sake. It is quite obvious, that the goblins have a very keen appreciation of property rights and it would not be wise to try and deceive them to take for you what isnt yours.