Why did Gollum fail to seek out Bilbo?

Gollum had a number of opportunities between losing the Ring and encountering the Fellowship in Moria where he could have gone after Bilbo, and each has a different reason; I'm going to treat each separately.

After Bilbo escapes the Misty Mountains

Most immediately, fear of the goblins:

As the count grew he slowed down, and he began to get shaky and weepy; for he was leaving the water further and further behind, and he was getting afraid. Goblins might be about, and he had lost his ring. At last he stopped by a low opening, on their left as they went up.

"Seven right, yes. Six left, yes!" he whispered. "This is it. This is the way to the back-door, yes. Here's the passage!"

He peered in, and shrank back. "But we durstn't go in, precious, no we durstn't. Goblinses down there. Lots of goblinses. We smells them. Ssss!"

"What shall we do? Curse them and crush them! We must wait here, precious, wait a bit and see."

[...]

"Thief, thief, thief! Baggins! We hates it, we hates it, we hates it for ever!"

Then there was a silence. But that too seemed menacing to Bilbo. "If goblins are so near that he smelt them," he thought, "then they'll have heard his shrieking and cursing.

^{The Hobbit Chapter 5: "Riddles in the Dark"}

After that, Gandalf surmises it was a combination of fear of orcs and of sunlight:

Oh yes, he came out. His longing for the Ring proved stronger than his fear of the Orcs, or even of the light. After a year or two he left the mountains.

^{Fellowship of the Ring Book I Chapter 2: "The Shadow of the Past"}

According to Appendix B, this brings us up to T.A. 2944:

2944 Bard rebuilds Dale and becomes King. Gollum leaves the Mountains and begins his search for the thief of the Ring.

^{Return of the King Appendix B "The Tale of Years" (ii) The Third Age}

After Gollum himself left the Misty Mountains

He was, initially, tracking Bilbo; as Gandalf says:

He found his way into Mirkwood, as one would expect.'

'Is that where you found him?' asked Frodo.

'I saw him there,' answered Gandalf, 'but before that he had wandered far, following Bilbo's trail.

^{Fellowship of the Ring Book I Chapter 2: "The Shadow of the Past"}

Gandalf also says that Gollum found his way to Dale at this point, so presumably he had followed Bilbo's trail through Mirkwood to Erebor, and was now following it back in the general direction of the Shire.

At this point Frodo himself asks why Gollum didn't continue to the Shire, and Gandalf supposes that Sauron's influence was stronger than the desire for the Ring:

I think Gollum tried to. He set out and came back westward, as far as the Great River. But then he turned aside. He was not daunted by the distance, I am sure. No, something else drew him away.

[...]

I am afraid there is no possible doubt: he had made his slow, sneaking way, step by step, mile by mile, south, down at last to the Land of Mordor.'

A heavy silence fell in the room. Frodo could hear his heart beating. Even outside everything seemed still. No sound of Sam’s shears could now be heard.

'Yes, to Mordor,' said Gandalf. 'Alas! Mordor draws all wicked things, and the Dark Power was bending all its will to gather them there. The Ring of the Enemy would leave its mark, too, leave him open to the summons.

^{Fellowship of the Ring Book I Chapter 2: "The Shadow of the Past"}

Unfortunately this is only a guess of Gandalf's, but it's the best guess we have.

According to Appendix B, we're now up to T.A. 2951:

2951 Sauron declares himself openly and gathers power in Mordor. He begins the rebuilding of Barad-dûr. Gollum turns towards Mordor.

^{Return of the King Appendix B "The Tale of Years" (ii) The Third Age}

Before his capture in Mordor

It's not exactly clear what Gollum is doing at this point; according to Appendix B, he meets Shelob in T.A. 2980, and according to Unfinished Tales he was captured in 3017 (3009ish according to Appendix B), which still leaves a period of fifty or sixty years where we can't account for his actions, except for running back and forth between Mordor.

We do know that Aragorn and Gandalf are searching for him starting around 3001, so he may have been trying to avoid them. Other than that, though, we have no idea why he's spending so much time around Mordor and not looking for the Shire.

After being released from Mordor

Gollum almost certainly intended to continue hunting for Bilbo, but he was prevented by Aragorn; in Unfinished Tales, Tolkien writes:

Sauron perceived the depth of Gollum's malice towards those that had "robbed" him, and guessing that he would go in search of them to avenge himself, Sauron hoped that his spies would thus be led to the Ring.

Gollum, however, was before long captured by Aragorn, and taken to Northern Mirkwood

^{Unfinished Tales Part III "The Third Age" Chapter 4: "The Hunt for the Ring"}

After escaping from Mirkwood

A version of the story suggests that he was hiding from his pursuers, and ultimately sought refuge in Moria, intending to pass through it on his way to the Shire:

It seems clear that pursued both by Elves and Orcs Gollum crossed the Anduin, probably by swimming, and so eluded the hunt of Sauron; but being still hunted by Elves, and not yet daring to pass near Lórien (only the lure of the Ring itself made him dare to do this afterwards), he hid himself In Moria. That was probably in the autumn of the year; after which all trace of him was lost.

[...]

No doubt he had intended to use Moria simply as a secret passage westward, his purpose being to find "Shire" himself as quickly as he could; but he became lost, and it was a very long time before he found his way about.

^{Unfinished Tales Part III "The Third Age" Chapter 4: "The Hunt for the Ring" (ii) Other Versions of the Story}

After being lost in Moria for so long, he likely followed the Fellowship more out of desperation than out of any intent of finding the Ring (though their having hobbits in the party likely helped). Over time he may have eavesdropped enough to learn of their mission, or else he felt the draw of the Ring itself, and continued following them once out of the mines.