What's the difference between "wasn't" and "hadn't"?

Both are grammatically correct, but they are used in different circumstances.

If the baby had been in the habit of not sleeping through the night, then choice A would be appropriate.

If the baby didn't sleep well just last night then B would be the one to use.

A is past continuous tense, which is used to describe an action that happened and is still happening at this moment. B is past perfect tense, meaning it happened in the past and is not happening at this moment. Because Simon's action is a reaction to a past situation (meaning the baby's action already happened and is not at this moment), the past perfect would have to be used.

B, which uses "hadn't," is a better description of an event that has already ended. In this example, Simon feels exhausted because he was unable to get any sleep during the night. So the night is over, and we switch to a more perfect tense to describe what went on inside it.

http://www.englishpage.com/verbpage/pastperfect.html

A's use of "wasn't" probably falls under the Simple Past designation. "The baby wasn't sleeping all night" — you might come across this sentence in a book when the narrator talks about a year when insomnia was a bodily habit of the baby. But it's not specific about the time in which this took place. B talks about a recent and contained span, while A could refer to a longer one. For example, "last month the baby wasn't sleeping through the night." In the page linked below, glance at uses 3, 4 and 5.

http://www.englishpage.com/verbpage/simplepast.html

In short, "had" indicates that an action has been completed. "Was" is a more general description of the past that can cover a whole time period. This example calls for a narrow description of when poor Simon had to stay up.

Hope that helps. Narrative tenses are tricky until you get a lot of practice.