Why don't children find Mike scary?

Ignoring the fact that Mike palpably isn't scary (he's short and goofy looking, his teeth aren't especially sharp, his "claws" are kinda short, his giant eyeball makes him look vulnerable, his arms and legs seem puny and bandy) the reality is that in a world of scary monsters he's definitely the least scary.

Alarms went off all over campus, sending Hardscrabble and her security guards racing to the lab. In the cabin, Mike tried to scare the children, But they didn't find him scary. They thought he was cute!

_{^{Monsters University - Disney Press}}

Added to that, the children that Mike tries to scare at the end of MU are likely to be those who've already been scared half to death on a regular basis by the most scary monsters in the Monster world. By comparison, he's not even in the same league.

As the young lady he attempts to scare puts it...

You look funny

There's also a note in the script (registration required) about what she's saying in the background.

KIDS (waking up walla): What’s going on?/It’s the middle of the night./He’s a little funny green guy....

Armchair psychology time!

Kids like childlike faces, and the signifiers include big eyes and rounded faces:

Borgi et al. stated that young children demonstrate a preference for faces with a more "infantile facial" arrangement i.e. a rounder face, a higher forehead, bigger eyes, a smaller nose and a smaller mouth. In a study that used three- to six-year-old children, Borgi et al. asserted that the children showed a viewing time preference toward the eyes of "high infantile" faces of dogs, cats and humans as opposed to "low infantile" faces of those three species.

Mike also has no nose, and no hair might work as a stand-in for a higher forehead. Also as mentioned in the comments and other answer, although he has a large mouth, the teeth are blunted and not scary.

So the cuteness factor for audiences would simply be more pronounced in children.