What is the species that Worf fights on the holodeck?

They aren't named in-universe or outside.

The skull-headed monster appears in 4 episodes within the trek universe, each time in a "Klingon Training Program". Evidently they're "Alien Warriors"..."from Klingon Mythology".

DS9 Way of the Warrior :

"Worf is wielding a Klingon saber, fighting a hulking ogre-like monster from Klingon mythology."

TNG Where Silence Has Lease :

As he rolls catlike to his feet, the massive form of an ALIEN WARRIOR leaps onto his back bringing him back to the ground. The alien raises a club-like weapon over his head and swings down at Worf's head.

TNG New Ground :

Alexander is wearing a small practice robe similar to the one we've seen Worf use. He is fighting the SKULL FACED ALIEN, seen in previous episodes, with Worf's bat'telh sword.

TNG : The Emissary

Then, seemingly from nowhere, a massive ALIEN WARRIOR springs into view, charges at K'Ehleyr.

---

Out of universe, this article on Memory Alpha describes the origin of the Skull-faced Warrior. Evidently it's the same mask used for Skeletor in the 1987 flop "Masters of the Universe"

Although it's not a canon book, the novel "Kahless" describes him as a Pandrilite. Don't deny yourself a good read because a book isn't canon! :)

Actually, it was an adaptation of a program Worf himself had used when he first arrived on the Enterprise.

Of course, a Klingon had been inserted in place of a Pandrilite and it was restricted to Level One, whereas Worf had bumped it up to Level Three at times. But otherwise, it was pretty much the same.