How do melee tourneys in Game of Thrones work?

In the melee you win or you die get injuries.

Robert wanted to participate in the melee at tournament he held for naming Ned his new hand. Ned suggested that Robert would win easily because no other combatant would dare strike him lest do harm to the king. So we know injuries are expected during the melee even though blunted weapons are used. At the end of it, Ned remarks on just how many injuries were sustained.

Ser Barristan Selmy spoke up. "Your Grace," he said, "it is not seemly that the king should ride into the melee. It would not be a fair contest. Who would dare strike you?"

Robert seemed honestly taken aback. "Why, all of them, damn it. If they can. And the last man left standing …"

"… will be you," Ned finished. He saw at once that Selmy had hit the mark. The dangers of the melee were only a savor to Robert, but this touched on his pride. "Ser Barristan is right. There's not a man in the Seven Kingdoms who would dare risk your displeasure by hurting you."

The king rose to his feet, his face flushed. "Are you telling me those prancing cravens will let me win?"

...

The melee went on for three hours. Near forty men took part, freeriders and hedge knights and new-made squires in search of a reputation. They fought with blunted weapons in a chaos of mud and blood, small troops fighting together and then turning on each other as alliances formed and fractured, until only one man was left standing. The victor was the red priest, Thoros of Myr, a madman who shaved his head and fought with a flaming sword. He had won melees before; the fire sword frightened the mounts of the other riders, and nothing frightened Thoros. The final tally was three broken limbs, a shattered collarbone, a dozen smashed fingers, two horses that had to be put down, and more cuts, sprains, and bruises than anyone cared to count. Ned was desperately pleased that Robert had not taken part.

^{A Game of Thrones - Eddard VII}

However, death is not unheard of in a melee.

In fact, Varys later explains to Ned that Lannisters were indeed plotting to have Robert killed during the melee which was considered a good way to make it look like an accident.

Varys shrugged. "There were forty riders in the melee. The Lannisters have many friends. Amidst all that chaos, with horses screaming and bones breaking and Thoros of Myr waving that absurd firesword of his, who could name it murder if some chance blow felled His Grace?" He went to the flagon and refilled his cup. "After the deed was done, the slayer would be beside himself with grief. I can almost hear him weeping. So sad. Yet no doubt the gracious and compassionate widow would take pity, lift the poor unfortunate to his feet, and bless him with a gentle kiss of forgiveness. Good King Joffrey would have no choice but to pardon him." The eunuch stroked his cheek. "Or perhaps Cersei would let Ser Ilyn strike off his head. Less risk for the Lannisters that way, though quite an unpleasant surprise for their little friend."

^{A Game of Thrones - Eddard VII}

We also know of one death during a melee during Dunk's adventures prior to the events of A Song of Ice and Fire

Prince Maekar turned to face him. "Some men will say I meant to kill my brother. The gods know it is a lie, but I will hear the whispers till the day I die. And it was my mace that dealt the fatal blow, I have no doubt. The only other foes he faced in the melee were three Kingsguard, whose vows forbade them to do any more than defend themselves. So it was me. Strange to say, I do not recall the blow that broke his skull. Is that a mercy or a curse? Some of both, I think."

^{The Hedge Knight}

If you are bested, you can yield to prevent further injuries.

During the melee Renly held after he declared himself (one of the five) King, Loras Tyrell is bested and yields to Brienne.

Loras Tyrell, on the bottom, took the brunt of the impact. The blue knight pulled a long dirk free and flicked open Tyrell's visor. The roar of the crowd was too loud for Catelyn to hear what Ser Loras said, but she saw the word form on his split, bloody lips. Yield.

^{A Clash of Kings - Catelyn II}

This scene plays out very nicely in the show as well.