Does Umbridge's scar prevent Harry from telling lies?

No it doesn't. It's kind of a way of saying that "I am hurting you because you lied. And if you continue I will continue too." I thinks it's more like a humane torture (not a magical spell or hex or anything)

Two things points against a magical compulsion to lie, as opposed to her merely doing it out of cruelty and hopes of making him fear to lie to her.

The first is that she tried to use veritaserum on him after it, so she knew he could still lie.

The second is that while she is torturing him and asks him to change his story, he does not automatically and instantly tell the truth. He could have lied and claimed that Voldemort had not returned, but choose to tell the truth.

There is no evidence that it magically prevents Harry from lying.

Instead, this is a form of psychological torture. Umbridge forces Harry to write with his own blood, causing pain and a scar on himself. From Umbridge's point of view, this pain should become associated with lying - in other words, Harry should associate lying with pain and thus not lie in the future.

Of course, Harry was telling the truth the entire time, and Umbridge is a sadistic abuser who was more concerned with scarring - physically and mentally - Harry rather than discovering what the truth was. The point is just to cause pain, both physical and psychological, for Umbridge's enjoyment.