not to infinitive

This following is from the book The pig that wants to be eaten by Julian Baggini. But I feel that one sentence is awkward. So I think "not" should be left out. What do you think about that?

Alongside this duty to make good our promises, though, there must also surely be a concomitant right to change our minds as our circumstances and beliefs change. Many people, for example, say things that begin ‘shoot me if I ever …’, especially when young. And although it is often just a figure of speech, it is frequently said with the utmost sincerity, and often by people who are of the age of majority and so considered adults capable of making decisions about their own futures. To hold people to these vows, however, would be ridiculous.

But why, twenty years later, is it ridiculous not to punish, if not actually kill, someone for going back on their vow not to vote Republican, but reasonable to expect them to try to maintain their marriage vows? There are significant differences. A marriage vow, like a mortgage agreement, involves responsibility and commitment to a third party. If we go back on these, others suffer. If we change our minds about matters of politics or religion, however, we do not, on the whole, breach any agreement we have with others.