Is it true that it's OK to omit the pluperfect (past-perfect) in casual speech?

The pluperfect is used to indicate what is relatively earlier than the compared clause. But in conversation I hear people omit it all the time. Examples:

I haven’t spoken to the President since the inauguration. We had a few
conversations before that.

vs.

I haven’t spoken to the President since the inauguration. We’d had a few
conversations before that.

Or,

The incident occurred yesterday, but things started deteriorating
before that.

vs.

The incident occurred yesterday, but things had started deteriorating
before that.

If I were writing something I would take care to include the pluperfect, but in conversation I often don’t have time to do the comparison. It seems to be omitted a lot by people. Am I correct in this perception?