Can I say "all you had to do is"?

You certainly “can” say any of these:

1. All you have to do is take out the trash.
2. All you had to do was take out the trash.
3. All you had to do is take out the trash.

It’s easy to find examples of all three on the internet, in speech, and in print.

The first two use coordinated tenses that match, either both present or both past. No one in the world would emit the least complaint about either of those two.

The third one uses past in the first part and present in the second. People certainly do say things like this. And everyone will know what you mean; few would read into the third example as meaning something different from what the second meant.

A careful writer, or a careful copyeditor, might well change the third version back into the second so that the tenses match up.

But most people would neither notice nor care.