With or without 'to'

Should it be ‘the place we went last night’ or ‘the place we went to last night’?

I feel like if I swap ‘restaurant’ in for ‘place’, I’d definitely include the preposition ‘to’. But with ‘place’, you have sayings like ‘It’s always the last place you look.’ (used without preposition at) ‘This is the place to go.’ (without preposition to after go)
Can ‘place’ alone function as an adverbial of location much like ‘the day’ as in ‘the day I graduated’, except ‘the day’ functions as an adverbial of time.

Which one do you see people more likely to say in spoken English, ‘the place we went to last night’ or ‘the place we went last night’?

Could that have something to do with the flow and rhythmn of sentences too?
I feel like ‘the place we went to last night’ flows better than without ‘to’, but if you have a couple more syllables in the sentence, leaving out ‘to’ would be preferable as in ‘the place we used to go on Saturdays’. What do you think?