to begin with vs in the first place

"In the first place" and "To begin with" are both ok to be used at the end of a sentence (in the examples you provided). However, I hear "to begin with" used more in the beginning of a statement, rather than the end. For example:

I should never have taken that job in the first place. To begin with, it didn't even meet the minimum wage requirement.

or whatever reason. Basically:

I should not have __________ in the first place. To begin with, **** reason for previous sentence ***** .

Ah, I can't compete with these other answers! I will still express my opinion though. I think that all of the examples you asked about are grammatically correct, although I do feel that "in the first place" is more commonly used and sounds better. Still, they are interchangeable. I have also looked at some other sites, as this question seems to be one that is frequently asked, although do take each response in these sites with a grain of salt; they are written by people who may not be entirely correct.

https://forum.wordreference.com/threads/to-begin-with-vs-in-the-first-place.3594621/ https://www.usingenglish.com/forum/threads/132798-in-the-first-place-or-To-begin-with-correct

In both cases, these “add-ons” at the end of the sentences are slang expressions. “In the first place” implies there is a second place, a third place, etc. “To begin with” implies there is a “to continue with” and a “to end with.” Although both of these expressions are expectable as slang expressions go (I use them myself), they are both grammatically incorrect, with no correct alternatives.

Both are used, "in the first place" more than "to begin with".
I've done three Google Ngram queries but take them with a grain of salt.
The first one is a general search of the phrases not considering their position in the sentence.
The second one does take the position at the end of the sentence into account but since Ngram can't do START or END operations with longer phrases the results are only for "first place" and "begin with".
So they're also not as accurate as I'd like them to be.
The third one is done with a "." at the end of the phrase but that excludes sentences that end with "!" or "?" or any thing else.
In the end both are valid I think.

"Never end a sentence in a preposition" is what they always told me in school. Therefore, I usually say "in the first place".

Yes, you can use it. You can google it and check for references; here's one that you can bear out, from Invincible By Dustin Humphreys.

"Listen, we should never have taken that job to begin with," he said.