Load the geometry package without having it change the document margins

I am using documentclass{book} along with the libertine and libertinust1math packages.

I would like to use the command phantom, which is in the geometry package (as seen in this answer). However, as soon as I introduce the geometry package, the margins of my document change. This can be seen by compiling the following MWE in two ways: once with the geometry package loaded and once without.

documentclass{book}
usepackage[semibold,tt=false]{libertine}
usepackage{libertinust1math}
usepackage{geometry}

begin{document}

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore 
et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut 
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse 
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa 
qui officia deserunt mollit anim id est laborum.

end{document}

How can I load the geometry package without having it change the document margins?