Statistics, for the mathematically rigorous

Kevin Arlin already gave my "best" answer in All of Statistics. That said, statistics is a large topic. If you're looking specifically at inference and statistical learning and want a rigorous exposition of most of the different methodologies used I would recommend Elements of Statistical Learning, although it assumes a heavier mathematics background than you might expect (in particular, matrix differentiation and probability).

As a student of mathematics taking a statistics course next semester, I have been plagued with this problem. I have found a solution in "Lectures on Probability Theory and Mathematical Statistics" by Marco Taboga, which is a collection of many lectures and exercises from the website https://www.statlect.com/.

I am not familiar with this book, but the title alone suggests it might be worth examining for your purposes.

Statistics for Mathematicians: A Rigorous First Course. Victor M. Panaretos. Compact Textbook in Mathematics. Birkhäuser/Springer 142 (2016). ISBN-10 : 9783319283395. Springer link.

"Intended for students of Mathematics taking their first course in Statistics, the focus is on Statistics for Mathematicians rather than on Mathematical Statistics."

      

I would suggest "All of Statistics" by Wasserman. It is reasonably concise and moderate in its demands on background, but much more mathematically serious, also covering a much wider range of material, than a typical first course in statistics.