Anime where everyone has a crystal inside their body to use magic

I don’t remember much about it but the main character was a child (about a few months old) born into a world of magic. To use magic in this world everyone had a crystal inside their body and you had to put your crystal together.

It took most people a few years to do, but since he started at a young age he did it fairly quickly and could use magic as a baby and he also loved to read. He would always go into the library of his house so the parents just decide to put his crib inside the library and when he got older his parents decided to take him to the Capitol to enroll him into a magic school.