Is there a historical reason for the changes in format and tone in E. T. A. Hoffmann's "The Sandman"?

The 1816 story "The Sandman" by E. T. A. Hoffmann is a very famous work of nineteenth-century horror fiction and a landmark in the development of the German Romanticism. (The original German text is here, and an English translation may be found here.) It has been adapted for the stage and screen numerous times—most famously as part of Jacques Offenbach’s opera The Tales of Hoffmann.

However, the story itself is very peculiarly structured. It begins as an epistolary story, with three letters back and forth between the main character, Nathanael, and his friend (and fiancee’s brother) Lothar. However, the story then shifts to having a completely separate narrator, which gives the tale quite a different tone. The central theme remains consistent—for a while—despite the change of narrative device. Nathanael, having had a troubling experience as a child with an acquaintance of his father (an acquaintance who may have been involved in the father’s death), has fears that the acquaintance, Coppelius, may have supernatural powers and may have returned to torment him again. When Nathanael returns home from the university, his fiancee Klara counsels him that he is being unreasonable, and she eventually convinces him that he is letting his fears run away with him.

Up to this point, the story has been a psychological thriller—with the central question being whether Nathanael’s fears are justified, or whether the evil he perceives is all in his head. It is entirely possible, that there have actually been no fantastical happenings at all. As Nathanael heads back to the university, the reader may expect another round of mysterious goings-on to torment him. However, although the Coppelius character does continue to be part of the story, there is a total change in tone, as Nathanael is seduced by a beautiful automaton that one of his professors has constructed and passed off as his daughter Olimpia. Here is her big scene from a production of The Tales of Hoffmann, which shows the lack of subtlety about her robotic nature.

Nobody in the story figures out that she is a robot, but the element is played heavily for dramatic irony. This marks another stark change in tone for the story, and it also introduces an unambiguous fantasy or science fiction element, in contrast to the earlier parts, where it could have all been in Nathanael’s head.

I think "The Sandman" is an excellent nineteenth-century Romantic horror story, but I find the way the tale completely changes character—not just once, but twice—very strange. Is there a known reason for these bizarre changes of structure and tone? Did Hoffmann compose the story in pieces, with a changing conception of what kind of tale he was telling? In these 1968 and 1975 scholarly articles about "The Sandman," the author largely sidesteps the role Olimpia plays in the plot and concentrates on the psychological elements that dominate the first parts of the story (taking a largely Freudian approach). The other scholarly literature about the story seems to be largely in German—and my German is not good enough for reading and understanding literary criticism. So is there an understanding of why Hoffmann’s story is so disjointedly structured?