Why does tilting a diffraction grating produce a parabolic pattern?

From your description, it sounds like you were creating a Moiré pattern between the fine spacing of the lines in the diffraction grating and the fine spacing of the pixels on your screen arranged in a cartesian pattern. Tilting the grating toward or away from the screen changes the apparent closeness of the lines in the grating at the top vs. at the bottom from your perspective (the part of the grating closer to you looks bigger and hence more coarsely spaced lines, while the part tilted away from you is farther away and hence looks smaller with finer spaced lines). This could produce the parabola-like curve you described.

A way to test this hypothesis would be to move the grating closer to or farther away from the screen, and see if that changes the size of the parabola. Alternatively, try rotating the grating by 45 degrees so that the grating lines no longer line up with the pixel cartesian grid of the screen. This should give rise to a new pattern.

Fun aside: this is also the same reason why taking a picture of a screen with a digital camera (such as your phone) produces weird lines. Moiré patterns form between pixels of the screen and pixels of the camera.

Maybe this is conical diffraction? Here is a paper describing it (unfortunately, behind a paywall): https://www.osapublishing.org/ao/abstract.cfm?uri=ao-37-34-8158

They have some neat graphical aids / geometric arguments to describe the phenomenon (which is surprisingly subtle!!!)

https://images.app.goo.gl/LbkduAbBLXjHKh598

The distance (D) between the grating and the screen is very important as shown in the attached formula. In your description, tilting the diffraction grating causes the top or the bottom to be closer to the screen than the other. This makes the equation different for the top and the bottom.

The effect, which is indeed sometimes called "conical diffraction", is very well known and well understood - although it might be a bit opaque for a person who's not familiar with 3D trigonometry. Here is a good place to go for a paper that describes it. First look at Figure 6.