How to transform a streaming IP video into a CATV signal?

I wanted to create a circuit,

Um, no. What you describe encompasses at least one relatively beefy computer that has a network interface and runs an operating system with a network stack.

What kind of things should I start to learn and where.

Honestly, "recognizing things you can and cannot build with your current resources" is what you should learn first.

Essentially, you're saying "I want to build a high-bandwidth SDR and a cable television encoder; then some amplifiers to supply a CATV network, potentially".

You, however, don't seem to be aware of much of the underlying technology. It would indicate that the first thing you'd need to learn is how to look up things!

For example, "Cable TV" is an article on Wikipedia, and you might want to look into the standards mentioned there.

Then, you'd look up the standards (and will hopefully decide that implementing them yourself is a job for a group of experienced engineers with lots of background in the field, not for a one-off project), and then look for existing products or implementations.

There's ready-to-buy Cable TV modulators, and that's very likely the route you want to take (PC to receive your video stream + GPU + Modulator). If you actually want to build an SDR system of your own, you'd start looking for DVB-C implementations on the internet (you'll find a lot about GNU Radio).