Is the set of spacetime manifold a set of event?

No, it's not.

A manifold is not just a set of points. It also has a topology and a smooth structure. In fact, there is such a thing as a pointless topology, where we only consider the topological structure and forgo the points. In a sense, here, points are extra structure.

Since QM suggests that points aren't physical, a pointless manifold is physically more realistic.

As long as you do all the math correctly you are free to use whatever English language words you want to describe it with. I would probably say that the spacetime manifold consists of points at which events could take place. I wouldn't say that the manifold itself consists of events, though.

The conclusion that I arrived, is that there is no manifold without objects in the universe. For example ,suppose our universe consist of a particle $gamma$ and the a metric field $g$.The interaction between this metric field and this particle, is in such a way, that we can model this system as manifold, with a metric $g$ and curve $gamma$ on it. So for example if this metric would act differently, between to different particles, $gamma_1$ and $gamma_2$ this model of universe as a four dimensional manifold with a metric $g$ on it, would be invalid.

A manifold is not just a set, but it requires a set on which additional mathematical structures are built (topologies, coordinates, charts, atlas, metrics, and so on). Spacetime of GR can be considered as based on a set of points where each point is an event (something which happens). An event requires that there is something.

The implicit assumption of GR is that, whatever this something is, there are as many as required by a theory based on a continuum of events (more technically, enough to allow a separable topological space). It is not necessary that all the events are recorded. It is just required that at least in principle they could be identified. The initial chapters on space-time in the classic Misner Thorne Wheeler (MTW) Gravitation will provide a readable introduction to this concept.

As a consequence, your hypothesis of a set of events reduced to the world-line of just one particle has a drawback. If there would be only one particle how can we identify an event?

I would say that the best intuition for the events of the GR spacetime, consistent with MTW point of view, is as a concept analogous to test-particles in the definition of the electromagnetic field. They are required to assign an operative definition to the concept of field, and we may have good approximations for them at the macroscopic level, although, at the microscopic level, we may have a hard time going below the elementary charge. However, arrived at that scale, we need new theories.