What limits the maximum reaction force that can be exerted by the table?

The weight of the object makes the table deflect (bend) slightly. This stretches the inter-molecular bonds in the table. The inter-molecular bonds resist the deflection, and this gives rise to the normal force exerted by the table on the object. However, the inter-molecular bonds have a finite strength, and if they are stretched too much they will break, and the table will no longer be able to support the object.

The whole story is a bit more complicated than this, because most materials are actually much weaker in bulk than the strength of their inter-molecular bonds suggests they should be. This is because stress tends to be concentrated in certain points instead of being spread uniformly throughout the material. See this Wikipedia article for more details.