What would happen if the attractive force at work in the nucleus had been equal in magnitude to the repulsive electrostatic force between the protons?

Firstly, it is necessary to mention that nuclei are not the only places where the attractive and repulsive forces are in competition. Molecules and solids are other obvious examples: e.g., in a metal the positively charged ions repel each other, just like the negatively charged electrons do. Yet, the system as a whole is stable, due to the attractive forces between these two, and exchange interaction (without exchange the system would be still unstable, as stated by the Earnshaw theorem).

What makes nuclei different is that the attractive forces between nucleons are of different nature than the electrostatic forces, and have different dependence on the distance, roughly modelled by the Yukawa potential. Thus, the attractive forces dominate at very short distances, but quickly decay with distance. Moreover, these forces act not only between positively charged protons, but also between protons and neutrons. In particular, the system of two protons could not be stable, but the system of a proton and a neutron (deuterium), of two neutrons and a proton (tritium) and of two protons and a neutron (Helium-3 nucleus) are all stable (or have very ling lifetime).

To summarize:

• Nuclear and electrostatic forces have different dependence on a distance, so they couldn't cancel each other completely - only in certain spatial configurations.
• Nucleon systems where nuclear forces do not dominate (at short distances) are unstable.