Rise of Nations modding units

Trying to address your specific questions, without writing a full modding guide.

Is a 'rock paper scissors' system implemented somewhere in the game code outside the .xml file, or can you change unit behavior in the .xml file alone?

Additional combat effects are in balance.xml, in the same folder as unitrules.xml. The balance.xml file is rather large and lists a damage muliplier percentage for all possible match-ups of units. This includes specific unit types, referred to as Object Masks. For example "Flag_M_OBJMASK_MOUNTED". The object masks are explained at the top of unitrules.xml, and each unit's set of object masks is defined in the same file.

As an example the Heavy Machine Gun has OBJ_MASK of FTG, corresponding to F=Foot, T=Tank/War Machine, G=Gun

Where can you alter the text which shows in the in-game popup windows that tell the player that unit x is strong vs. unit y but weak against unit z?

This could depend on what you want to do with tooltips. But for existing units look in translated_strings.xml, at lines 2150,2151,3167,3169 and 2203-2261. They aren't really id'd but you should be able to figure it out.

For example, changing the entry on line 2240 to "Foot Archers, Stack Exchange" changes the tooltip as shown in the screenshot below

Is unit strength controlled only by the hit-point and attack strength in the .xml file or are there hidden combat effects of unit specific code elsewhere in the game?

In addition to damage bonuses/penalties balance.xml, there are other factors in unitrules.xml. The other important combat effect you are forgetting is Armor, which mitigates damage. Low damage units are ineffective against units with high armor. Other indirect factors you may be forgetting are Range and Speed, which for example, contribute to why Ranged Cavalry are effective against Heavy Infantry.

What are the specific effects of any 'rock paper scissors' settings? Does a unit strong vs. another unit inflict 2.5x normal damage or some other value?

A unit being strong or weak vs another does not indicate a specific multiplier. The data in balance.xml sets the damage multiplier to a specific value. It can be read as a percentage. Some examples below, with excerpts from balance.xml

War Elephants EMW are listed as "Strong vs. Heavy Infantry". In practice they do double damage (200%) to Heavy Infantry

ENTRY name="War_Elephant"
...
Flag_H_OBJMASK_HEAVY_INF="200"

Destroyers get varying bonuses against Submarines and Aircraft.

ENTRY name="Destroyer"
...
Submarine="700" Attack_Submarine="700"
...
Flag_3_OBJMASK_AIR="300"

While Light Tanks are listed as "Strong vs. Modern Infantry", but neither the tank or infantry have bonus or penalty against each other. This includes when checking their applicable object masks.

ENTRY name="Light_Tank" ... Flag_I_OBJMASK_MODERN_INF="100"

Flamethrowers are not listed as being weak vs. tanks, however Flamethrowers do less damage (66% normal) to various Tanks

ENTRY name="Flamethrower"
...
Light_Tank="66" Tank="66" Main_Battle_Tank="66"

Jumping in because this is a difficult topic and the thread/question shows up fairly high in search results for not just specifically making mods, but how RoN damage works in general.

Angzuril's answer covers most of what you need to know. A couple of important details to add:

1. Masks in balance.xml are bugged in RoN:EE, so don't rely on them for modding in EE (whatever values you put for or against masks are ignored and become functionally 100% instead - I don't know why). You can work around this by applying the modifiers directly to units - this open source utility makes it relatively easy to do so. Doing it via this method has a small margin of error (the average is perhaps ~1%, but ranges from perfect matches to around 5% in the absolute worst cases such as with submarine units), although it should be close enough for most practical purposes. I'm 98% sure the error is due to aggressive rounding during the calculation which causes it to ignore the "math rule" where multiplication order doesn't matter - it does if you round down each of resulting numbers after each/some step(s) of multiplication(!).

2. Masks are not only used in balance.xml. They have baseline "hardcoded" or hidden modifiers (these work in EE), and this is where the bulk of the rock-paper-scissors effect comes from. This is, for example, how Foot Archers counter Heavy Infantry independent of just range/speed/armor, and how Tanks actually do get favorable modifiers when fighting Modern Infantry, despite this interaction not showing up in balance.xml. The wiki page (linked below) will explain the values on a per-mask basis. For the total effect of a combination of masks on every unit, you can actually extract a huge 493x493 table from the game's save files with all the final modifiers after all the various bonuses and penalties (excluding basic damage and armor values) are applied. An easy way to access this is using this spreadsheet by Vanshilar (who wrote most of the wiki page) which has that extracted data with/without additional factors such as age differences (units deal bonus damage to units of lower ages).

More or less everything is explained in good detail on this wiki page but the number of interactions could fairly be described as "absurd", and so there is a huge amount of total information to cover if you're trying to know all of it!

Masks being bugged in EE is something where some of the results have been known by the community for a while (e.g. Horse Archers used to 2-shot Citizens pre-EE, but 3-shot them in EE, and nobody knew why), but something that I only discovered in 2020 when I personally got involved in the modding scene. So if working in EE, factor the discovery into any modding information / documentation published without that knowledge.