Should retail investors study exchange rules?

I don't think that US retail investors or active traders need to study the details of the many NYSE/FINRA rules that govern investing and trading other than for or the ones that apply to your trading, you need to understand what the rules mean on that level.

For example, yesterday someone posed a question about ex-divided cancellation of market orders. In the US, FINRA rule 5330 deals with the "Adjustment of Orders". All I need to know is that my stop order will be adjusted down on the ex-div date or that by broker preference, they may be cancelled. If the latter, all I must do is place the order again. Reading the rule 500 word explanation of the rule is wasted time.

Another example is standard Reg T margin and Pattern Day Trading margin. All I need to know are the basic outlines and limits. My broker explains that in a fraction of the words that FINRA does and in simple understandable language rather than as a detailed list that includes Rule 1, section A, sub categories (a) through (e) through Rule 11, section A to E, each with sub categories.

An investor should understand what they are investing in, to the level of being able to make informed decisions.

There are many things to understand - multiple careers worth! - from the basics of cash management, personal tax implications, risk profile of different asset classes, and on and on ad infinitum. An amateur can only learn so much out of such a broad possible knowledge base, which creates a general rule you should consider: don't invest in what you don't understand. What this means is that, at whatever level you are able to learn, don't invest one step further than that.

So if you don't understand why a share has value, either don't invest in shares, or learn about them first. If you don't know what a 'future' is, don't trade futures.

The question here is: do you need to read all NYSE rules before investing in shares on the NYSE? As you say, many of those rules will be directed at the institutions on the exchange, either institutional traders, or the listed companies. So those are not so relevant. But how will you know what's relevant to you? By using a non-biased source of information to help you with the basics. Reading the rulebook is likely less valuable to you than finding a good introductory guide to investing, and starting your own personal financial education.

From the tone of your question, I suspect you are quite new to retail investing, and therefore I suggest that you take a deep breath and educate yourself some more before getting into it. For most people, educated investing is an advanced goal, beyond picking a few basic index / mutual funds with their 401k provider (or similar). That's completely fine, and is far better than the alternative of investing in something you don't understand.

So, when would you need to read the rules explaining how your broker must behave when you place a stop-loss order? Before you place a stop-loss order! When should you learn about rules governing futures trading? Before you trade in futures!

If you feel your current financial institution is not giving you unbiased advice, you should solve that problem, before you begin investing. Feeling comfortable in what you are doing will help you to align your personal financial goals, with your investment choices. Investing when you feel uncomfortably uninformed, is likely an indication that you have taken on more risk than you understand.