How does one identify the antecedent of a relative pronoun?

Your question doesn't seem to be so much about what a clause modifies, but about how to determine the antecedent of a relative pronoun, (the noun referred to by that pronoun), in this case, "which." There are several considerations in making that determination.

• Proximity. The modifying clause has a strong attraction to the nearest preceding noun, which as you've spotted, makes "hardware components" a candidate.

• Syntax. Sometimes you will have grammatical help that will tell you whether the nearest noun is the antecedent. Consider the following version of your example:

  Complete virtual systems composed of virtualized hardware which are available for the installation of an unmodified operating system.

  The plural verb "are" makes the only possible antecedent the plural noun "systems."

• Punctuation. When a comma precedes a relative clause, that clause is non-restrictive, that is, informational only. If there's no comma, that clause is restrictive or defining, and restrictive clauses almost always modify the thing they immediately follow. (In AmE, restrictive clauses use the relative pronoun "that"; non-restrictive clauses, "which." This rule is unknown in BrE.) Let's look at two other versions:

  a complete virtual system composed of virtualized hardware, which is available for under $100. (non-restrictive, antecedent is ambiguous)

  a complete virtual system composed of virtualized hardware that is available for under $100. (restrictive, antecedent is "hardware")

• Semantics. Sometimes you have to look at the sense of the sentence to determine the antecedent. In your case, it's very unlikely that an operating system will be installed on hardware components, virtualized or not. Operating systems are installed on computers or computer systems. The problem with relying on the semantics is that you interrupt the flow of reading as your audience searches for the other clues. Rephrase to be kind:

  We can install an unmodified operating system on a complete virtual system composed of virtualized hardware components.

Note that silly things can happen if you rely on meaning:

I had a hat on my head, which went astray.

What got lost has to be the hat, but the sentence still conjures up the image of a missing head.