Should a prepositional phrase beginning with “of” that follows a plural noun be singular or plural?

Which of these is correct? “Types of citrus fruit” or “types of citrus fruits”? I suspect it is the first example, but I would really like to know what the grammar rule governing these situations actually reads. What determines whether the object of a preposition should be singular or plural when it modifies a plural noun? Any help would be appreciated. Thank you.