When asking someone to smell something, is it correct to say “smell of this” or “smell this”?

Sherry, while I might tend to agree with the previous two comments in intent, in do not in tone.

“To smell of” something means the object has a similar smell. So “the coffee smells of chocolate” describes how the coffee smells, much like “he looks like a rock star” describer his appearance. In these examples “smell” and “look” are not being used as verbs in the simple sense. I will not get into the minutiae of grammatical terms here, as from your question I suppose that is not relevant.

For your needs “smell this coffee” is correct. Here “smell” is the verb and coffee is the object.

The first sentence literally asks the listener to make themselves have a personal scent similar to that of that particular coffee.