How do you tell a spelling mistake from a grammar mistake?

You can’t always tell which type of mistake it is. It might be one or the other, or even both or neither. Consider the following:

• This sentence has a spelling mistak.
• This sentence are ungrammatical.
• If the miss steaks are over there, perhaps it’s both.
• if the missed steaks are over here, though, that might be intentional.

But to answer your question: you need to first try to understand the intent. Then if the sentence doesn’t match that intent, you can look more carefully for causes. Typos that result in words not recorded in dictionaries would be obvious spelling mistakes; but if all the words in the sentence are valid English words, things become a lot more messy.

If you consider grammatical correctness to be independent of intent (which is perfectly legitimate to do), it becomes a matching exercise to see if any combination of the tenets of your chosen grammar can produce the sentence.

The trap, however, is that there are numerous English dialects. It’s even worse with rhyming slang such as “He’s on the dog”: “dog and bone” rhymes with “telephone”, then you drop words out. It’s not a spelling mistake if it’s really what they wanted to say. Likewise, “Who ya callin’ short?” can be considered ungrammatical, but it’s completely idiomatic in some dialects.

In summary:

• Spelling: pick a dictionary and try to find the words.
• Grammar: pick a grammar and check if the sentence confirms to it.
• General rule: check the text against the intent.

The distinction depends on what is on the mind of the person who is making the mistake: is that person mistaken about the relevant rules of English grammar, or merely about the spelling? When we see a mistake, we can usually think of a reasonable explanation of what led the person to make the mistake, and then classify the mistake accordingly. For example, it is reasonable to think that a person who wrote 'This iz the end' is relatively clear about how to structure this sentence and is only mistaken about the spelling of is. We would thus say this is a spelling mistake.

Sometimes, it is, however, difficult to be sure what was on the person's mind. Did the person who wrote 'Your the best' want to write 'You're the best' and was mistaken in thinking that you're can be spelled as your? If so, this would be a spelling mistake. But maybe the person really wanted to write your and mistakenly thought that English syntax permits combining your and the best in this way. In that case, the person would be making a mistake about the grammar. Or, perhaps, the person knows that this combination does not fit the standard rules of the syntax, but mistakenly thinks that 'Your the best' is some sort of an idiom that is an exception to the standard rules. That would be a mistake of yet another kind. If we are not sure what was on the person's mind, we cannot be sure how to classify the mistake.

There is thus no simple rule for classification of such mistakes that can be applied solely on the basis of what the mistake looks like; the classification is always based on our (more or less reliable) reconstruction of how the person was led to make the mistake.