Why is "I didn't mean to, not miss it" incorrect?

The Purdue link link you cite in comments to your question isn't the final word on commas. However, your friend's quote could possibly fit item 7 in that list ("separate contrasted coordinate elements").

The expanded, non-negated form is more common:

• He meant to watch it, not miss it.

In response to a question such as "Did he watch it?", an ellipsed version might work:

• He meant to, not miss it.

Your friend's negated, ellipsed variant (containing a comma) is arguably of the same pattern, though it takes more effort to parse it. This is not to say your friend's sentence is idiomatic, only that it is not necessarily ungrammatical.

Alternatively, your friend could be providing more information, as in the second example of item 7 of your list:

• The chimpanzee seemed reflective, almost human.

Under this interpretation, your friend would first be saying that they "didn't mean to" do whatever it was they were denying. The part after the comma elaborates on what they didn't mean to do: they didn't mean to "miss it". This phrasing sounds colloquial.

Grammar is a low bar to cross; there are often weird and quirky interpretations that pull seemingly-ungrammatical sentences back into acceptability.

Regarding your second point, you've provided insufficient information for us to determine whether your friend's phrasing "conveys the incorrect meaning". If we don't know the meaning intended, we can't know whether any specific phrasing conveys it.

I'm finding it hard to get links on this offhand, I may find them later (or someone could point me in the right direction...) but on reflection I tend to agree the second phrase is ungrammatical (the first is fine, if hard to parse - as others pointed out it's hard to tell whether it's fine or not if we don't know the intended meaning).

Basically there are two uses for "to mean to". Either you are saying "I mean to do something", in which case you cannot have a comma between "to" and "do something". Or you are saying "I mean to", which is a fine standalone sentence where what it is you mean to do is derived from the context of the previous sentences.

In this case, "I didn't mean to not miss it" is grammatically fine, just a bit confusing and only context can tell us if it's the best sentence for the job or not. I guess if it has a flaw it's that usually you don't have a negative sentence after things like "I mean to" or "I want to", you say "I don't mean to" or "I don't want to" instead, but it's still fine to do in certain contexts or for stylistic reasons (like, "I fully intend to not fail the test this time!"). Similarly, you could have a negative phrase after a negative "I mean to" for stylistic reasons ("I didn't mean to offend them but, like, I didn't mean to NOT offend them either...").

On the other hand, putting the comma after "I didn't mean to" suggests we're no longer saying "I didn't mean to [do something]", but using the standalone phrase "I didn't mean to". Like you'd find in the exchange:

You didn't get milk like I asked!

Sorry! I meant to, but things got in the way.

Notice there "things got in the way" has nothing to do with what was meant, it's an additional independent clause (you could rewrite it as "I meant to get milk, but..."). In the case of "I meant to, not miss it", "not miss it" doesn't stand on its own as an independent clause, making the sentence as a whole nonsensical and indeed I'd say ungrammatical.