When does a pronoun function as demonstrative or relative?

In order to simplify the discussion, I will change your examples so that the phrase of interest is integrated into the syntactical structure of the sentence.

^{This is not the case in your examples; the that-phrases there are suplements, meaning that they are only semantically, but not syntactically, linked to another part of the sentence, the so-called anchor. For more on supplements and anchors, see e.g. this answer.}

Here they are:

[1]  i  His bold program was [that of sending a man to the moon].
      ii  The theory I'd like to discuss is [that which is most famously associated with Toynbee].

For contrast, let's also record a relative construction:

[2] She liked the gift [that I got her].

So, how can we tell that the that in [1] is different from the that in [2]?

One strikng difference between [1] on the one hand, and [2] on the other, is that the that in [1] inflects for number, but the that in [2] does not. For example, here is the plural counterpart to [1i]:

His bold programs were [those of sending a man to the moon].

But if we pluralize gift in [2], that does not change to those (indeed, if we change it, the result is not acceptable grammatically):

She liked the gifts [that/*those I got her].

Another difference is that in [2], the that can be either omitted or replaced by a wh-word (namely which), whereas the that in [1] cannot.

Still further difference to notice is that in [2], the bracketed expression is a clause, while in [1i] and [1ii], it is not.

Analysis of [1]

In fact, the bracketed expressions in [1] can serve in the same roles as typical NPs, in particular as the subject or the object of a clause. At the same time, this and that (and their plural forms,these and those) on their own can function in those same roles. Moreover, this, that, these and those mark nouns as definite, and they are mutual exclusive with the articles. Based on arguments like these, CGEL concludes that the bracketed expressions in [1] are best analyzed as noun phrases (NPs) whose head is the demonstrative determinative that.

Analysis of [2]

As far as that in [2], CGEL argues that it is not a pronoun (relative or otherwise), but that to rather belongs to a seprarte lexical category of subordinators. CGEL gives four reasons for such an analysis (pp. 1056-1057).

First,

If that were a pronoun, or pro-form, its use would be much wider than that of the uncontroversial relative pronouns, or indeed of any pro-form at all in the language.

Indeed, a putative pronoun that could function as any number of wh-words: as who (They gave the prize to the girl that spoke first), which (Have you seen the book that she was reading?), when (He was due to leave the day that she arrived), and many others. It would also appear in a variety of constructions where no wh-word could replace it (It wasn't to you that I was referring; She seems to be the happiest that she has ever been).

Second, in relative constructions with that, there can be no 'upward percolation'.

There are no that relatives matching wh relatives with a complex relative phrase:

[72]  i  a. the woman [whose turn it was]    b. *the woman [that's turn it was]
         ii  a. the knife [with which he cut it]    b. *the knife [with that he cut it]

If that were a pronoun we would have to stipulate that it has no genitive form, and that it never occurs as complement of a preposition - or rather that when it is complement of a preposition the latter must be stranded, for the knife that he cut it with is quite grammatical. The severe restrictions here stand in sharp contrast to the remarkable versatility of the putative pronoun that illustrated [above]. In the analysis where that is a subordinator the ungrammatically of [72ib/iib] is predictable. Subordinators do not inflect and must occupy initial position; there is no relative word and hence no possibility of the relative feature percolating upwards into a larger constituent.

Third,

That relatives are always finite [i.e. never non-finite], as are the declarative content clauses introduced by that. Note, then, that we cannot insert that into non-wh relative infinitivals like a knife to cut it with - cf. *a knife that to cut it with. If that were a pronoun this would be a special fact needing explanation, but under the subordinator analysis it is exactly what we would expect, given that that is a finite clause subordinator.

Finally, we note that that is in relative constructions is largely omissible.

As we have noted, that can be regarded as very largely omissible in relative clauses in the same way as in declarative content clauses. The conditions under which omission is prohibited are not the same in the two cases, but in both they have it in common that they are related to the need to mark explicitly the beginning of a subordinate clause under certain structural conditions. And in both cases, moreover, that is more readily omitted in simple structures than in complex ones. There is no pro-form in English that is systematically omissible under remotely similar conditions.

Since that in [2] is not a pronoun, it cannot be an anaphoric element—unlike wh-words such as which, it cannot be anaphorically linked to an antecedent elsewhere in the sentence. Instead, the anaphoric element is realized by a gap, symbolized as follows:

This is the letter_i [that ____i drew our attention to the problem].

In [the above sentence], where that marks the clause as subordinate, the subject position is empty, but there is still an anaphoric link to the antecedent letter, which we indicate by attaching the same index to the symbol marking the gap. (CGEL, p. 1037)

In the above sentence, the relativized element (letter) is the subject of the subordinate clause. In contrast, in sentence [2], the relativized element (book) is the object of the subordinate clause. Using the system we just introduced, sentence [2] would be marked as follows:

She liked the gift_i [that I got her ____i].