Are there rules about using "that" to join two clauses?

In the example you gave, "I was not joking" is a subordinate clause. One way to think about this is that there are two sentences

1. He will understand X.
2. X = I was not joking.

"that" is not exactly a conjunction as Vincent said. A conjunction joins two independent clauses together, but here 2. is sort of the object of 1. Any time you want to use a clause as the object of a sentence, you can optionally precede it with "that." However, if you want to use a clause as the subject of a sentence, it will always be preceded by "that"

That I was not joking will be clear to him.

*I was not joking will be clear to him.

Sometimes clauses modify nouns. In these cases, they're called relative clauses, and what pronoun appears before the clause varies. One of the most important factors of whether or not "that" or "which/who" precedes the clause is whether the clause is "restrictive". To modify somee examples from Wikipedia:

(Restrictive) The house that Jack built fell down.

(Restrictive) The house __ Jack built fell down.

(Non-restrictive) The house, which we all thought was in fine condition, fell down.

In the first two examples, the relative clause picked out a particular house from all possible houses. In these sentences, there is a tendency to use "that" or nothing. In the third example, the particular house was already determined, and the relative clause simply introduced extra information. These relative clauses tend to be preceded by a wh- word.

For relative clauses: If the missing element (which is said to be "relativized") of the relative clause is the subject, "that" is obligatory, otherwise, it is optional:

Peter tasted the wine (that) his mother bought for him

Peter tasted the wine that/*() had cost only 12$.

If your sentence is unclear or difficult to parse for the listener/reader without that, then insert it; otherwise, feel free to omit it.

The understanding of your audience is always priority #1.

"That" is a conjunction in this type of sentence. (In "the cat that jumped over the wall", it is a relative pronoun.) It is in general OK to leave out the conjunction "that" now and then, as long as no ambiguity arises. Everybody does it all the time, even in formal style, though it happens more often in informal writing and speaking. It is possible too in German, though not in Dutch, as you probably know.

*She believed the judge, who was older than her father, was putting the moves on her.

In this sentence, the reader is led onto a false scent because "that" is omitted after "believed". This is a serious mistake. The reason why it is bad here is that "to believe" can have an object, so that the reader thinks "the judge" is simply the object of the verb, instead of the subject of a dependent clause, which in fact it is. When he finds out he is getting the sentence wrong, he needs to read back to repair the damage.

They say she was unable to realize her husband had left her.

This sentence is doubtful, because "that" is left out two times in a row. It makes the sentence a bit too loose, a bit messy: though the reader will probably get it right in one go, he will still need to spend a little more energy on it than necessary.

As most of the other answers here point out, including that in a sentence like "He will understand that I was not joking" is optional. But I'd like to offer some examples where that, despite performing the same basic function that it does in "He will understand that I was not joking," either is crucial for sense or vastly improves the flow of the sentence. My examples fall into four categories.

Awkwardness/Unnaturalness

I appreciate [that] CAN-SPAM creates a single set of rules for an industry previously struggling with a hodgepodge of 34 existing state antispam laws.

Long Beach Opera recognizes [that,] in order to preserve and strengthen opera's significant in American culture, it must continue to develop new approaches in programming.

He claimed [that] customers complain [that] the new product is heavier than the iPad2 and lacks any major improvements from the old version.

In each of these sentences, readers can supply the implied that themselves, but with more effort than the author should require them to make.

Parallelism

But he insists [that] the charges are false, and that he's sending as many e-mail messages now as he did before the suit was filed.

Google says [that] new movies will be $3.99 and older ones $2.99, and that most rentals will be for 24 hours.

Political consulting firm Election Data Services, which tracks voting-machine usage nationwide, estimates that 40 percent of voters will cast ballots on touch-screen machines this year, and [that] about 42 percent will cast them either on optical-scan machines or on traditional paper ballots.

In these sentences, using that in one branch of a parallel construction calls for using it in another, for balance and clarity.

Ambiguity: Modifier Placement

You’d think [that] by now [OR HERE?] I’d be used to Windows’ mystifying default behaviors.

The ISPs promise [that,] under the new agreement, [OR HERE?] content providers will not be given your subscriber information such as your name, address, telephone number, and e-mail address.

A spokesman for the ISP says [that] online [OR HERE?] criminals are stealing AOL users’ account passwords and using the profiles to advertise their sleazy wares.

In these sentences, that could fall in either of two plausible places, and the meaning of the sentence would change somewhat depending on its placement.

Ambiguity: Attribution

Microsoft says [that] the transition to a 64-bit OS on the desktop will take a few years, and [HERE, TOO?] typical users will be on the far end of that distribution curve.

AOL says [that] it will offer online backups, but [HERE, TOO?] the feature won’t be included in the July 13 beta.

Providers of the service say [that] the bugs have been all but exterminated and [HERE, TOO?] the service is ready for prime time.

In these sentences, the ambiguity relates to whether both clauses are assertions made by the named source, or whether the first is from that source while the second is a follow-up remark made by the author.

There are several different kinds of that in English (besides the ordinary demonstrative this, that, these, those). All of them are used in linking clauses, but they link different kinds of clauses, and follow different syntactic rules.

These that's are not, however, conjunctions; they are Complementizers. Complementizers are one of the parts of speech that have been discovered since the list of POS was drawn up in the 5th century. Not surprising, really; why should we use Roman science in the 21st century?

There are three different kinds of subordinate clauses, depending on what the clause is acting like:

• Adverb clauses normally contain a subordinate conjunction.
  When I arrived, he was already here. He was already here when I arrived.
  That used to be a possible complementizer for all tensed clauses,
  as in Chaucer's Whan that Aprille ... hath perced ... 'When April has pierced'
  or Andy Griffith's before that we set up the tent in What it Was, Was Football

  That is used as a complementizer in tensed adverb clauses in Modern English only in local dialects like Griffith's North Carolina; that's what makes Chaucer sound strange. It is, however, used in tensed complements and relative clauses.

• Noun clauses (also called Complements or Verb Complements) act like nouns, as subject or object for a predicate. The clauses are boldfaced below. There are four varieties, each with its own complementizer:

  Subject: For me to leave early would be a mistake. (infinitive complementizer)
  Subject: Leaving early is not recommended. (gerund complementizer)
  Subject: What he told me is not for publication. (embedded question complementizer)
  Subject: That I have to leave early is unfortunate. (that complementizer)

  Object: They told me to leave early. (infinitive complementizer)
  Object: I hate his playing the piano at all hours. (gerund complementizer)
  Object: I didn't hear what she told you. (embedded question complementizer)
  Object: They told me that I had to leave early. (that complementizer)

  Note that one of the complementizers is that, for tensed clauses.
  This that may be deleted, provided it is not the first word in a sentence.

• Adjective clauses modify nouns or noun phrases. There are two kinds: relative clauses and NP complements; that occurs in restrictive relative clauses, and also in NP complements.
  Relative: the man that came to dinner (that is subject; cannot be deleted)
  Relative: the man that I saw in the station (that is not subject; may be deleted)

  NP Complement: the rumor that I have to leave early (NP complements are just complements that modify nouns formed from verbs or predicate adjectives.)
  It's possible that she needs a job ~ the possibility that she needs a job
  That in NP complements can be deleted, but not nearly as often as in verb complements.