Your first two examples are very different from the third example in more ways than just saying that the third is an arithmetic question while the first two are not.

In the first two examples you are asking about conventions regarding notation. Given two functions $f$ and $g$ on a set $X$, you are asking why the composition $fcirc fcirc g$ is denoted $f^2g$ rather than $2f+g$. But again, these are questions of notation. If I use $f^2g$ to denote $fcirc fcirc g$, I am not at all suggesting that there is some kind of multiplication of numbers involved. Similarly, if I were to use $2f+g$ to denote this function instead then, again, this does not mean that addition of numbers is involved. I am simply choosing to denote $fcirc fcirc g$ in a different way.

So your question is: Why is the notation $f^2g$ more common than $2f+g$? The answer is that people often use the addition symbol $+$ to denote binary operations that are commutative: $x+y=y+x$ for all objects $x$ and $y$. Since composition of functions is not commutative, people usually don't use the addition symbol in this way. On the flip side, people do use the multiplicative notation for general operations that are not necessarily commutative. So this is why $f^2g$ is more likely to be used than $2f+g$.

Now, your third question is not a notation question. It is a mathematical question that is asking about something very different from the first two questions. You are asking why the number of outcomes of flipping 3 coins is 8 and not 6. For one thing, you can count them:

HHH
HHT
HTH
HTT
THH
THT
TTH
TTT

So perhaps the real question is the following. Suppose we have a task that can be broken into two steps. Say there are $m$ ways to do step 1, and $n$ ways to do step $2$. Why is the total number of ways to do the whole task $mn$ and not $m+n$? This question is equivalent to the following:

Suppose $|A|=m$ and $|B|=n$. Then why is $|Atimes B|=mn$ and not $m+n$?

This is an equivalent question because I can think of $A$ as the set of ways to do step $1$ and $B$ as the set of ways to do step 2. So $Atimes B$ is the set of ways to do the whole task since I can represent doing the whole task as an ordered pair $(a,b)$ where $a$ comes from $A$ and $b$ comes from $B$.

The proof that $|Atimes B|=mn$ is not too hard. Write $Atimes B=bigcup_{ain A}X_a$ where $X_a={(a,b):bin B}$. If $aneq a'$ then $X_{a}cap X_{a'} = emptyset$. So $|Atimes B|=sum_{ain A}|X_{a}|$. For any $ain A$, there is a clear bijection between $X_{a}$ and $B$ in which one sends $(a,b)$ to $b$. So $|X_{a}|=|B|$ for all $ain A$. So $|Atimes B|=sum_{ain A}|B|=|A|cdot |B|=mn$.

Your example with coins had three steps instead of two, but you can generalize to any number of steps using induction. In combinatorics, this is called the "multiplication principle". See: https://en.wikipedia.org/wiki/Rule_of_product