What is the difference between the action of $Z$ and $exp(-i Z t)$ on a state?

Effectively, the Z operation (represented by the Pauli $Z$ matrix) applies a rotation about the $Z$-axis. As you note, rotations can also be written in the form $e^{-i Z t}$. To see that, you can use a trick pretty similar to the one used to derive Euler's identity ($e^{i theta} = cos(theta) + i sin(theta)$) to rewrite the Taylor series that you quoted in your question.

In particular, to derive Euler's identity, you can use that $i^2 = -1$ to separate the even and odd powers of the series $e^x$ and identify the series for $cos(theta)$ and $sin(theta)$. Since $(Zt)^2 = ?t^2$ , you can pull the same trick with $e^{i Z t}$:

$$ begin{align} e^{i Z t} & = sum_{j = 0}^{infty} frac{(iZt)^j}{j!} & = ?left[sum_{k = 0}^{infty} frac{(-1)^k}{(2k)!}t^{2k}right] + i Z left[ sum_{k=0}^{infty} frac{(-1)^k}{(2k+1)!}t^{2k+1} right] & = ? cos(t) + i Z sin(t) end{align} $$

Thus, at $t = pi / 2$, $e^{i Z t} = iZ$. Since $i$ is an example of a global phase, evolving under $Z$ for time $t = pi / 2$ gives you the same unitary transformation as $Z$. Indeed, you can check the equivalence of the two matrices using QuTiP:

In [1]: import qutip as qt
In [2]: import numpy as np
In [3]: Z = qt.sigmaz()
In [4]: -1j * (1j * Z * np.pi / 2).expm()
Out[4]:
Quantum object: dims = [[2], [2]], shape = (2, 2), type = oper, isherm = True
Qobj data =
[[ 1.  0.]
 [ 0. -1.]]

You can also check that the quantum program Z(q); does the same thing as Exp([PauliZ], PI() / 2.0, [q]); using the AssertOperationsEqualReferenced operation:

open Microsoft.Quantum.Arrays;
open Microsoft.Quantum.Diagnostics;
open Microsoft.Quantum.Math;

operation ApplyZ(qubits : Qubit[]) : Unit is Adj + Ctl {
    Z(Head(qubits));
}

operation CheckIfOperationsEqual() : Unit {
    AssertOperationsEqualReferenced(1,
        ApplyZ,
        Exp([PauliZ], PI() / 2.0, _)
    );
    Message("Operations are equal!");
}

Try it online!