Is an automorphism a function or a group?

I think this

In class we've been thinking about automorphisms as a function that maps elements of one group to another.

is not quite right. Homomorphisms are functions that map the elements of one group to another (and preserve the group multiplication).

When the domain and codomain of a homomorphism are the same group and the homomorphism is bijective (so an isomorphism) then you call that homomorphism an automorphism.

If you have a group $G$ (say, $S_3$) then you can consider all the automorphisms of that group. The set of those automorphisms is itself a group (under functional composition) called the automorphism group of $G$.

You have been asked to find that group - start by listing its elements.

.

The automorphism group $operatorname{Aut} S_3$ is the group of automorphisms on $S_3$. So it’s a group, but its elements are automorphisms of $S_3$,

$$operatorname{Aut} S_3 = {σ colon S_3 → S_3;~σ~text{is a group automorphism} }.$$

Keep in mind that automorphisms are meant to be maps from a group to itself – and not to another group. It stems from the greek word αὐτóς, meaning “self” or “same”, so its something that shapes (grk. “μορφóω”) the group into itself, the same group. Automobiles are things that movable (lat. “mobilis”) by themselves. Most terms in mathematics actually have meaning.