$A^{alpha}=(phi,vec{A})$ or $A_{alpha}=(phi,vec{A})$?

It's $A^mu = (phi, mathbf{A})$, no matter your metric convention.

Proof:

I can never remember the signs in the correspondence between $F_{munu}$ and the electric and magnetic field (especially since they depend on the metric signature), so instead let's look at the Lorentz force per unit charge

$$f^mu = F^mu{}_nu u^nu.$$

The part of $f^i$ proportional to $u^0$ will then be the electric field $E^i$:

$$f^i = F^i{}_mu u^mu = F^i{}_0 u^0 + dots = (partial^i A_0 - partial_0 A^i) u^0 + dots.$$

Now, no matter the metric signature we have $partial^i A_0 = -partial_i A^0$, because we're flipping one time index and one space index, so we arrive at

$$E^i = -partial_i A^0 - partial_0 A^i.$$

Comparting with the known formula $mathbf{E} = - nabla phi - partial mathbf{A}/partial t$, we see that the contravariant components of $A^mu$ are the potentials with no sign change.

As the existing answer says, it's $A^mu = (phi, mathbf{A})$. Here's a simple way to see that. In Lorenz gauge, the equation of motion is $$partial^2 A^mu = J^mu.$$ We also know that $J^mu = (rho, mathbf{J})$, and that the components of this equation are $$partial^2 phi = rho, quad partial^2 mathbf{A} = mathbf{J}.$$ There are no minus signs anywhere, so we must have $A^mu = (phi, mathbf{A})$.