How can the Normal Force on an upside down roller-coaster car point upward?

Note that it is not always the case the positive is up and negative is down.

You are most likely just misunderstanding their sign conventions. For them positive is most likely inward towards the center of the circle, and negative is outwards away from the center of the circle.

Regardless of the sign conventions though, you just need the normal force to be pointing inward towards the center of the circle. You are correct in assuming that the normal force cannot point away from the center here. So if your work concludes that you need some extra additional outward force to keep the carts on the circular path, then you know that the ride will fail with just carts coasting on the track.

---

From comments:

This still leaves me perplexed. So if $N=m(v2/R)−mg$, then for N to be positive means N is in the same direction as the centripetal force, right? But the centripetal force is directed outward. So this still seems to be telling me that the normal force has to point outward. But I don't see how that is possible, given the definition of normal force.

This is a common introductory misunderstanding. There isn't "the centripetal force" like there is "the Normal force" or "the force of gravity". "Centripetal" is a direction, just like horizontal and vertical, up and down, left and right, etc. It is always true that $mathbf F_text{net}=mmathbf a$ by Newton's second law. When you first see this usually we break this into horizontal and vertical components (Cartesian coordinates) $$sum F_x=ma_x$$ $$sum F_y=ma_y$$

But we can also use Polar coordinates in terms of a radial component $F_r$ and a polar component $F_theta$ $$sum F_r=ma_r$$ $$sum F_theta=ma_theta$$

And you can show that for polar coordinates in circular motion, we must have that $a_r=v^2/r$, where a positive $a_r$ indicates acceleration towards the center of the circle. So we end up with $$sum F_r=frac{mv^2}{r}$$

So now we look at the problem. At the top of the coaster we have gravity pointing towards the center of the circle (positive), and we will have the normal force pointing towards the center of the circle (positive). This means that our net radial force component is $$sum F_r=mg+N=ma_r=frac{mv^2}{r}$$

Notice how there are only $2$ forces here! The normal force and gravity. There isn't an additional third force that is "the centripetal force". Rather, at the top of the loop both the normal force and gravity contribute to the net centripetal force.

If $sum F_r>0$ then the net radial force component points towards the center of the circle and is centripetal. For the coaster to work properly this needs to be the case: circular motion needs to have a net centripetal radial force (as you can see, $mv^2/r$ can never be negative. i.e. once we assume circular motion, $sum F_r>0$ is always true).

I think I'm just terribly confused about the role of the normal force here. The solution states the velocity has to be large enough to make the normal force positive. I don't understand what a non-positive normal force would even look like here, so I'm confused. Unless we're just saying N can't be zero. But in that case, why not just say that the centripetal force has to exceed the car's weight? I'm just unclear what work the normal force is doing in the reasoning here.

You are right, this is a little confusing. Technically here the normal force acting on the carts from the tracks at the top of the loop can only act towards the center of the circle. In reality this specific force can only be positive (towards the center of the circle) or $0$ (the carts lose contact with the tracks). It cannot be negative.

But it helps to shift your perspective here... Instead of thinking about $N$ as this specific normal force, instead think of it as "the extra force needed to keep circular motion going on this vertical circle in a gravitational field". Then this $N$ could certainly be negative. If the velocity is too small, then we will need some extra force point outwards to keep $N+mg=mv^2/r$ valid. Maybe we have put rocket thrusters on the top of the carts that can apply an outwards force when the cart is upside-down and moving too slowly.

And indeed this is a nice trick for many physics problems. Say "what would this extra force need to be in order to make this motion possible?" If you find "ah, this extra force needs to point outwards" then your next thought would be "oh, but there isn't anything in this system that could do that." The point of this problem is for you to look right at that interface. Right when you switch from $N>0$ to $N<0$. All the math knows is that you are trying to make the constraint $N+mg=mv^2/r$ valid. The math doesn't know that there isn't anything stated physically in the problem that would allow $N<0$.

The given equation assumes that down is positive and that the force from the track on the car is helping supply the required centripetal acceleration. If the car goes too slow, the normal drops to zero, gravity supplies the acceleration, and the car drops away from the track. Note that there is an equal and opposite reaction normal force which the car exerts on the track.

Note that in the equation

$$ N+mg=mv^2/r $$

the (downward) weight force $mg$ has a positive sign, so positive $N$ means that the normal force and the gravitational force are parallel, rather than antiparallel.

Some students learn the non-fact "$g$ is negative," and sometimes (rarely) that convention is useful. But you can see that's not the case here by considering the motion if the roller coaster just barely comes off the track, so that $N=0$ but the trajectory is approximately the same. The right-hand side $mv^2/r$ must be positive, so the convention in use here must be positive $g$, and the authors have chosen a coordinate system where the positive direction is "down."

For that matter, you know that the centripetal ("center-pointing") force $mv^2/r$ points towards the center of the motion --- another vote for positive-down. I see in a comment that you may be thinking of the fictitious centrifugal ("center-fleeing") force instead.

I think you have a confusion about signs, which is not helping, and which someone has addressed in another answer. (In the problem, the sign of the normal force is positive when the force points inward. This makes sense because the normal force we are considering here is the force that the track exerts on the car.)

But beyond that, I think there's a bigger picture about the general structure of this problem that it's important to keep in mind. Our approach looks like this:

1. We start out by assuming that the car goes all the way around the loop. (This may not turn out to be true! We will need to verify this assumption later.)
2. We then calculate the downward force that the track is exerting on the car at the top of the loop, as a function of the starting height, under the assumption that the car makes it around.
3. We then calculate the zero of that function (that is, the starting height for which the force is zero.)
4. Then we NOTE the following CRITICAL FACT about the physical interpretation of that function: If the force we compute is positive or zero, everything is great. The car will never leave the track.(*) But if the force we compute is NEGATIVE, that does NOT mean the track exerts an upward force on the car! Instead, it means our original assumption was false: the car does not stay on the track in this situation. Something else happens, which requires a different model.

The mathematical function we have computed -- N as a function of h -- sometimes has a negative value. But the physical quantity we are thinking of -- the downward force on the car by the track, as a function of h -- never has a negative value. The actual physical quantity we care about is undefined(**) in the cases where the mathematical function would be negative. In those cases the car has already left the track by the time we would be computing the force.

This kind of general approach to physics problems is not uncommon -- make an assumption, do some math under that assumption, then check at the end whether our assumption was valid -- so I hope I have shed some light on it here.

(*) This depends on another assumption which the solution does state, and which I am taking for granted: The highest point of the loop is the "most difficult". The car will make it all the way around the loop if and only if it makes it past the highest point without leaving the track.

(**) Well, you could plausibly argue that it's zero, since when the car and the track are not in contact the force exerted is obviously zero. But I think it's better to think of the quantity we're looking at as being something like "the force exerted in the model we're looking at", and call it undefined in cases where the model we're looking at just doesn't match reality. In any case this doesn't affect the main point, that the mathematical function we've calculated only reflects the physical quantity of interest over the range where our model is valid.

Here is an answer that fully address the conceptual issue.

The point is simply that in order for something to go around in a circle with radius $r$, at every point where it has speed $v$ (in the reference frame where the circle is stationary) it must simultaneously have acceleration $v^2/r$ towards the centre of the circle. If that fails at any point, then it will not continue going in that circle!

An ideal roller coaster car has only two things that cause it to accelerate, namely gravity and the track. We assume gravitational acceleration is just a constant $g$ downwards. If the car is on the circular track at angle $t$ from the topmost point, then gravity contributes $g·cos(t)$ acceleration towards the centre of the circle. If $g·cos(t) ≤ v^2/r$, then the track must provide the remaining acceleration towards the centre of the circle to keep the car on the circle, and it indeed can! If however $g·cos(t) > v^2/r$, then gravity is providing more acceleration towards the circle centre than needed to keep the car on the circle, and the track must pull the car away from the circle centre if you want the car to stay on the circle, which is impossible if the track can only exert a force towards the car.

This should address both the conceptual difficulty as well as explain rigorously why the 'worst case' is at the top of the circle if we ignore friction. Firstly, $g·cos(t) ≤ g$ with equality at $t=0$, so the acceleration towards the circle centre due to gravity is greatest when $t = 0$. Secondly, if the car remains on the circle, then the car's speed is lower when it is higher up because the track's force on it is perpendicular to its trajectory and hence does not change its speed, but gravity slows it down when going up and speeds it up when going down. Thus the acceleration required for the car to remain on the circle is least at the top.