How can I stop playing to win?

The way the tell the difference is not by looking at mechanics but rather what is the focus.

Wargames have specific scenarios that you are dealing with. That the scenario have a defined end either in time or victory conditions. More sophisticated wargames will have a campaign that will generate scenario to play out. The result of which influences the direction of the campaign. But even then the campaign generally has a point at which victory is declared.

Roleplaying in contrast resolves around the experience of playing a character in an imagined setting. A virtual reality or holodeck done with pen, paper, dice, and imagination. The imagined setting and the character are adjudicated by a human referee. The rules may appear similar to a wargame but in a tabletop RPGs they are a tool used by the referee and player to decide the outcome of specific actions rather than the point of the game itself.

The example I typically use is SPI's Freedom in the Galaxy vs Traveller

Both games feature individual characters, have options for fighting out combat with units, and managing military campaigns.

The crucial difference is that the SPI games revolve around specific scenarios. Freedom of the Galaxy is about presented as a game where a Rebel Player attempts to liberate the Galaxy from the Empire players. Traveller in contrast is about creating a fictional science fiction setting for players to adventure in. That setting could feature the player as rebels liberating the galaxy from an empire. Rather than focusing victory conditions, it is focused on the experience of being there as rebels trying to liberate the galaxy.

Another example is SPI's War of the Ring versus ICE's Middle Earth Roleplaying. Both games have a similar range of mechanics (characters, combats, battles, etc). But because of their focus War of the Rings is a wargame while Middle Earth Roleplaying is a tabletop RPG.

Because of this difference, the key point to get across to wargamers trying RPGs for the first time is the idea is figure out what interesting for you to experience. Unlike a wargame scenario where you need to be efficient as possible to achieve victory, the players need to think of what most interesting to try.

It may sound simplistic but I started playing wargames and RPGs in the 70s. The most likely issue, if one occurs, is the wargamer player trying to "win". Once I explain the above and get them to understand it. They tend to relax and get more into the spirit of being inside of a setting as that character. Players will still try to make the most of the rules but because of all the possibilities that could occur it becomes a more natural optimization rather than a laser like focus on specific objectives.

Goals vs. Win Conditions

Most RPGs have a goal in play, but not a win condition (a few do, though). Consider the difference between a game about Greek warriors and making a story about their struggles vs. a game about Greek warriors played for tactical success. Figuring out what the Goal of the game is, tells you what good play for the game will be like.

The biggest hurdle, not just for wargamers, but everyone, is that many of the game rules are designed (...or, in some cases...un-designed) to potentially fulfill different roles in this way, so you may need some discussion up front to be clear on exactly what good play looks like for this particular game group.

Clarity on Game State

One of my friends who is a big wargamer pointed out that key point of play in that hobby is clarity on "the game state" - where every character stands, what the conditions are, etc. Clarity is necessary to make better decisions.

RPGs on the other hand... are often more fluid. The reality is that much of what happens in an RPG is because the group consents to that being true. This makes it more tricky to handle - a good game has a clear focus on goals so everyone knows what parts demand clarity vs. what parts are not that important. If you're playing a game about soap opera style drama, the important things are what everyone's relationships are to each other, and who's lying to whom, etc. but not what they're carrying or how strong they are. Spending too much time focusing on things that aren't relevant to the game disrupts play.

Genre Expectations Matter

RPGs often give you the option to do many things, but good play is often bounded by expectations within a given genre. For example, if you're playing a game about Golden Age Superheroes and you start destroying buildings full of innocent people just to kill the bad guy - that's bad play.

It's worth noting that some expectations are supposed to never be violated, and some are to be stepped over, but only as a Big Deal (TM) - and some are not even in play at all. Again, a good understanding of what the game text advises and a discussion among the group helps a lot.

Optimize/no don't optimize

Many RPGs suffer from a problem of having strong optimization rules built in, but then the advice around the game is to not use it. This is generally a bad game design set up, but it does make up a lot of games trying to cater to two different crowds. Talking to your group and the GM to figure out what they're looking for is key, but it's also worth noting that several games with this problem also punish poor choices severely - a character that's too far below the expected curve will die quickly, or be a 3rd rate character the whole campaign - as you are often locked into particular builds early on.

I don't really have a simple answer to that, as it's both common and endemic to many designs and often groups are terrible at really getting into how far the expectation is to go one way or another with it.

I too come from a wargaming background before I discovered P&P RPG, so I've been through this as well. A few pointers.

• It's a group thing. Wargaming is often one on one (or if you have a lot of time on your hands, multiple VS multiple). In RPGs you have several players who play alongside a DM. Note that you play WITH the DM, and not AGAINST the DM: roleplaying is a group thing. Your fellow players are not your rivals (unless you play something like Paranoia or Black Crusade), and you should not try to beat them to the reward or impede them in some way, since messing with your fellow players will sow the seeds of distrust and may end up destroying your group. Don't forget the social aspect: in RPGs everyone's on the same side, and should have fun together.
• Roleplaying is important. Some wargamers give their armies entire backstories, write stuff about their characters and paint elaborate banners displaying their history. Some wargamers do not do this, but that's fine. With roleplaying however it is pretty much required that you think about who your character is and what they do. If you're familiar with the concept of Your Dudes, it's kind of like that. And remember that combat is frequently NOT the best answer to a problem.
• The importance of one VS the importance of many. In tabletop wargames you have an entire army consisting of many units and characters, while in RPGs you usually (though not always) have only one character. This means that your character is NOT expendable, and unless you somehow have access to large groups of minions who do not object to being thrown into the meat grinder, your character's health is VERY important.
• The presence of the DM. The DM is, if they're not terrible, NOT your enemy. But they're not your friend either: they're here to challenge you. Sometimes with combat, sometimes with sneaking into a castle, exploring a spaceship or outrunning Shoggoths. The DM presents you with a scenario and has you come up with a solution. This leads to the next point...
• Not everything is set in stone, or the difference between Roleplaying and Rollplaying. The rules in wargaming are (I do hope) very clear: if you have a disagreement you roll a die and continue playing. In RPGs combat is well-defined, but there's a lot of out of combat stuff that there are often no rules for. Say that you need to get into a castle to obtain a macguffin, what do you do? Fight your way in? Have your wizard teleport you in? Sneak in during the dead of night? Strip to your underpants, paint half of yourself green, wear a feathery boa and act like you've got one hell of a hangover and mutter something about drinking heavily as you stagger into the castle past the perplexed guards? Or just clutch your butt and tell the guards that you really need to use the bathroom as you storm past? Good roleplaying has you thinking outside the box. Houserules are also more common in RPGs than in wargames, but they ALWAYS come at the DM's discretion.

• The effects of death. If you lose a wargame you shrug, pack up and have your army ready for your next game. In roleplaying this is not the case: if your character dies and is not raised from the dead, he's dead. This means that you can't play that character again (unless you're being passive-agressive with the DM and make Larry the Fighter the successor to Barry the Fighter, but I advice against this) and have to make a new one. This will eventually happen, so be prepared for this both mentally and with a new character at the ready (though it is smart to keep your stack of sheets with backup characters out of the DM's sight, many do not take kindly to the implication)

• It's not all about you. Sure, your character may be a badass, but there's an entire party of badasses. Know when you can shine and when you should not: let them do their thing and they will (hopefully) let you do yours.

• Build your character with care, and check with the other players. There's a difference between building a reasonably powerful character and minimaxing. The latter is often frowned upon, expecially if either it comes at the expense of the rest of the party OR it breaks the game system. But don't be afraid to play ruthless in combat: a smart DM won't be either. A smart dragon for example will not land and have Conan and friends chop his legs off: he'll just keep strafing you and breath fire at you until you die. Think of it like your first fight with Kalameet in Dark Souls.
• Most important: What does the group want? Do they want dangerous games? Silly games? Dark games? High powered games? Sci-fi? Fantasy? Horror? Superheroes? Cartoon characters? Wuxia? Star Wars? Talk to everyone before you get started, and they'll be forever grateful. The Same Page Tool is useful to gather up the expectations of the players as well.
• The DM is always right. If the DM makes a decision, you follow it. Don't grind the game to a halt by arguing with the DM unless it is something that makes you really uncomfortable: if you have problems with the rule you talk to your DM after the session's over.

I hope this is at least some help to you, and if you have more questions I'd love to hear them.

I would say that it's a very hard question because there are many ways to play RPG. From my experience, there are also various ways to play wargames, I personally played Warhammer with friends and we tended to make balanced games, of battles with heavy history components, and we had a lot of fun, but as you say, some tend to powergame and it's kind of contagious for wargaming.

To move from wargaming to RPG, you should be aware about what kind of RPG you are moving to. Some group of players enjoy powergaming, focus on tactical combat and gaining wealth and experience, some other focus on interesting plot or roleplay... If you really manage to grasp what the game will be about, you have a good start (I'll assume you agree with that goal).

You should also notice that I talked about a group. You are playing with the other players, not against. This should include the Game Master, altogether, you are about to build a story where the group is "the heroes", and most group try to work together to build a story as epic/funny/frightening/mind-shattering (really depends on the game) as possible. Discuss with your group about what kind of game you are going to have. There are powergamers in RPG, but they are not half as annoying as players who plays against the DM (trying to show him he's wrong) or against the group (how to handle treachery and PvP should be heavily discussed before the beginning of a game, and should never be done just for your own fun).

Finally, keep in mind you are not playing a small scale skirmishing wargame (even if you use figurines), it's a very different kind of game, with very different goals. Be as open minded as you were when you started wargames, learn the new codes and enjoy yourself.