How can I play monsters and NPCs up to their potential?

I personally consider a monsters mental stats, particularly Int and Wis) when deciding how a encounter will play out. A monster with low Int and Will (i.e. vermin) will typically attack the closest threat possible. I use Wisdom to gauge how effectively monsters will use tactics such as flanking and surprise. When I am behind the DM's screen, Intelligence determines how much pre-planning a monster can use before the encounter begins. One of the most difficult things to meta-game is a creature that has an arbitrarily higher Intelligence than you do. My response to this, is to give monsters with extremely high Int scores the benefit of the doubt and have their buffable spells and spell-like abilities pre-cast before the encounter begins. Higher Intelligence warrants more preparation. A monster with high charisma should be able to detect when things are going, or about to become, badly for it. It may use diplomacy or bluff to talk its way out of becoming the party's next XP snack.

@Lord_Gareth's answer is fantastic - this answer merely extends it

Know Your Players

Be careful not to give your villain knowledge they do not have, however. Your knowledge of the PCs is second only to the players - what is it reasonable for a villain of a given alignment, class, race, intelligence, wisdom and charisma to know?

However, a powerful monster is aware that powerful characters exist in the world (even if they are not aware of the PCs per se) and will take reasonable precautions. I remember a particularly twisted DM (with a mind not unlike my own) who designed the evil wizard’s lair as if it was their own – what pattern would YOU use to trap the tiled floor? How about a completely random one and just remember where to walk.

Which leads directly to ...

Know Your Monsters

What are the monsters' motivations? Why are they doing something so mind-bogglingly dangerous as tangling with the PCs? What’s their payoff?

Keeping with Erinyes:

• why are they here? do they have a specific mission that their masters in hell will be disappointed if they do not complete? Their fear of their masters will be infinitely greater than that of the PCs and their commitment to their mission will be paramount. This may induce fanatical resistance or early retreat and regroup depending on how they best see this playing out.

• how did they get to this plane? Some methods mean that they can’t actually die while others mean they can – this will greatly influence their tactics.

• how long have they been in the area?

• what allies have they recruited/enslaved?

• how will they use these to gain knowledge and/or impede the PCs? Evil monsters will take joy is compelling charmed/dominated villagers into attacking the PCs or being human shields.

• how have they come to the PCs notice? and vice-versa?

• how notable are the PCs?

• are they well known and have therefore come to the Erinyes' attention as a specific threat?

• does that intelligence extend to the PCs capabilities/tactics?

• is this a chance encounter? If so neither side may have made specific preparations.

• are the PCs hunting the Erinyes or vice-versa? If so then one side is probably buffed before contact is made - this by itself will make the encounter much easier or harder respectively.

• are they fighting on the Erinyes' home turf and what general and specific precautions have they taken to defend their base? General preparations are fair enough, after all, everywhere is hostile territory to an Erinye. Specific preparations are only fair if they know the capabilities of the PCs and the time of their assault – clever monsters will have cannon fodder guards and traps; not so much to damage the PCs but to serve as an early warning system.

Stupid Monsters are not Suicidal

First, non-intelligent monsters aren’t stupid – they are mindless and they will behave mindlessly. A construct will keep doing what it was told to do.

Creatures with animal level intelligence will behave like animals – they will seek to avoid or neutralise threats, find food and protect themselves and their young. If the fight is not worth the candle they will run away.

Communities of creatures with low intelligence (3-7 say) are as a whole dumber than a human community (Int 8-12). Remember that in both there will be exceptional individuals and these people will have a disproportional influence – becoming leaders and motivators. 10 stupid kobolds are one thing, 9 stupid kobolds under the direction of one with human level intelligence are something else. Up at the skinny end of the bell curve, in a large enough community, there will be kobolds as smart as people even if they are not as smart as smart people.

Communities are built to suit their inhabitants. Sticking with Kobolds, clearing out a nest of these sounds like a good challenge for 1st-2nd level PCs; until you remember that the tunnels are built for creatures 2-3’ high. Halflings and gnomes will find them cramped, humans and elves will find them impassable. A huge chuck of the wizard/sourcerer’s daily resources are going to be spent on reduce spells.

Even Stupid Monsters can get Smart

If the PCs are clearing out a Kobold nest, even these relatively stupid monsters will learn a thing or two if the PCs keep coming through the same entrance – maybe they will reinforce it, maybe they will booby trap it, maybe they will fill it in and dig another one (they do tunnel after all), maybe they will simply bugger off, maybe they will hire some orc mercenaries?

Smart Monsters are Smarter than You Are

When you are dealing with monsters with superhuman (19+) Int, Wis, Cha, these guys are smarter than you are. If the PCs come up with an idea you haven’t thought of this doesn't mean that the monster hasn't and hasn't planned accordingly.

This means that you are free to improvise pre-planned responses that you haven’t actually pre-planned!

Lawful vs Chaotic is more important than Good vs Evil

Good vs Evil is generally why the monsters and PCs are fighting, Law vs Chaos is more about how (unless you are dealing with Modrons vs Slaads).

Sufficiently smart monsters will make plans. Clausewitz said:

No plan survives contact with the enemy.

In response to this, lawful monsters will tend to cover all the contingencies while chaotic ones will have looser guidelines. Lawful creatures will tend to stick to their plans even when they aren't working, chaotic beings may abandon them at the first setback.

Study, Study, Study

Information is really the only way to overcome this problem, and it comes in a few varieties:

Know Your Players - This one is the most important. You should familiarize yourself with the capabilities of the PCs and with the tactics they tend to favor. Do they search a room when they walk into it before they do anything else? Does the party wizard like to summon? What is a Factotum anyway? This is the single-most important consideration because it helps you to determine what kinds of monsters they might enjoy fighting, where the strengths and weaknesses of the PCs are, and most importantly how you can challenge them without automatically throwing their characters into the killing pit. An important distinction to make is that a player's character might be capable of doing something that they have not, to date, done. Plan for their current tactics, but keeping the capability in mind for later will help you adapt, and can also help you if you want to teach your player how to use it.

Know Your Monsters - This is a lot like knowing your players, except for monsters and NPCs. However, you have additional considerations - how intelligent is the monster? How wise? Can they cast spells, and how often? Does their entry list any preferred tactics, and are those tactics intelligent? If the monster is mindless, can it be directed by another monster that can give it orders? Pre-built monsters are rarely designed in an optimal fashion, but you can increase or decrease their power level by changing around even such small things as the items, feats, or skills the monster is trained in.

A very important consideration is to look at how intelligent a monster is. Let's take the Erinyes as a specific example of doing the above. The Erinyes is a CR 8 devil (an embodiment of Lawful Evil), which means that in theory four of these lovely ladies is an appropriate boss fight for a level 8 party. The first place I look when deciding how to characterize this monster is her alignment (Lawful Evil) and then her ability scores, which are listed as "Str 21, Dex 21, Con 21, Int 14, Wis 18, Cha 20". What this tells me is that, compared to humans, the Erinyes is unnaturally strong, quick, and tough. That 14 intelligence indicates that they're smart and cunning but not genius-level intellects like a wizard, but her exceptional Wisdom means that she's perceptive, quick on the uptake, and aware of her environment, while her 20 charisma speaks of unbreakable self-confidence, self-assurance, and a strong self-identity (which, since she's Lawful, might indicate discipline, restraint, and a sense of patriotic pride in Hell). So right there we can reasonably say that this monster is going to try and use effective tactics, take full advantage of her natural and supernatural abilities, and come at the party from odd angles.

Little Details - Relating to the above, you want to familiarize yourself with some of 3.5's terminology so you can know it at a glance. Keeping with Erinyes, above, we can see that she's an "Outsider [Extraplanar, Evil, Lawful]", which while it doesn't have further explanation in the entry causes her to be strong - or weak! - against a variety of spells and abilities all by itself. If it helps, print out pages or cards explaining what the abilities of a monster do before you run an encounter with it. Look at how a monster might use their abilities in a synergistic fashion and spend their entire turn doing stuff instead of just part of their turn. A great example for this Erinyes is combining her power to fly and her longbow, or her at-will Unholy Blight ability, which puts her out of the range of non-ranged attackers and lets her rain down death with impunity. She's immune to fire, so perhaps she might use her flaming arrows to burn the building the PCs are fighting in, or confront them in the midst of a blazing wildfire, trusting in her fiendish nature to protect her while they're forced to spend resources avoiding the flames.

Of help in these considerations is the System Reference Document, and specifically the Types and Subtypes, Special Abilities Descriptions and Spells section.

The PCs Don't Have to Kill It - A lot of monsters and spellcasters have unparalleled ability to run away. Whether it's an Erinyes teleporting to safety, a worm-like creature burrowing away or a sorcerer invoking a flight spell to escape combat, most villains and monsters don't want to stick around to be killed by adventurers. This doesn't have to be a bad thing - not only does it potentially give you a recurring villain, but the XP system in 3.5 rewards players for overcoming challenges, not for killing monsters. If they make the fiend flee, then they're overcome the challenge. Additionally, the ability to teleport to regroup can make for a more extended encounter as the PCs deal with recurring harassment by a determined enemy until or unless they can pin it down and force the end of the battle.

Another interesting option for this is the monster or NPC surrendering. Aside from letting players with certain concepts (like a bounty hunter, or a merciful paladin) indulge in the ability to turn an evildoer over to justice or help them on the path to redemption, this can be a way to further the story. A great example is in the module Expedition to Undermountain, where the players encounter a Mimic. If they defeat it, it surrenders and begs for its life, offering to report on the activities of people that pass through its hallway in exchange for food. In this fashion a monster or NPC still gives XP to the player characters, but also can become a long-term ally (even if it's a difficult ally) or a beloved companion.

3.5 Has No Aggro Mechanic - This one is the biggest source of people wondering why a supposedly strong monster is weak. A common misconception people have about 3.5 is that, like World of Warcraft or Everquest, the guy wearing armor is automatically a more appealing target. Nothing could possibly be less true, especially for an intelligent monster. Why should a flying monster stay toe to toe with the fighter (and get cut to ribbons for the trouble) when she could fly, ignore him, and take out the dangerous sorcerer? There's no mechanic that lets fighters or other melee types "hold aggro", which means that an intelligent monster will assess threats based on what's happening, their own experiences, and their Knowledge ranks. Demons and devils, for example, will go after clerics and wizards because they know those dangerous beings can banish them back to the Lower Planes, while the sapient-but-sorta-stupid orc might prioritize a fighter or cleric first out of a desire to fight a great warrior. In most cases, however, a monster or NPC will target whoever is making themselves the biggest problem, so if a melee concept wants to hold down aggro they need a reason to make monsters care about them or a way to control the battlefield enough that the monster must needs go through them to get anywhere.

And, finally, Even Smart Monsters Can Be Stupid - Yes, part of the point of this guide is to encourage DMs to play intelligent monsters up to their potential, which can lead to more tactically interesting games. However, D&D is also a roleplaying experience, which means that personality is part of it. Yes, the Erinyes is incredibly clever - but she's also incredibly Lawful and incredibly Evil, which means that she might be operating under oaths of service that restrict her, or that she might get sloppy in an undercover operation because she took the chance to torture a homeless man to death instead of covering her tracks. Unless your game is high-optimization (in which case, by all means, pull out every stop you have) these sorts of little mistakes both help slate the odds towards the PCs and help flesh out the personality of the monster so that they're not just a generic being.