Gaming Mods Security Risks

I'm going to throw a piece of paranoia in here (simply because of stuff I have been doing recently on console hardware).

While direct executable content is definitely makes life very easy for an attacker, it's not the only vector. All it takes is one mis-checked buffer in a game's data file, and all of a sudden you have the potential to smash through the stack.

Although beyond the ability of your average Joe, an executable with disk and network capability could quite possibly load a "map" or "model" file and actually be reading / writing files on the disk, opening sockets, maybe writing to the Windows registry, installing a key logger, or who knows what.

It's only a small step from there to be executing your own code as opposed to mangled function calls in said game. And then you have complete user land ownage of a given system.

There are different kinds of mods.

• Mods which add or replace game content files like images, models or maps. These should usually be harmless, unless they exploit a bug in the game engine which handles these assets.
• Mods which add game logic in form of scripts. Many games have a scripting engine which allows mods to perform limited programmed logic. Sometimes these scripting engines are very powerful and not properly sandboxed, so they allow mods to do things a mod isn't supposed to do, like for example accessing files which are not part of the game and opening network connections.
• Mods which patch or replace the game executable. These are always dangerous, because they can turn the game executable into malware when they want to. Also be careful with any mod which comes with a .DLL, because these also contain binary code which is executed in the context of the game executable.
• Mods which have an executable component themselves. This can be an installer which runs once or a companion program which runs while the game is running. Needless to say that these are also privileged to do anything they want.

Mods certainly can be used as infection vectors. A lot of it comes down to a question of trust. A mod with tens of thousands of downloads and nobody suggesting they've had any problems is likely to be OK (though still no guarantee!).

In an ideal world:

• Wherever possible, avoid mods that have an installer. There are some situations that this can't be done, but mods with installers are probably the biggest threat. A lot of users will happily click through a request for administrative privileges from an installer.
• Download mods from reputable sources. Ensure you're always downloading the mod as the creator released it, and not repacked by anyone else!
• Check comments sections for the mod, and do a quick google search with keywords like "virus".
• As per the first section of this answer, favor popular mods over ones with barely any downloads.

Sections of your question are not possible to answer with any real useful information, as there are so many different types of mod available with great differences between them. However, generalizing:

How dangerous are gaming mods, does language and file-type matter, and how can I decide these for each individual game?

How long is a piece of string? Mods can be very dangerous or not dangerous at all. File type certainly matters. Mods that just replace textures, or add new models etc are pretty unlikely to cause you any problems. Mods that use .exe installers are something to be wary of.

Does the maliciousness/exploitability depend on the mod file I am downloading, or the type of files the mod-file is modifying?

Both. If the mod you are downloading is a .exe, it's a bigger risk than if it's a .zip or .rar (generally). If the mod file just replaces some textures or config files, it's a much smaller risk than if it replaces the game executable...

Will a decent AV with signatures/heuristics pick up a malicious payload?

Hopefully, but don't rely on it.

Can mods be examined in a debugger (OllyDBG) or other tool (which type)?

Again, this would depend on what the mod is and what it does. You can certainly check .exe files in a debugger, but frankly I just wouldn't run them at all. Or run them sandboxed in a VM if you really must run them to check them in a debugger.