How to stop a behaviour that certainly is negative but technically does not break the rules?

The behaviour you're describing sounds like the user is most likely a troll without a dedicated agenda. He knows where the current boundaries are and how you manage them so he always tries to hit this sweet spot where he is sure to gain some attention while simultaneously avoiding breaking rules.

If this user really is a troll, you should just ignore him. It's likely that his behaviour isn't only apparent to moderators but to the community as well. So long as he doesn't break any rules, you just let him do his thing. Either he fast becomes bored and stops on his own or his doings slowly transition to being a general annoyance for your community. If this happens, your community will sooner or later call him out for not being a nice person and disrupting the community. Often, this is enough to justify temporary actions like suspending his account for a limited time or putting all his comments up for inspection before being allowed to post. All these obstacles will pretty much do their thing and disturb him in his doing, possibly annoying him beyond a certain point where he leaves or learns to participate productively and peacefully.

If he stays on track and keeps trolling, your course of action is: ignore, wait for reasons to sanction and repeat. Don't let it become an emotional thing for you. Be professional about it and be done with it.

If this is a rare occasion where he is actually disgruntled with inner workings of the community, you shouldn't miss out on asking him what's amiss. However, you shouldn't argue with him. Instead of answering moderators need or don't need to be experts on certain topics, you ask him why he thinks so. Maybe he isn't content how complex questions are answered or he misses citations.

The same goes for a community which, in his point of view, is nicer than the one where he spends his time writing stuff like that. (Pretty silly, isn't? I wouldn't waste my time on a community that is not as nice as I wish it to be. So just another sign that he is trolling.) You ask him why he thinks so and emphasize that you like to take every opportunity to help make the community a better place.

Whatever he answers, you just give it to the community and ask for their assessment. Don't comment on it, don't rate it, don't do anything with it. Asking the community how it thinks about it allows you to gauge how relevant his impressions are.

• If they aren't, well, problem solved. You tell him the community largely disagrees with him, nevertheless he's welcomed to further participate to his heart's contents.
• If they are, well, problem solved as well. You were successfully able to find something that's amiss, involved the community and (depending on how your users discussed) already have solutions at hand.

The approach involving asking him what's amiss can also be used if he's a troll. You essentially trash his intentions by turning his emotionally driven comments into a factual and objective discussion on how to improve the community. Most trolls are very conscious about how they are perceived as they want to keep getting attention. By not responding appropriately to your questions, he might just throw all his "build up work" away exposing himself as a brick. (At this point, you can just return to ignore, wait for reasons to sanction and repeat.) Or he responds appropriately and you just gained the upper hand by turning his unproductive behaviour into something productive.

These approaches are written with the community in mind. You don't want to be seen as unjust or unfair - the approaches explored keep this in mind by keeping the community in the loop and extinguishing fire places before they can lit up. Don't get invested and you're good to go, helping foster your community.

The communities that I see deal with the issue of "bad behavior but not breaking the rules" use rules like these: "Don't be a d###"

"These rules are not indicative of a full list, and moderators will act at their own discretion."

"Admins can do whatever they want"

Or they forgo having rules, and have a code of conduct instead, and set behavioral expectations, and when conflicts come up, they try to settle them. When obvious bad actors appear, they ban them. Simple, but it has produced a tight and caring community from what I've seen.

There can't be a one size fits all answer. It depends much on the kind of site, it's goals, and even more so on exact circumstances of each comment/action.

comments like "It would be so great if moderators were experts in the topic of the site!"

Looks like a good example where context is everything as it depends quite on what happened before. If there was a comment by a moderator regarding any topical item, or more important any moderation decision based on topical content, then this may be a valid critique. Being a moderator automatically comes with the bonus assumption of being an expert. Showing otherwise will draw more blunt response than any other commenter would get. This may sound unfair, but it comes natural.

Fighting such comments may easy create a downward spiral for the site or at least create an unease feeling among all participants.

Now, if it was clearly unprovoked in the actual context (needs to be checked quite carefully), then a clear warning about staying on topic needs to be given.

or "That other community is so much nicer than this one!"

Again, this needs a check if there is a previous context he may refer to, supporting his point. If not (check carefully), it's again a case to give a clear message.

When moderators contact the user, the user replies with gibes or sarcastically.

That's an irky area. Private communication should usually not be of any consideration for public action. It will of course help to build your opinion about the offending user. He should as well (using the same medium) given a clear, non offensive, but firm reply, pointing out all short comings in simple language.

But unless he issues explicit threads, all public action should be only based on public available knowledge. Acting in public is as much about the impression other members will get as it is about resolving the case.

How should moderators deal with such a behavior effectively?

By carefully looking at each instance and react as close to the exact circumstances. Using global and or meta argumentation will always leave room for further unease. Maybe not so much with the offender, than with other users unaware of the circumstances decisions are based on.

Moderation privilege does not make one immune from overreacting or bad behaviour, so always check circumstances and act according when herding the flock.

I have been managing forums since I was 11 years old.

There are rules that have to be followed. If I were you I would sanction those kinds of behaviours instantly, for 3 reasons.

1. These behaviours are contagious.

If you allow one person to behave in this manner, in the future other users could behave in the same way. This would lead to a worse environment, difficult to manage and keep serious.

2. These behaviours show bad organization and bad situation management.

It shows that you are not capable of keeping your community in order. This could lead some users who are not used to this type of behaviours to leave the community, and some new users to abandon the community or at least to not employ energy and time in the project.

3. Mocking the Staff should be never ever allowed.

Because the staff is the authority of the community.