However your table wants

Is it more usual to ask the other players 'out of character', or is it more common to roll some kind of in-game check to identify the creature?

Yes, it's usual and yes, it's common. And unless you poll every group that's ever played (or at least a statistically significant sample - PhD anyone?) you won't know which is more frequent. Look, some people like to create deep and complex worlds where each session is a new revelation and some people just like to chop things into mincemeat - you do you, ok.

Metagaming

I hate that word.

For a start it has two meanings:

1. The game about the game. Looking at the character classes and choosing to play a ranger rather than a wizard is a metagame decision; just like a coach watching video of their opponent in this week's football match and consequently adjusting strategy is metagaimg. This usage is not problematic and is usually not what's meant when a role-playing gamer uses the term.

2. Particularly in a role-playing game; it's applying knowledge that the player has that the character ostensibly doesn't have. It's this usage that is problematic and I will now indulge in a short rant because the concept sets my teeth on edge.

Its a game of make-believe elves in a made-up world using rules that cannot be anything but the most abstract model of non-reality!

I cannot for the life of me see that rolling a d20 to see if you "hit" is not metagaming¹ but claiming that your character, who supposedly grew up in a world where vampires are real, knows that sunlight hurts them isn't. Hell, vampires aren't real in this world but everyone knows your best defense is a garlic-wrapped wooden stake!

OK. Rant over.

Basically, it's offensive metagaming if it breaks your table's verisimilitude and it's ok metagaming if it doesn't. So, you and the people you play with need to set your own boundaries on this, just like you need to decide how naked you can be while you play.

Options

1. It can be fun to find out by trial and error. For the DM, yes, this is a laugh riot. For everyone else, it can be an exercise in frustration. I've sat on both sides - don't do this.
2. Finding out by being clever. This is good for the DM and the players. The DM places clues. The players find the clues. The players put the clues together. The DM and the players feel clever. Feeling clever is fun. For example, the Spear of Osiris in The Mummy Returns - while this was incredibly clunky for a movie it is 100% perfect for a role-playing game because players don't have scriptwriters to tell them the answer. See the three clue rule.
3. If the player knows the character knows. This rewards experience and, in most games, people with more experience are better at the game than those with less. Why not in role-playing games? How does the character know? Who cares. Or, see below ...
4. They just know. I know the difference between a lion and an antelope. This is amazing because I don't live in Africa, have never been to Africa and have never hunted/been hunted by either. How can I possibly know something about the creatures that I have never encountered? Books, stories, myths, legends are all things that exist on Earth - why aren't they in your fantasy world?

Of course, you can and should mix and match - trolls are common so everyone knows about fire, but kdsja² are rare, so only the most arcane tomes and learned sages know about their vulnerability to tulip bulbs.

¹In the second sense - it clearly isn't in the first sense because that is the game.

²Don't bother looking for kdsja, I just made them up.

It is not usual to ask out of game

Asking out of game is known as 'meta-gaming', and this is not the usual way to do things. @pwi posted a link in a comment, which is highly relevant here.

Of course, every game is different and if you're playing D&D as a mere strategy game like chess or as a "battle game", where everyone clearly knows the capability of every game piece, then that is entirely possible. I don't believe this is generally how role-playing games go; players are expected to take the role of their characters to some extent, like actors in a play. The extent to which this is done varies by group (my group is more 'roll-play' than 'role-play'!), but keeping player knowledge separate from character knowledge is still expected to maintain the RPG experience.

@pwi's answer mentioned many other ways your group can do this, but I just wanted to add some of my own DM/player experience.

Describing Resistances/Immunity

As a DM, if a character hits a creature with a damage type it is resistant or immune to I will often say "your fireball singes it, but it the burns don't look as bad as you expected", or "your fireball explodes around it, but the creature - unharmed - merely grins and turns towards you...".

Character Knowledge

On encountering a new creature, players will often ask if their characters know anything about it. This is where I may call for an ability check of some kind, generally Wisdom(Nature) or Intelligence(Arcana) for more obscure magical creatures.

Success will reveal 'common known' information, but that should still be worded in an 'in-game' way. Don't just say "yes, these creatures have around 66 hit points and resistant to non-magical weapons". Better to say "you've heard these creatures are pretty tough and mundane weapons barely scratch them".

The DC for this check is a bit of a judgement call, based around how common the creature is. The rarer the creature, the higher the DC. Then again, certain legendary creatures may be incredibly rare but knowledge of them is known through tales and stories. Many people will have heard of beholders - but not all the stories about them may be true.

If your players made backstories for their characters, it is also possible that a creature may be part of their backstory or the character came from an area where such creatures are known, so that is also a way their characters could know something about them (including their culture, society etc).

No metagame at my table

Each DM can handle the situation however they want, but in 5e metagame info generally isn't given to players, they are expected to interact with the game world. At my table I never give players metagame information. If they want to figure out the weaknesses and resistances of a monster, they need to think about how they would go about that.

The same process and logic applies to other situations, but let's discuss the monster as an example:

Observation

The simplest way is to observe the creature and its surroundings. What kind of environment is it living in? What does it appear to eat? Does it have any telling wounds? Can we stake it out and try to figure out its habits?

Research

If the monster is a significant threat, I expect players to do their research and observation. Go read books, go talk to old guys in the bar, watch the monster from afar and determine its habits.

Learn by doing

A dangerous but straight forward method is to attack the creature, and if an attack is ineffective switch to something else. If you hit with a bludgeoning attack and make good rolls but the monster doesn't seem to care, that's a good hint that bludgeoning just won't cut it.

Learning by experience

I like to theme arcs and have players learn during the arc. Fighting a goblin one off doesn't give much room for learning and research. However fighting against goblins for days or weeks gives lots of room to learn from experience. Over time their conversations with NPCs, observations between battles, and experience in combat all add up. The players go through the whole process of being surprised by the goblin's capabilities, experimenting with tactics, and finally becoming effective goblin hunters.

When they fight the final big bad goblin boss and he pulls out every trick in the book, the players are able to go toe-to-toe and feel the how powerful their mastery has made them.