I was confused by this from the poorly written explanation in the free-to-download basic rules (which is clearly designed to encourage you to buy the full books).

However, the way I think of it is that Perception is to spot something like a clue, and Investigation is to work out what that clue means. As an example, say Sherlock and Watson enter a murder scene - a bedroom with a man's body on the floor, lying face down in front of a wardrobe. Sherlock turns the body over and sees a dart stuck in the front of the mans neck (obvious to spot, no Perception check needed). Watson then sees another dart on the floor behind the body next to the wardrobe (easy Perception check).

Sherlock realises that the dart must have hit the wardrobe and bounced off (easy Investigation check). Sherlock looks and finds mark on the wardrobe (Perception check). The mark indicates that the dart was fired from the right side of the wardrobe (Investigation check). Sherlock looks to the right and sees a small broken pane of glass in the small leaded window (Perception check). Sherlock tells Watson that their killer is a rather incompetent but acrobatic assassin who shot the victim from outside the window, but missed as the victim walked past the wardrobe. The victim then turned around on hearing the dart bounce off the wood and was then shot in the throat, before collapsing where he stood, probably from a poison on the dart (Investigation check)!

This is of course dice roll overkill, and should be cut down to just a couple of rolls to cover all outcomes, but explains the difference. The problem in distinguishing between the two may well come from reducing the number of rolls required. A generous DM that lets their players use their best abilities to solve problems will stress about this less than a rules-lawyer DM. The way that the rules are written, there is overlap between the two skills, but if you just play generously, then you shouldn't have a problem.

From the examples given in the rules, you could say that realising that a dark alley is a perfect ambush spot and the thugs are likely down there is an Investigation check, but actually spotting one hiding behind some barrels is a Perception check. Mind you, seeing some 'fresh' footprints in the mud in the alley could be a Perception or Investigation check... Just let the players have their best crack at it and don't be a stickler; let the wizard use Investigation and let the cleric use Perception.

Perception is used for senses

"You walk down the hallway and you see a row of tiny holes along the bottom of the right-hand-side wall, what do you do?"

Primary sensations you can see or hear or smell or touch or feel in the air or similar. PHB p 178.

Investigation is for knowledge & reason

"Yes, the lack of dust in a quarter circle in front of the wall does look like a secret door has been recently opened here, and then closed again."

Knowledge of architecture, design, common sense… and reason to draw conclusions from the clues you've used perception to gather. PHB p 178.

Active vs Passive

When you're just moving in the dungeon with your eyes, ears or nose open, sometimes you (as a character) perceive things even without you (as that character's player) asking to actually look. As an example, sometimes the character sees a tripwire even without the player going "I'm going to look to see if there's a tripwire" in every room.

Using a skill passively means walking around with a permanently rolled 10 (or 15 if you have advantage), modified by your bonus, and then compared to the various DCs. For example, a particular tripwire might have a passive perception DC of 15 and then anyone with passive perception 15 or higher will see it without actively looking. A set of footprints might look weirdly overrepresented by steps leaving a small, closed room, and anyone with a passive investigation of 15 or higher will realize that without actively pondering it.

Suggestion

Passive perception is OK. It allows you to selectively describe the dungeon, normally to everyone and then to high-perception characters add things like "[character name], you see a tiny crack along the left-hand-side wall". Some modules have DCs already listed. When writing homebrew locations, or prepping modules that don't have DCs listed, write the DC down beforehand and stick to it. One idea is to roll 2d6+8 (or 4d6+1, or 4d3+7) to get a DC that's centered around 15 [the standard, "medium" difficulty] but doesn't make any single perception score actively best (i.e., no "the optimal passive perception to shoot for is 15, it's worthless to go lower and a waste to go higher"). Again, do that roll beforehand so you can write it down & stick to it.

Active perception, and both passive and active investigation, interacts weirdly with player skill. Therefore, don't use them. (It's true to some extent to passive perception as well.)

"I open the drawer and lift up the clothes, do I see the key there?"; this is Perception. PHB p 178 has a similar example.
But if the prep says that the figurine is actually in there, should you then really have to roll to see if you can see it? At some point it becomes ridiculous. "I grab my sword!" "OK, roll to see if you can feel your own sword."

The PHB's example points out that if the key is in the drawer under the clothes, looking at the walls of the room would give you no chance of success. So obviously player skill can enter into the equation, and you have to decide how player skill and character skill should interact.

So you have four options:

• The character skill becomes a lifeline for when player skill is insufficient. You let them find the key without them looking under those clothes, and if they don't look in the right place you let a high perception roll give them another chance to find it. (This seems to be the implied option from the PHB text; their skill leads them to the drawer, the roll leads them under the clothes.)
• The character skill becomes an additional hurdle to jump even when player skill is high. You don't let them find the key even if they look in the exact right place until they've also made a perception check.
• The player skill is ignored. Don't dwell on precise description until you've seen the outcome of the die rolls. It becomes "Roll to solve puzzle."
• The character skill is ignored. Don't roll these skills and instead rely solely on player skill.

Decide (together with the group) which of these four interactions you prefer and apply it consistently.

As a side suggestion, if you do go without using a lot of skill rolls—for example, if you are also not using Persuasion, Deception, Intimidation and Insight, for the same reason—is the Ability Check Proficiency variant in the DMG on p 263.

As an example, our group uses the fourth option and that ability check variant. Last week, we played two four-hour sessions and in those eight hours of gameplay there was a total of four ability checks. (One strength check to grapple, two strength tests to budge open doors, and one int check to recognize that a book about golems was specifically only about clay golems.) It's perfectly fine to play D&D without every action or statement being accompanied by an ability check.

Pouring out flour to see trip wires, pouring water to see trap doors.
Just describe the room and ask "what do you do?"

What's fun to roleplay out vs what's fun to roll is different between groups. You might want to roll for trap finding if you hate pixel hunting but then play out social interactions, or you might want to play out trap searching if you love exploration but then just roll to quickly resolve social interactions if that's what you want to do.

Exceptions

When is that fourth option no good?

• When you are winging it and you don't know if the key is in the drawer or not because you don't have that in the prep.
• When the group has no desire for player skill but instead are highly invested in finding out "what would the character have done in the situation?"
• When there's a large time skip: "OK, we search through every room on the top first two floors. Three days later, have we found anything?"
• When you desire a high differentiation between characters; a low-int character being played ignorantly and a low-wis character being played obliviously.
• When you desire a hilarious amount of slapstick and mistakes. "Oh, wow, I can't believe my character didn't see that tripwire even after I looked so carefully, hahaha, well, time to roll up a druid, I guess."

If these exceptions apply to you, so that you still want to use these skills, hopefully the differentiation at the top of the answer can be useful.

A framework for choosing

Is the player trying to understand a problem, notice if there is a problem, or trying to solve a problem?

In the first two cases, you will usually use Perception; in the last case, use Investigation. As a practical matter, the mental skills that are represented by Wisdom and Intelligence overlap, which leads me to ...

The DM makes rulings in D&D 5e, so you are free to use either

You made this mention in comments:

Maybe I'm just confused coming from a 3.x standpoint.

Perhaps that is a root of your confusion. An explicit design aim in D&D 5e is to encourage DM's to make rulings, rather than to be bound by the rules (or have to look things up on yet another table). It's liberating. The rules serve the DM, the DM doesn't serve the rules¹ (DMG p. 5, "Master of Rules"). Embrace the D&D 5e philosophy.

Giuseppe's bolded point in his answer is how each DM I have played in 5e with does it: if you aren't sure, give the player the option to use either Perception or Investigation. DM stress relieved, confusion over, and play continues.

It is not critical to be exactly right in which one you choose in a case where your judgment tells you that it could be either. What is important is that you describe a situation, the player describes what they do, and you narrate the results (using dice when necessary).

I notice from your profile that you have been a database developer, and an IT specialist. Use your in life experience to help you frame the in-game situation: are the characters in a problem definition situation, or a problem solving situation?

Handy tip: if you use a DM screen, have a card or a field on that screen with the bullet points from the Chapter 7, and the best points from the answers to this question, to give you an aid in making a decision if you would like easy reference.

I'll offer one last thing to chew on as well: "how do I know which one to use?" isn't necessarily the right way to approach this question in D&D 5e. "What do I think is the one to use?" is how to approach it from a D&D 5e point of view. And lastly, "Does it really matter which one I have the player use when it could be either?" is in general answered "no." The play's the thing.

¹ A nice short essay on D&D 5e

Perception is for noticing something, whereas Investigation is for understanding it, but in reality it's impossible to do one without the other.

There are 4 scenarios:

• Things that are trivial to both notice and understand require no check.
• Things that are more difficult to notice than to understand require a Perception check.
• Things that are more difficult to understand than to notice require an Investigation check.
• Things that are roughly as difficult to notice as they are to understand require either check, so give the player the option to use either Perception or Investigation.

Precedents:
Tasks that can be solved with more than one ability check already exist, for example players choose to use either Acrobatics or Athletics to avoid grapples and shoves.

Rationale:
People lean on what they do best, so if both Investigation and Perception are applicable, it makes sense to let the players the choose which skill they want to use.
Requiring players to roll both Perception and Investigation would be like giving them advantage or disadvantage, thus altering the challenge of the task, so it is better to require just one roll.

TL;DR: When you can't decide easily, let the players choose.

Given that Investigation is an Intelligence based skill and Perception is a Wisdom based skill, the easiest way might be to ask whether a member of a group with high Wisdom and low Intelligence could do it.

Fortunately, we've got a really big group of those, namely animals, and we have a lot of real life experience teaching them to do things, namely trained dogs.

So a simple question to tell the two skills apart can be "Could you train a dog to do it?".

Some examples of things dogs can do:

• look for traps (there are bomb detection dogs; closest thing in real life to traps)
• look for drugs or money
• track other creatures
• recognize a person they know from a picture
• notice suspicious behavior in other people (although this could also be Insight)

Some examples of things dogs can't be trained to do:

• figure out which of the three tracks to follow
• look for non-obvious valuables (you could teach a dog to find gold; you can't teach a dog to find the really expensive painting)
• figure out which picture belongs to a person they don´t know
• find out where to find someone without a trail to follow

Generally speaking, anything a dog could not be trained to do is almost certainly an Investigation skill, since dogs lack the intelligence to do deductive reasoning, whereas anything dogs can do as well or better than humans is likely Perception, since they have sharper senses than we do.

I've found only one notable exception, which is that you can't teach dogs to read. So if you have 5 paintings, a dog wouldn't be able to deduce which is of the villain, but the players will if they know the name and it's written on the frame.

Normally that'd be Investigation, but in this case it would either be Perception (if the text is scrawled in tiny print somewhere on the portrait or something, like maybe on a document held by the character) or just no roll at all (if the painting has a nameplate on the frame).