Can I recruit random passers-by?

You can't invite them, but you can capture them. First, select one of your colonists, turn them into a militia member (R), and then right-click on the wanderer and select "arrest [name]".

You have to have a prison set up - an enclosed room with a bed marked for prisoner use is all that you need - and then wait for your colonist to bring them to that room. (Don't forget to take your colonist off of militia mode when the person is in the room!)

Once they're there, you can do a bunch of fun things like executing them or trying to recruit them. Make sure that you're actively recruiting them (click on them and select the prisoner tab) and their resistance will slowly lower to 0, after which every time your warden visits, there's a chance to recruit them.

Enjoy your increased population, but keep in mind that this will automatically lower your standing with the person's faction! Which means instead of visitors, you'll have a lot of angry townsfolk coming to attack you.