Why does terraria get stuck at "connecting to...."?

You need to post more details in order for us to help you. (I cannot comment with my current reputation; I understand this answer sounds like a comment, but the information below should help the OP.)

If you are connecting from your own computer, you either need to connect directly via your own IP, or using localhost. When playing multiplayer, and hosting on the computer I am also playing on, using the official dedicated server .exe, I have had the most success joining using the IP of localhost. (It might seem weird to type text into a field asking for an IP, but Terraria can join servers based on domain names - which can be handy for servers using dynamic IPs.)

I have run into this issue a number of times. When you first start up the server, it should ask you how many player spots to reserve, from 1-8. I'm assuming you selected 2 - one for yourself, and one for your brother. Try increasing this to 3 or higher (I set mine to 6). This seems to get rid of this problem, or at least reduce the occurrences by a significant amount.

The following is speculation, since I don't have the client/server source code in front of me, but I believe this error happens because whenever you disconnect, the server keeps temporary session in memory in case you reconnect. This "ghost session" uses up one of those reserve slots, and sometimes the server bugs out and doesn't end those sessions for a long time, which prevents you from logging back in. By increasing the number of available slots, you bypass this problem... at least for a while, until hopefully the server un-bugs itself. If there are enough disconnections to take up all the slots before the server fixes itself, you may still need to restart the server.

I believe one of the latest server updates was supposed to fix this error, but I have occasionally run into it again anyways. The above workaround still works for me just fine.