How can I specify the server port in the client?

This is assuming that you are attempting to access the server from outside your LAN (e.g. the IP isn't 192.168.x.x).

1) Port Forwarding - If you haven't port forwarded, do so. Without port forwarding, you'll only be able to access it from within your LAN. To do so, go into your router's admin interface and forward the port that you're going to use to the same port - and specify it to be forwarded to the server's internal IP, assuming it's on the same network. If this doesn't work, but you can ping the machine from an external network (e.g. ping xxx.xxx.xxx.xxx from the cmd on a windows machine), there may be a problem with the server's firewall.

2) Using ports that are reserved by other services - Usually on *NIX systems (and many more!) if you try and run the server under a certain port, it may not let you unless you have the relevant access rights. The first 1024 ports, in particular, are reserved for system stuff. If you're using any port under that, there's your problem. Change it, and it may work.

3) Firewall Problems - Make sure you've port forwarded first. Then, depending on what OS the server is running on, you'll need to allow the communication from someone not on your network for the specific port that you're going to be using for the server. By default, not all ports are open on certain machines and are blocked by a firewall for security reasons. There are plenty of guides for whatever OS you're using, just find one of those. If you're running in some Linux environment, it's probably "iptables" that's blocking those ports.