Difference between "come by" and "come to my office"

What is the difference between come by and come to with a destination?

1. Come by my office
2. Come to my office

Such a phrase might be used by a lecturer:

If you want my help, please come by (or to) my office this evening.

Collins doesn’t really help with come by (it says "to obtain or find", which makes no sense here).