Why does the Donnager use railguns at close quarters and torpedoes at long range?

Railguns are also more used as a stealth weapon, there were scenes where a railgun was used to take out a ships drive in season 4 that was moving in a straight line that, a railgun can easily hit something like that. In a battle though, Missiles are the primary weapons but medium range, railguns are used as they can track far more easily. At point blank range i.e. a few kilometers ships just use their CIWS/PCDs.

It's quite realistic really, a modern destroyer would work in the same way, it's guided missiles are it's main long range armaments', it's 5 inch gun is its medium rage anti ship weaponry and CIWS and the bushmasters are for close range combat.

At distances of many hundreds of miles a missile is a more ideal weapon in an engagement. Namely because said weapon can maneuver and redirect itself in flight. A rail gun can only fire in a straight line. If the target is moving that requires tracking, Missiles are fire and forget weapons largely. Railguns require aiming precisely until a firing solution is computed or decided. Then again having an entire battery of guns firing simultaneously takes the guess work out by reducing likelihood of missing per salvo. After upon firing, the round now in a straight trajectory is now destined to continue traveling until it either hits or misses. They make excellent short range weapons, volume of fire is high if you have more than one gun, ammunition is cheap assuming it's not made of exotic material. Nevertheless; as a long range weapon the railgun is detrimental for several reasons.

1. Miss and hit: In a three dimensional battlefield of space, a railgun round, should it miss the target and continue traveling, may run the risk of hitting a possible friendly. A battle involving firing dozens of rounds per minute with a fleet of ships tangled in an engagement; risks filling the battle field with stray projectiles that could potentially damage your own fleet, or if continued travel the planet or celestial object that may be inhabited.

2. Targeting: The vast distances involved between ship to ship naval fire exchange makes tracking a target a challenge if there's not sufficient lead. Real life CIWS systems designed to hit moving targets use computers to fire in advance of the targets actual movement Like Duck hunting, you don't fire at the target you fire at the targets future direction and they fly into it.

3. Unavoidable Error: Angular distance between two point objects, as viewed from a location different from either of these objects. If firing in a perfectly straight line at a target, the round will hit. However angular errors creep the further you go out. As pointed out in an answer to a related question on Movies Stack Exchange:

At a distance of 60 feet, a 1° error in aim causes a 1 foot error in impact point, a large target say 100 meters long, that error is acceptable, the round will hit the target, albeit not exact dead center... At 60 miles, 1° error causes missing by a mile.

A target 100 meters long means you're off by nearly 5000 feet. So the gun must be adjusted to a fraction of a degree. In a moving situation where warship and target are moving simultaneously this is unacceptable. On the other hand if the goal is to fill the sky/Space with fire, then a mass salvo is desirable.

4. Recoil: Newton's laws are undeniable in space. Even real world attempts to build railguns; engineers understand even a electromagnetic weapon cannot avoid the massive recoil forces of pushing a projectile at hypersonic velocity. The US Army spent a decade developing an anti-tank rail gun, but ultimately found it too expensive, complex, and offered less capability than current tank guns. The Army determined railguns are less accurate because of the “aerodynamic jump” from the armature or in the gun industry as "Muzzle Climb", which is visible in slow-motion videos of rail guns being tested and the recoil worsens the accuracy. That's why prototype guns seen on the internet are fastened down using a bracing unit. Look at all the infrastructure needed to keep such a small gun stable.

The second problem with recoil is it runs the risk of propelling the ship, if the gun is off center of the ships center of mass the rounds fired may cause the ship to roll off it's center of axis. If the goal is keeping the ship steady to find a firing solution, every firing may force the ship to produce opposing thrust. In a ocean going battleship the firing produces mild thrust but this is overcome by gravity and the weight of the vessel destabilizing it. In the vacuum of space with no gravity to hold down the vessel it'll be pushed in the opposite direction (Newton's third law).

In The Expanse, the efficient Epstein spaceship drives, combined with steroid cocktails (the "juice") that make humans withstand such maneuvers, are capable of doing high-g maneuvers to evade incoming fire and point defense.

Rail guns fire large slugs at high velocity, but they only shoot in a straight line, and are trivial to evade at long range. Therefore, guided missiles/torpedoes are used at long range instead, making it a fight between the onboard spaceship autopilot computers (or even human pilots) and the torpedoes' guiding computers.

At short range, however, rail guns are more effective as they're harder to evade, travel faster than torpedoes, and pack a larger punch.

The book that the first season of the TV show is based on, Leviathan Wakes, explains this in Chapter Eleven during the corresponding scene:

In the history of the Coalition, no capital ship had ever gotten into a close-quarters battle. But here they were, firing the ship's big cannons, which meant that the range was sufficiently short that a nonguided weapon was viable. Hundreds or even dozens of kilometers, not thousands. Somehow the Belt ships had survived Donnager's torpedo barrage.