In which order are tax credits calculated if you earn both refundable and non-refundable?

Look at the sequence of items, and related instructions, on 1040. The 2020 version isn't out yet, and according to the (nonfinal) VITA/TCE training it apparently will have some items renumbered, but as of 2019:

• lines through 12b: compute income/AGI less adjustments and deductions, and resulting tax, including additions to tax for AMT and excess APTC within limitation

• lines 13a,b,14: subtract CTC, ODC, and other nonrefundable credits from schedule 3 (foreign tax, child&dependent care while working, education, saver's, residential energy, general business, prior AMT, and your writein adoption) from tax, but not below zero

• lines 15,16: add other (non-income) taxes from schedule 2 if applicable

• lines 17-19: total payments (e.g. withholding and passthrough, estimated, extension, excess SS/RR) plus refundable credits (EIC, ACTC, partial AOC, net PTC, HCTC, fuel) (note ACTC is an additional refundable credit for child/ren when the standard nonrefundable CTC is limited by tax, not a credit for additional child/ren :-)

• lines 20-24: get refund for overpayment (19 minus 16) or pay underpayment (16 minus 19)

If your 'overpayment' is really an excess of refundable credit(s) over tax, it is refunded just as if it had been an actual payment. People with low taxable income and thus no actual income tax due or paid can still receive a 'refund'.