Is 'the course' a complement or an adjunct in 'Stay the course'?

Neither is correct. "Stay the course" is an idiomatic expression and therefore the verb "to stay", as far as its use in that expression, is not classified as a transitive verb or an intransitive verb (ref.); in consequence it is not possible to say whether "the course" is an object or an adverbial.

Addition

Here is a complement to consolidate the notion that the grammatical analysis of idioms is not really possible. It is contrary to the notion of compositionality (ref.).

Compositionality is the key notion for the analysis of idioms. This principle states that the meaning of a whole is constructed from the meanings of the parts that make up the whole. In other words, one understand the whole if one understands the meanings of each of the parts.

This is what can be read in a pdf on idioms available in The International Journal of English Linguistics.

They are expressions, not subjected to analysis, only some syntactic changes may be carried out in them. Moreover, the diagram from the same source shows clearly the non grammatical approach on the left for the idiom and the usual grammatical analysis on the right for the "regular" meaning of this phrase.

The original meaning of "stay" was

I. intransitive. To cease moving, halt.

1640 tr. G. S. du Verdier Love & Armes Greeke Princes i. xxii. 96 Their Bark (= boat) staying at an Island,..they went on shore.

Note that “He stayed the night” can be rendered as “He stayed for the night” or “He stopped the night” or “He stopped for the night”.

or transitive = to cause to halt:

1816 "The History and Antiquities, Ancient and Modern, of the Borough of Reading" By John Man "And also his majesty further stayed his horse until the mayor had taken his horse."

In most cases, this meaning gave way to the idea of “remaining” and, in this sense, both of your examples are much the same and basically carry the meaning of “to maintain or remain or endure in a constant state or action. [1]

In both cases, the verb acts only on the subject: neither “the course” nor “the night” experience being “stayed”. Thus, in these cases, the verb “to stay” must be intransitive.

There is no reason why a word cannot be both a complement and an adjunct: in your case, what follows ‘stay’ can also be described as a dative (= for the course / for the night)[1] Thus in all cases, the noun in question becomes a modifier/adjunct.

The OED describes the verb “to stay” (the course/the night) as

6.a. With predicative complement:* To remain in the specified condition.

And gives examples:

1640 J. Suckling Ballad on Wedding 38 Her finger was so small, the Ring Would not stay on which he did bring, It was too wide a Peck.

1871 B. Taylor tr. J. W. von Goethe Faust II. ii. iii. 150 She grows not old, stays ever young and warm.

In the entry for “to stay the course” we have:

12.a. Sport. To last, hold out, exhibit powers of endurance in a race or run. Also, to hold out for (a specified distance). [? Derived from sense 7b[2]]

1860 Rous in Baily's Mag. I. 18 There is another popular notion that our horses cannot now stay four miles.

1897 T. C. Allbutt et al. Syst. Med. II. 841 [Alcohol] may enable a man ‘to spurt’ but not ‘to stay’.

[1] II. quasi-transitive and transitive uses derived from I.

17.a. quasi-transitive. To remain for, to remain and participate in or assist at (a meal, ceremony, prayers, etc.); to remain throughout or during (a period of time). *= to stay for —— vb. at sense 14. *

1599 J. Hayward 1st Pt. Henrie IIII 26 The rest of the lords departed, except the Earle of Darby, who stayed supper with the King. 1888 G. Gissing Life's Morning II. xi. 135 I'm obliged to ask them to stay tea.

17.b. to stay the course: to hold out to the end of a race. Frequently figurative. 1885 Daily Tel. 11 Nov. 3/7 Doubts are also entertained..concerning her [sc. a horse's] ability to stay the course.

1966 Listener 10 Mar. 365/3 There was much to be learnt from this programme—about metal fatigue, for instance—for those who could stay the course.

[2]7. With emphasis or contextual colouring:

7 b. To stand one's ground, stand firm (as opposed to fleeing or budging). Now rare.

a1616 W. Shakespeare Henry VI, Pt. 3 (1623) ii. iii. 50 And giue them leaue to flye, that will not stay . [ And give leave to flee to them who do not wish to remain here.]

1851 E. B. Browning Casa Guidi Windows i. xxvii. 74 Who, born the fair side of the Alps, will budge, When Dante stays, when Ariosto stays, When Petrarch stays, for ever?