It's the phallic symbolism. In early Greek theatre, there were satyr plays put on where men dressed up in goat like costumes - that is as satyrs - with over-sized phalluses to symbolise their lustful nature. They were seen as low comedy in counter-point to the high tragedy which was the centre-piece of Attic theatre. This is where the notion of satire comes from.

As unbridled lust is seen as a capital vice, for example Aquinas writes in his Summa Theologicae:

A capital vice is that which has an exceedingly desirable end, that in his desire for it, a man goes on to the commission of many sins, all of which is said to originate in that vice as their chief source

This, given this, it's no surprise that this symbolism in Christianity, given its Greek roots, is now seen as Satanic.

Leviticus is mentioned as a source on the scapegoat as noted by @Obie 2.0.

Then Aaron shall offer the goat on which the lot for the LORD fell, and make it a sin offering. 10"But the goat on which the lot for the scapegoat fell shall be presented alive before the LORD, to make atonement upon it, to send it into the wilderness as the scapegoat.

Leviticus 16:10

Since Leviticus is originally the third book of the Torah looking into this gives an extra insight.

And Aaron shall present the goat upon which the lot fell for the LORD, and offer him for a sin-offering. 10 But the goat, on which the lot fell for Azazel, shall be set alive before the LORD, to make atonement over him, to send him away for Azazel into the wilderness. _{[Leviticus Chapter 16 וַיִּקְרָ}

_{Azazel drawing from 1863 by Collin de Plancycourtesy of Wikimedia}

Here the word scapegoat is actually Azazel, who is named in chapter 68, and according to the Book of Enoch is a fallen angel and as written in part 10:12

All the earth has been corrupted by the effects of the teaching of Azazyel. To him therefore ascribe the whole crime. _{Book of Enoch chapter 10:12}

In chapter 43 of his works, early church father Origen Adamantius mentions Azazel as a possible comparison to Satan.

Mark now, whether he who charges us with having committed errors of the most impious kind, and with having wandered away from the (true meaning) of the divine enigmas, is not himself clearly in error, from not observing that in the writings of Moses, which are much older not merely than Heraclitus and Pherecydes, but even than Homer, mention is made of this wicked one, and of his having fallen from heaven. For the serpent — from whom the Ophioneus spoken of by Pherecydes is derived — having become the cause of man's expulsion from the divine Paradise, obscurely shadows forth something similar, having deceived the woman by a promise of divinity and of greater blessings; and her example is said to have been followed also by the man.

And, further, who else could the destroying angel mentioned in the Exodus of Moses be, than he who was the author of destruction to them that obeyed him, and did not withstand his wicked deeds, nor struggle against them? Moreover (the goat), which in the book of Leviticus is sent away (into the wilderness), and which in the Hebrew language is named Azazel, was none other than this; and it was necessary to send it away into the desert, and to treat it as an expiatory sacrifice, because on it the lot fell. For all who belong to the worse part, on account of their wickedness, being opposed to those who are God's heritage, are deserted by God.

Nay, with respect to the sons of Belial in the book of Judges, whose sons are they said to be, save his, on account of their wickedness? And besides all these instances, in the book of Job, which is older even than Moses himself, the devil is distinctly described as presenting himself before God, and asking for power against Job, that he might involve him in trials of the most painful kind; the first of which consisted in the loss of all his goods and of his children, and the second in afflicting the whole body of Job with the so-called disease of elephantiasis. I pass by what might be quoted from the Gospels regarding the devil who tempted the Saviour, that I may not appear to quote in reply to Celsus from more recent writings on this question. In the last (chapter) also of Job, in which the Lord utters to Job amid tempest and clouds what is recorded in the book which bears his name, there are not a few things referring to the serpent. I have not yet mentioned the passages in Ezekiel, where he speaks, as it were, of Pharaoh, or Nebuchadnezzar, or the prince of Tyre; or those in Isaiah, where lament is made for the king of Babylon, from which not a little might be learned concerning evil, as to the nature of its origin and generation, and as to how it derived its existence from some who had lost their wings, and who had followed him who was the first to lose his own. _{Source. Translated by Frederick Crombie. From Ante-Nicene Fathers, Vol. 4. Edited by Alexander Roberts, James Donaldson, and A. Cleveland Coxe. (Buffalo, NY: Christian Literature Publishing Co., 1885.) Revised and edited for New Advent by Kevin Knight. http://www.newadvent.org/fathers/0416.htm}

In Märchen (aka "Fairy Tales"), the Grimms “Kinder und Hausmärchen” (Children’s and Household Tales) include a tale on how God created all animals and selected the wolves to be his hounds (KHM 1857 #148, "Des Herrn und des Teufels Gethier" (The Lord's and the Devil's Creatures). Since God forgot to create the goat, the the devil also wanted to create and made the goat with fine long tails. When they went to pasture, they usually got their tails stuck in the brambles. The Devil had to go in and with much effort free them. This angered the devil and he bit their tails off, as is still to be seen today in their stumpy tails.

The story continues....

https://khm.li/Des-Herrn-und-des-Teufels-Gethier

https://sites.duke.edu/adhoc_fairytalespublic/fairy-tales-by-khm-number/khm-141-160/148-the-lords-animals-and-the-devils/

"The Bolte Anmerkungen state that “An age old reason, says W. Grimm breaks everywhere through this fable; it belongs, as out of Dänhardts rich Nachweisen in the Natursagen 1, 127 – 205 belongs, to the large group of dual creation tellings. Out of envy the Devil copies Gods creations, without being able to achieve it; he creates a wolf, a goat, a horse, but only enlivens them through the word of God; instead on the human he creates an ape, instead of the bee a wasp, instead of the nut an acorn; or he is cheated in such a contest.”"

The Grimm text is from Hans Sachs 1556 Schwank “Der dewffel hat die gais geschaffen, hat in dewffel-augen eingeseczt” – “The devil had created the goat(s), had set devils eyes in.”

You can at least find 2 main reasons:

1. Goat gods are pretty old beliefs. You find them in almost every cultures spreading everywhere. Celts have Cernunnos. In norse myth you have Tanngrisnir and Tanngjostr. In roman myth Faunus and Hammon/ In greece Pan. In sumer you have Dumuzid, the Tammuz of the Bible. For what I am aware, as we found cavern drawings of anthropomorphic goats, it could well be one of the oldest cult. And the Bible is very keen to point out the falsehood of other Gods by demonifying them.
2. Per se goats are strange creatures, if you saw some in action. Specifically the male one, the buck. Those animals have an incredible libido and you see them having hard on all the time. And when there are females around them they are extremly demonstrative. I have seen some almost strangulating themselves to reach them. And they very casually piss on themselves; Specifically the legs, the face and in the mouth (yes). Not something you see the other flock animals like horses, sheep or bull do. Beside that, goats are prone to eat a lot of things. Give them paper and enjoy them chewing on it... An horse will ignore the paper simply.

Bible uses goat to symbolize sin rejection with the concept of scapegoat. It just makes clear the proximity between goats and human beings. In other words, Bible replaced human sacrifice by goat sacrifice. And hence why devilish creatures tend to use the feature of the goats. Because devilish creatures are full of sin.