How is "æ" supposed to be pronounced?

I have never heard an American say it any other way than "Encyclo-pee-dia". I have no research to back up that pronunciation, but you will not sound strange if you say it that way. (among Americans)

English orthography is rule based...except it's not very good at following the rules. Sometimes it uses a regular literal one-to-one pronunciation, at oher times the spelling got stuck centuries ago but sounds changes occurred in speech, and sometimes, the word is written as from the foreign language it was borrowed from but the impossible or unlikely pronunciation is adapted to English mouths and ears.

The pair 'ae' or the single mushed together symbol 'æ', is not pronounced as two separate vowels. It comes (almost always) from a borrowing from Latin. In the original Latin it is pronounced as /ai/ (in IPA) or to rhyme with the word 'eye'. But, for whatever reason, it is usually pronounced as '/iy/' or "ee". Encyclodpeeedia, alumneee (for many female 'alumnae'). Another variant is /ɛ/ in an-eh-sthetic for 'anaesthetic'. Note that many of these spellings are now variants and the more common spelling removes the strange looking 'a'.

Another pair borrowed from Latin is 'oe' is in (the old fashioned spelling) 'oesophagus' where it is pronounced /ɛ/ 'eh' eh-sah-fuh-gus.

Encyclopaedia is a Greek work. It is a compound word and it has three morphemes: en - cyclo - paedia, meaning in - cycle - education (general education).Paedia comes from the Greek word παιδεία /pε:δΙ'Λ/. So, the spelling is influenced by the Greek spelling just like all the other Greek words mentioned above in other posts. In some cases pronunciation stays the same as in Greek like in anaesthetic.

In most cases, "ae" or "oe" will result in a long or short "e" sound. These spellings originated in Greek and found their way into English. Many of them have changed as spelling is "reformed," but others have not.

Examples:

• Oedipus - "oed" = "ed"
• oesophagus - usually spelled "esophagus" now
• Aegypt - now spelled "Egypt"
• anaesthetic - sometimes spelled "anesthetic"
• paedophile - now spelled "pedophile"

As for "daemon" -- despite what you will hear from some computer people, it is pronounced "demon" -- and despite what you will hear from some others, they are really only variant spellings. The older spelling "daemon" came to be used in the computer sense, similar to when the "compact disc" was introduced to an international English-speaking audience, the original "disc" was used, even though the spelling of "disc" had mostly been reformed to "disk" by that time. This resulted in the current situation in which "compact disc" and "hard disk" are spelled differently.

Now... When "ae" is used at the end of a Latin word, it is technically pronounced "eye." I say "technically" because it's confusing that the "real" pronunciation of "alumnae" sounds like the the popular pronunciation of "alumni" (which "really" should be pronounced "ah-loom-nee" which probably only happens inside a Latin classroom.

The Danish alphabet has both the mentioned vocals, ae = æ and oe = ø. Æ is pronounced very close to e in echo, and when I read the English word encyclopædia I naturally pronounce it as described, confusing an American listener. ø or oe is pronounced as the German ö, also as a single sound. Needless to say, I'm Danish.

You may be mixing up the IPA pronunciation symbol æ and the alphabetic letter æ.

In English text, the letter is used as a slightly old-fashioned form of the Latin digraph ae (also in Latin-mediated Greek words) and in some names from Danish, Norwegian, Old English and a few other languages that use the letter natively.

The pronunciation doesn't have to be anything like the IPA [æ].

For Latin loanwords in ordinary English text, it's essentially equivalent to the letter "e" (so always "encyclopEEdia", "julius cEEsar") but in the study of Latin language and culture it's common to pronounce names and terms in ways more similar to how the original speakers did.

In Danish (etc.) names, you'd adapt to what ever approach you would otherwise use for those names in English text. It may end up actually sounding like the IPA [æ].

To find out how a word is pronounced, you can look in a dictionary. To find out how a word "should" be pronounced, you have to decide what you mean by that, and then look up whatever information you think is relevant (e.g. the word's etymology, the frequency of any variant pronunciations, what usage "mavens" have written about the word).

The symbol æ has multiple distinct uses.

A ligature of the Latin ae digraph

In fact, æ isn't used much at all in modern English writing, but most of the time when you see it in that context, it is not being used as a distinct letter, but just a ligature of the Latin-derived digraph ae (which is the letter a followed by the letter e). Writing this digraph with a ligature has fallen out of fashion, but some writers may use it to create a particular effect, or just out of eccentricity (compare perhaps the similarly uncommon use of the diaeresis in words like coöperation). Related questions: Is the word “formulæ” valid English?, Is it acceptable that I use ligatures and diæreses? In many words, it is usual to write e instead of either ae or æ: this statement is true especially in American English, but there are many words for which it applies to British English as well. For example, the spelling demon, which is usual in both American English and British English when writing about evil spirits or supernatural entities, comes from a spelling variant of Latin daemon. On the other hand, one word where ae is still common in both British English and American English is aesthetic(s).

As others have mentioned, the Latin ae digraph in English words is usually pronounced as /iː/ (the "long e" or "ee" sound found in the word fleece) : this definitely is true for aeon, aether, and encylopaedia.

Sometimes, especially in American English, Latin ae is instead pronounced as /ɛ/ (the "short e" sound found in the word dress). This pronunciation tends to occur in "closed" syllables (a closed syllable is a syllable that ends in a consonant) or in stressed syllables that are followed by at least two other syllables. An example is the word aesthete, which is pronounced with /ɛ/ in American English, although in British English it may be pronounced with /iː/. This same pronunciation difference (/ɛ/ in American English, /iː/ in British English) may occur for words that are spelled with "e" in American English but "ae" in British English (Brian Nixon mentioned the example pedophile vs. paedophile in a comment).

A third pronunciation, /eɪ/ ("long a", as in face), exists for ae in words taken from Latin, but it's a bit harder to explain, so I've postponed my discussion of it to the end of this answer. Also, I think it's not very common for words pronounced with /eɪ/ to be written with the ligature æ.

Latin ae was not always a digraph

In Latin, not all instances of the sequence ae were examples of the ae digraph. Occasionally, a followed by e in Latin just represented an "a" sound followed by an "e" sound, pronounced "in hiatus"; that is, in separate syllables (for example, in the Latin words aeneus adj. "of bronze/of copper/brazen" and aer "air").

But related word in modern English may have developed pronunciations without hiatus, perhaps in part because of the ambiguous spellings or analogy with related words that are pronounced without a hiatus: e.g. aeneous (OED pronunciation: "Brit. /ˈiːnɪəs/, /eɪˈiːnɪəs/, U.S. /ˈiniəs/, /eɪˈiniəs/") and aerial (OED pronunciation: "Brit. /ˈɛːrɪəl/, U.S. /ˈɛriəl/").

In theory, words with ae in hiatus should not be written with the ligature æ, but in practice, it's possible to find examples of this from times when æ was more common.

The Old English "ash" letter

In Old English, ae and æ were used to represent the monophthong /æ/ (which could be short or long). In this context, the symbol æ came to be considered a letter of its own, with the name "ash" (in Old English, "æsc").

Rarely, you will see this symbol in a modern English text when somebody is using an Old English name like "Æthelred".

The pronunciation is usually an approximation of what we think the Old English pronunciation was, so something like the "short a" sound in the modern English word trap (IPA /æ/, or in some systems /a/), or perhaps for some speakers the "short e" sound in the modern English word dress (IPA /ɛ/).

The pronunciation of Old English "ash" is irrelevant to the pronunciation of the ligature æ in English words taken from Latin.

History of the Latin "ae" digraph

The Latin ae digraph replaced an ai digraph that was used in Old Latin. Scholars think that the sound was pronounced as a diphthong [ai] in the Old Latin stage. Its reflexes in the Romance languages point to a monophthong *ɛ in the common ancestor of these languages ("Proto-Romance").

Based on the evidence, the reconstructed development of the Latin sound is from a diphthong [ai] or [aj], to a diphthong with a more open final element [aɪ] or [ae], to a monophthong [ɛː], which merged with the reflex of the Latin "short e" sound to become *ɛ in Proto-Romance. (In some cases, Latin "ae" instead corresponds to Romance /e/: this is thought to represent an alternative development where at some earlier point before the loss of vowel length, ae became a long monophthong and merged with Latin "long e", which became Proto-Romance *e.)

During the Classical period, Latin ae was used as a transcription of the Greek digraph αι. The words demon, aether/ether, aeon/eon all come from Greek words spelled with αι: δαίμων, αἰθήρ, αἰών.

The conventional correspondence of Latin ae and Greek αι was maintained in "Classical" words in modern languages for a while, but eventually some scholars came to feel that the direct transliteration ai was somehow better or more appropriate. So in the modern English spelling of recent borrowings from or coinages based on Ancient Greek, we see variation between ae and ai (compare similar variation between oe and oi, y and u, u and ou, c and k). For example, the spelling daimon has been used by authors who wanted to use the Greek word δαίμων in English while avoiding the connotations of the word demon.

Part of the variability in the pronunciation of ae in modern English may also be based on the efforts of classical scholars. As I mentioned, Latin ae is reconstructed as having the pronunciation /aɪ/ at some point after the Old Latin period and before the end of the Classical Latin period. Latin teachers in favor of using a "restored" pronunciation decided that because of this, the English sound /aɪ/ (the "long i" sound found in the word price) should be taught to English-speaking Latin students as the pronunciation of Latin ae.

A number of speakers now use these "restored" qualities in certain contexts in English words that come from Latin. I mainly hear /aɪ/ for ae from modern English speakers when the ae is word-final—there is a separate question asking about the variation between /aɪ/ and /iː/ in this context: Pronunciation of words ending with “‑ae”.

The "ay" pronunciation (IPA /eɪ/)

As mentioned in the question, there are a few words spelled with ae that seem to have pronounciations with the "long a" sound, as in the word face (IPA /eɪ/).

The origins of this are not entirely clear to me. It seems likely that in at least some cases, it is based on the medieval or "Ecclesiastical" pronunciation of Latin ae as a monophthong like [ɛ] or [e] (compare the use of English /eɪ/ in certain English pronunciations of a number of words that have /ɛ/ in modern French, such as ballet and crêpe).

It may also be influenced by the use of "a" + a consonant letter + "silent e" as a representation of /eɪ/ in native English words, and the use of "oe", "ie", "ue" as a representation of word-final "long o", "long i" and "long u" in a number of words.

In addition to daemon, the word aegis (mentioned in a comment by Lambie) has a variant pronunciation with /eɪ/ instead of /iː/. The sound /eɪ/ is also used as a third variant pronunciation of the -ae plural ending, in words like vertebrae.