Is there a rule for how to pronounce words such as "dance", "prance", "castle"?

Sorry to be pedantic, but it's not grammar. In linguistics, grammar is the set of structural rules that governs the composition of clauses, phrases, and words in any given natural language [Wikipedia]. The way words are pronounced is phonetics: the branch of linguistics that comprises the study of the sounds of human speech. [Wikipedia].

Pronunciation is one aspect of dialect. It's true that Americans and Southern Englishmen pronounce dance differently, although both can be represented /dɑːns/ because the /ɑː/ symbol between slashes is a broad transcriptional device. A more accurate transcription of the American dance might be [däns] or [dɐns]. Northern English makes the a in dance much shorter than Southern English and it's more like /dæns/. In the English West Country it's different again: [dɐᵊns].

As for rules, well, no: it's custom and practice. The American pronunciation of mass is different to the English, but Northern and Southern English are generally almost identical to each other, with a short vowel (some Southerners pronounce it /mɑːs/ but it sounds affected these days). Because pronunciation of vowels in particular is not matched to spelling — think of bough, cough, dough, tough as obvious examples — it's difficult to come up with any rules for how to differentiate between [ä] [ɑː] [ɐ] [æ] and [ɐᵊ] in dance. And there are some words where rules simply don't apply: one text-to-speech program I used insisted on pronouncing without as /woʊ'ʒiːtɑ/! I still don't know if there is any dialect of English which actually does that.

The only way of learning pronunciation truly accurately is immersion and practice.

A linguistic abstraction for comparison of vowel pronunciation in different dialects is called lexical sets, for UK and US English there are Wells lexical sets, of which you need BATH set. Some rules about trap-bath split show which words may fall into BATH set, though it is unpredictable. In the table there is a mix of BATH and PALM words, as they are indistinguishable in UK accent alone, so see also a more correct example list based on Wells' Accents of English or the book itself.