Rules for forming demonyms

As you have observed, there are a lot of ways to make demonyms in English. While we can draw some generalizations for which kinds of place names (toponyms) get which kinds of demonyms, there are no hard and fast rules. The various suffixes come from different linguistic sources, though, and that has some bearing on which ones are used where. We can observe some trends:

The Latin-derived suffixes -an and -ian are by far the most common choice for English demonyms. One reason is that English names for non-English-speaking places are often Latin-derived: Italian, Croatian, Arabian, Indian, American (the name predates America's settlement by English speakers). -an is also common for Spanish and Portuguese-speaking places: Mexican, Paraguayan, Mozambican. The simple suffix has become the go-to for most newer demonyms, even if the place names are not Latin at all: Hawaiian, New Jerseyan, Chicagoan. Variations include Torontonian and Panamanian. For toponyms that end in O or U, if the suffix is true to its Latin roots it becomes -vian: Oslovian, Peruvian.

Some places have Latin-based demonyms even though the English toponym is quite different from its Latin equivalent: Guernsey → Sarnian, Newcastle → Novocastrian, Halifax → Haligonian. And of course Norway → Norwegian, which spawned the analogous demonyms Galloway → Galwegian and Glasgow → Glaswegian.

-ite comes from Greek. Few Greek demonyms have survived into modern English, but the suffix lives on in the public consciousness due to its use for nations and tribes in the Old Testament. It has become another common choice for recently coined demonyms: Manhattanite, Wisconsinite, Tokyoite, Delhite, Vancouverite.

Many places in Germany have identical demonyms in German and English using the suffix -er, such as Berliner and my favorites: Hamburger and Frankfurter. -er is Germanic in origin and is used with many toponyms of pure English origin: Londoner, New Yorker, Marylander, Aucklander; and of other Germanic languages: Amsterdammer, Stockholmer.

-ish is also Germanic in origin but is less versatile because it's a adjectival suffix; you don't refer to someone as an Irish, a Spanish, a Turkish, or a Swedish. Some places have one demonym for the people collectively, and another for an individual: Irishman, Spaniard, Turk, Swede.

Suffix-less demonyms like Turk, Swede, Dane, Czech, Pole, and Afghan derive from their local languages. In fact, these are not demonyms in the strict sense because they do not derive from place names; rather, they are the opposite phenomenon, wherein the place name derives from the name of an ethnic group or tribe (thanks, @phoog).

-ese is Romance in origin. It comes from Latin -ensis, which is commonly seen in the scientific names of species. It is common in demonyms for Romance-speaking places and places that became connected with Europe as a result of explorations by Romance speakers, particularly ones in East Asia and French-speaking places in Africa. These demonyms also are more often used as adjectives or collective nouns, rather than referring to individuals (Japanese art, the Japanese, not a Japanese). When the toponym ends in -o and isn't Romance in origin, the suffix becomes -lese (Congolese, Togolese).

Several places in the Middle East and South Asia use the suffix -i, which probably has sources in several unrelated languages: Israeli, Iraqi, Emirati, Azerbaijani, Nepali.

And of course there will always be outliers with their own unique linguistic history: French, German, Dutch, Swiss, Greek, Icelandic, Argentine, Cypriot, Montenegrin, Edinburgoynian.

To answer your question, no, there are not concrete guidelines about which suffixes to use for which places, except to look it up. But when in doubt, if you choose -an people will get your meaning.

Lots of examples from https://en.wikipedia.org/wiki/Demonym