Which one is a correct way to write: a) two functions problems, or b) two-function problems?

Having corrected the tag, its description gives a hint to the answer:

A compound adjective is when two or more adjectives are combined to modify a noun. In many instances, such compounds are hyphenated.

Adjectives in English are not inflected for number. We talk of red apples not reds apples, even when there may be several reds (shades of red) involved.

It's the same as ten-dollar bill, ten-pound note, six-foot man, thirty-kilo bag of potatoes... All of those adjectives have a singular dollar, pound, foot, kilo. The hyphens make it clear that the adjective comprises two words.

While a ten dollar bill is understandable, something like "six star fish" may not be. That rather poor example could refer to six creatures or a single fish with six stars. The hyphen is the customary way of representing which words are linked, and it can be used in all compound adjectives.

Thus you need a hyphen and a singular form: "two-function problems"