Set of functions whose

I have always used the first one despite the occasional qualm. The principle is that "whose" refers to the collection and so does not get the plural. We do not often use the word "derivatives" everyday but we do in mathematics often. I think that is where we get the rub.

The first one would imply the derivative of the set, or at best it will be ambiguous, if functions somehow share the same derivative.

The second - derivatives of the functions. Correct.

And "the" instead of "a" is more appropriate.