Loading fontspec after hyperref/cleveref in preamble/format

I know that, normally, it is recommended to load hyperref and cleveref last. However, to cut down on compile time, I’m trying to precompile a my header as described here. Per these instructions, fontspec cannot be contained in the format file. However as hyperref and cleveref are large packages, it would be nice to have as much contained in the format as possible. Should doing this cause any issues or should those packages function fine?