What material properties determine a substance's photoelectric work function?

Take only a pure material.

The concept of a work function for a bulk material can be initiated starting from the idea of the first ionization energy of an atom. As the atoms form clusters to make a bulk material, the hole that is created as the electron leaves can delocalize throughout the cluster. In an infinite bulk material, the work function is then conceptually the first ionization energy of the atom minus the delocalization energy as the hole delocalizes. Essentially, when an electron leaves a single atom, it feels the pull of the single (positive) hole left in the orbital on the atom. When it leaves a bulk material, to the extent that the hole is delocalized, the force felt by the electron that leaves is lowered. Therefore the energy needed to remove the electron is lower, and therefore the work function (an energy) is lower than the first ionization energy.

To first order then, the work function indicates the depth of the valence level in the material. The depth of the valence level in a bulk material is correlated with the depth of the valence level in a single atom of the material itself. The are both the equivalent to what is called the binding energy of the electron, either in the bulk (work function) or the atom (first ionization energy).

For single crystals, the work function also depends on the orientation of the crystal plane where the electron leaves. This is inherently because the surface charge density distribution varies from one crystal plane to another. For example, the closest packed plane in an FCC metal structure is the {111} family versus the {100} family. As you can see in this reference the work functions of the closest packed planes are lower than the more open planes. The higher electron surface density "pushes" the escaping electron more, causing a lower work function.

Bulk mixtures such as alloys for metals or solid solutions for other substances have their own interesting behavior that I would dare say requires its own treatise. You could start with this or this and expand your search as your interest dictates.