What is the best substitute for sugar when cooking thick jam?

The conservation issue aside: It depends heavily on what fruits you want to use. Fruits that contain a lot of fiber might not need any thickening agent to get a good, spreadable consistency. This includes for example certain variants of apples or pears, but also things like plums if you don't discard the skins. The consistency may be a bit different from traditional jam, though.

However, if you have something that contains a lot of water, such as berries, you only have two options: either add pectin or cook it until it reduces very much. The latter option may come with a loss of nutrients and taste, though, so let's concentrate on the pectin.

Normally, when you make jam, you add sugar that is already mixed with pectin to make the process easier. However, pectin can be bought in powder form, too (at least in Germany, you can find it in any better sorted supermarket). Furthermore, some fruits already contain a lot of natural pectin in their skins: black currants are a prime example [1]. If you like the taste and add enough of it, it might be enough to achieve the texture you want.

As for the amount of pectin needed: it depends heavily on what fruits you want to use. Pectin is activated by acid, so you might need to add some kind of acid to your jam to achieve the right consistency. It also needs to be cooked boiling for a few minutes. It's always a trial and error. Before I can my jam i always make a test by dropping a bit of jam into a bowl and put it in the freezer for a minute or two. If it has the right consistency when it is cold, I can it immediately.

Keep in mind though: The sugar is also a major component when it comes to a classic jam texture, so the texture without sugar will be different.

As for the sweetening without sugar: any sweetener should work fine, as long as you like the taste.

[1] side note: my mom once wanted to make black currant juice and instead it turned into jelly by accident. :D It was yummy, but a mess to get out of the bottles.

Edit: Regarding @rumtscho's comment: There are indeed several different kinds of pectin (I was only vaguely aware of that). Some need sugar to gel. The one I use is specifically marketed for jam production, so maybe that's something to keep in mind when buying pectin powder. In any case: it's worth a try. Just experiment a bit with it and hopefully you'll find what works for you.