Can a non-cylindrical grip be better for jumping?

With good grips, your hands should stay pretty much planted in place. Since grips are relatively cheap (you can get a decent pair of lock-ons for $7). It is not recommended to "death grip" your handlebars when mountain biking as it increases the amount of shock that goes into your wrists, rather you should hold them with enough force that you are still holding on but loose enough that you can turn them slightly to absorb shock. One thing that may help as well is to make sure that your thumbs are on the underside of the grip, not parallel with the rest of your fingers.

As for non-cylindrical grips, that is probably not the solution. You will notice that most professionals use cylindrical grips, because non-cylindrical grips change the way you are connected to the bike, your posture, and in general negatively impact your control on rough terrain, especially jumps.

As a final word, if you are having trouble keeping your hands in one place, check to see if your grips are good enough (you may want to swap them out with some higher quality ones). In general, wearing gloves can improve grip but many people do not wear them so it is not a necessity.