Interpreting a precision recall curve

a precision recall AUC score of only .44 seems very low.

I agree, but your model has definitely learned something because you beat your naïve baseline (technical term not criticism!) Depending on what you’re trying to predict, that AUC might either be impressive or disappointing. By a similar token, the model you have trained here might be useful as it is, or might not have reached a minimum viable level of performance.

How do I interpret this?

When your model is able to retrieve half of the positive samples (~50% recall) it happens that about half of the samples it identifies as positive are actually negative (~50% precision). Beyond that, the precision decreases as we attempt to get greater recall, which is what we’d expect.

Is it actually a bad score?

Has anyone tried to predict the thing you’re trying to predict before? Maybe there is a more complex baseline you could compare yourself to. If not, you’ll need to do more work to establish whether you could do better. For an unbalanced dataset, gradient boosted trees are a good baseline beyond the naïve (no learning takes place) baseline. Try something like xgboost if you haven’t already. Take the model you just trained as a new baseline and try some other stuff.

How can I understand this curve in reference to the baseline?

The baseline you are using represents the precision you would see if you just always predicted that samples were positive. This would obviously result in 100% recall. If your model moves above that in terms of precision, anywhere on the graph, it may be learning something. It’s clear this is the case as it is doing consistently better.