How can I convert audio to a series of notes using pitch detection?

Instead of looking for note decays into silence, you might try looking for high amplitude or delta amplitude attack sounds, and associate them with the following estimated pitch, if that pitch is detected shortly enough after.

You could have your pitch detector return, not only the estimated pitch, but some statistical probability or reliability factor of the pitch being a certain note in a given musical temperament (versus noise, or some other note, or halfway between notes, etc.). Then look in your pitch detection stream for when the probability values of two adjacent detected notes cross over.

If training an ML, you might look at the values returned by an inference vector of "note" bin weights, and compare ratios.

A simple approach would be an alternative mechanism for detecting silence that provides you with a more adequate response time. You could return null if your silence detector detects silence, else return the state of your pitch detector. If you are happy with your pitch detector, this has the benefit of not requiring modification to it.

This should work using a threshold on the by-bin-difference of the magnitude spectrum.

• Calculate the abs fft for current window and normalize it.
• Do a bin-wise difference with the normalized abs fft of the last window.
• Sum up the bin-wise difference and compare it to a threshold.

Explanation: A change of note means a change of energy distribution over the fft bins. So the sum of bin-wise difference should be high, if a change of note has occured. Normalisation is to suppress false detection of note change due to change of volume.