What is the plastic inside this barrel adjuster, and is it okay to be worn?

I do not have the name of the part, but it's likely that it is a Teflon sleeve. A similar piece is found inside linear pull brake noodles. In cheap ones it is just plastic. In good quality ones it is a somewhat resistant plastic with very low friction (I'm not really sure if it's actually made of the material commercially known as Teflon)

In the case or a brake noodle, as you see, it is a curved part so the cable makes full contact with the inner part or that curve and it is subjected to very high force. (I don't know how to calculate, bu I guess its almost all the force your hand can apply, multiplied by the brake lever's advantage).

This sleeve is part of the noodle and I've never found it sold separately, but a noodle is an inexpensive replacement part.

If you needed to replace the sleeve on your adjuster, you could get a sleeve from a noodle, just cut it to size. But also a barrel adjuster should be inexpensive to replace, you may be able to find ones that include that sleeve; although I've never seen a barrel adjuster with a sleeve before.

For MTB brake levers, most designs I've used have a mostly direct cable route from the cable stop through the adjuster and into the cable outer jacket (housing) and there is very little fiction between the cable and the adjuster. So, my guess is that in any of my bikes, if such sleeve where used, It would have got almost no perceivable benefit, and also, very little wear. None of my brake levers has a protective sleeve inside their barrel adjuster.

In your case, it seems like the cable exits the adjuster at an angle and that caused wear as the cable slides back and forth. Most likely you can make your brakes work without it, but the cable will eventually carve a channel into the adjuster, causing more friction and a rough feel when operating the brakes, at which point you'd need to replace the adjuster (They should be cheap, so once with every cable change shouldn't add much to the cost)

In good quality brake noodles that sleeve lasts somewhere between one to two years (In my case usage is once a week MTB use on wet, muddy environment).

All this time Im thinking of the adjuster in the lever assembly (I'm guessig that's what you call the handle) But similar adjusters may be found in cable stops in the frame, in the other end of the cable where it attaches to the calipers or in rear derailleurs.

Some cable housings have an inner protection layer made of similar material. If your replacement housings have it, and you have cut a short surplus, you may be able to extract a piece and use that to replace the sleeve in your adjusters.