How an I make use of an existing SE cable for a new outbuilding circuit?

The staples are fine and normal with the exception that more than 1 cable was under the staple in 1 location.

As far as the ampacity the aluminum #6 is limited by code to 40 amps it’s not a voltage drop issue.

Could you add a ground? If all 3 conductors are individually insulated it is a multi wire cable the latest code NEC 2021 allows individual conductors in a multi wire cable to be reidentified since you have conduit large enough a pull string could be blown and a new ground wire pulled in this has been allowed for a while now but the #6 al cable will not be enough for a 50 amp charger,

Downgrading to a 40 amp feeder breaker would be the appropriate path to take.

You can install a +100 amp sub panel with a 40 amp feed and update the wire size later this would be code compliant.

You need to replace this cable anyway

Your chief problem here doesn't have to do with ampacity, insulation types, or temperature ratings, actually: it's that the nitwit who installed this continued what appears to be a Type SE Style U (SEU, not USE) cable run through a buried conduit in direct contravention of NEC 338.12(A) point 2:

338.12 Uses Not Permitted.

(A) Service-Entrance Cable. Service-entrance cable (SE) shall not be used under the following conditions or in the following locations:

...

(2) Underground with or without a raceway

As a result, you'll have to replace at least the portion of the SE cable that's in the conduit, and at that point, you might as well rip the whole run out from the sounds of things.

What you can put in instead

At this point, you have two options. You can either replace the entire run with a special dual-rated TC-ER-JP/SER cable that can be used both inside a house as SER and in an underground conduit as TC (tray cable), or you can fit a pull box at the house end of the conduit and transition from ordinary SER cable inside the house to a mobile home feeder (tri-rated USE/RHH-2/RHW-2) cable, or individual XHHW-2 wires for that matter, in the conduit run. The former option requires that special cable, and limits you to a total cable size that fits in the 535mm² of single-cable fill available in a 1½" Schedule 80 PVC conduit, but it eliminates the need for the pull box and saves the labor involved with transitioning wiring methods as well. The latter limits you to a combined wire fill in the conduit of 442mm² and requires more work and parts, but can be done entirely using commodity parts.

For the former case, that 535mm² fill constraint limits us to a 2-2-2-4 TC-ER-JP/SE-R cable, using 471mm² of fill in the process. This does give you the ability to bring 90A to the outbuilding provided your panel and breakers are rated for 75°C usage, though.

In the latter situation, you'll need to use a 12x12x4" NEMA 1 pull box in the wall to transition from your buried conduit to SER cable inside the house. The conduit itself will admit a 2-2-2-4 mobile home feeder (80-90A), or even a set of 1/0 XHHW-2 Al (125A) for the hots and neutral with a 4AWG Al or 6AWG Cu grounding wire, which also determines what size your SER cable inside the house will be. You'll need to use suitable dual-rated mechanical splice connectors or power distribution blocks and a dual-rated grounding bar in the pullbox though.

SE rated dry and damp locations above grounded and can be exposed to sun. NOT RATED FOR UNDERGROUND USE IN CONDUIT OR WET LOCATION.

The underground section of the cable needs to be replaced

with underground-rated wire or cable, per ThreePhaseEel's answer here in this question.

Depending on the ease of doing one vs the other, you can either replace the entire run, or fit a junction box at the end of the conduit and splice to THWN or XHHW individual conductors in the conduit for the run underground. For 50A these need to be #6Al or #8Cu, because THWN and XHHW are definitely allowed 75C operation if the terminations at both ends are 75C. (NEC 110.14). You will need lug connectors to splice the aluminum; "MAC Block Connectors" are just the right size.

If you replace the whole shebang, consider replacing it with larger wire, but stay aluminum. For instance #2 cable is barely $1/foot, and is good for 90A - so it'll support your next 60A-tier (48A actual) EVSE which is fit for a nation-roaming EV.

Since you only need 30A for the EVSE, my suggestion would be do a subpanel today.

The subpanel also positively settles the question of whether the terminal is rated for aluminum or 75C, since subpanel lugs definitely are.

Your question: "At this distance my calculator indicates this should only be rated at 40amps with the 6 AL cable.".

Because the calculator thinks you have wire good for 60C thermal (40A). Actually, you have wire good for 75C thermal (50A), as below.

You will also need all the other rigmarole of a subpanel, including local ground rods and a main disconnect comprising no more than six switch throws. (and you can get 3-breaker handle ties (intended for 3-phase) to reduce the number of switch throws. So a 12 or even 16 space panel could be used with confidence.

Other than that, Mrs. Lincoln, 50A is fine on #8 SE cable if...

If... the socket and breaker are 75C rated. See this answer by Ed Beal:

if all components in the system are listed for use at 75C 110.14.C.1.B.2 allows 75 Column to be used.

So if your breaker and socket aren't listed and labeled for both 75C thermal and aluminum wire, get ones that are, easy peasy.

SE cable itself is allowed to operate at 75C because of this answer by ThreePhaseEel:

Just about all SE cable you use is not subject to the 60°C limit, unless it's limited by old or cheap terminations

Although there was a glitch in NEC 2008-2014 per my answer here, which reflect the constantly changing rulemaking on this particular point, although the "limit SE to 60C" faction has been roundly defeated at this point.