Why was Susan treated so unkindly?

There isn't a great deal of in-text evidence - other than Susan having grown too worldly and her fascination with lipstick, nylons and invitations having replaced her faith in Narnia.

The explanation, considering the author is probably most simply that Lewis had a dim and uncharitable view of women. Girls were fine - see Lucy and Polly as innocent, inventive, nurturing figures - Jill has a little more agency, but is still more a 'plucky gel adventurer.' Grown women however seem to have a rougher time in his work - Jadis (the White Witch) being central.

Lewis's strong Christian faith seems to cling heavily to the 'sins of Eve' idea of womanhood. Even though it's Edmund who succumbs to temptation, he repents and Jadis is to blame as the tempter. Given his era, and general priggish attitude, Lewis was of the opinion that women were the weaker and more fallible sex and that their role was best as helpmeet. Mentioned in that while Aslan gave all the Pevensie siblings weapons, Lucy and Susan were cautioned not to use them except in dire need as he (Aslan/Christ/Lewis) did not intend them to fight.

His Pelelandra novels also repeat the theme of Eve/Woman being the weak link which the Adversary can exploit. In this novel it's for his rugged Christian hero to debate the tempter, to protect Tinidril (space-Eve) as she can not be trusted to withstand evil.

Finally, Lewis's general approach to society and morals valued the virtues in the Pevensie siblings, but saw male virtues (as held by Peter and Edmund) as ones which remained through life because men did not get distracted away from them through fripperies and moral weaknesses (as discussed obliquely in The Screwtape Letters.)

This is in keeping with an Edwardian upbringing, and he was known to be resistant to ideas outside of his own - the classic academic 'If I'm not interested in it, then it can't be important' viewpoint. Hence his dismissal of lipstick, nylons and invitations as being wasteful distractions, unlike presumably his tobacco, correspondence and drinking with the Inklings.

It could also be that Lewis just needed an example that 'Not everyone gets saved' and that living a moral Christ-devoted life was part of it (he disliked the Catholic idea of absolution through 'words not deeds') and needed a character to be distinguished.

Neil Gaiman wrote a interesting little short story on The Susan Problem and Phillip Pullman cites it as a key inspiration for his Dark Materials Trilogy (His Anti-Narnia series.) Rowling has also mentioned it (as Adamant points out in the comments) but given the core position of Narnia in the British Children's literary canon it's a question every reader must have asked themselves.

If you consider the characters as free agents, Susan chose to throw it away. Why should that choice not be honored?

If you consider the literary perspective of writing the Last Battle and you want the symbolic number 7, one has to be excluded and there aren't too many options that don't break things.

But if you don't like the answer and want to be a writer, consider the following scenario:

A few days later, an official from London calls upon Susan, and returns to her the effects of her siblings found in the train wreck, including a box with some rings in it. She picks up a green ring, and then a yellow one, and screams and screams. When she's done screaming, pick a pool, any pool. The parallel time rules being what they are, this adventure can last tens of thousands of years.

And there's still no reason to believe she doesn't make it through in the end.

Here is an interesting quote from The Horse and His Boy:

“And where is the Queen Susan?”

“At Cair Paravel,” said Corin. “She’s not like Lucy, you know, who’s as good as a man, or at any rate as good as a boy. Queen Susan is more like an ordinary grown-up lady. She doesn’t ride to the wars, though she is an excellent archer.”

So it seems that Lewis thought most women are inferior than men. There may be some exceptions like Lucy, but not Susan.

According to a letter to a young fan, Lewis made the decision to keep her out of Heaven because he felt the alternative would be too complicated a story for him to write. He then asked the girl to consider concluding the story herself.

I could not write that story myself. Not that I have no hope of Susan ever getting into Aslan's country, but because I have a feeling that the story of journey would be longer and more like a grown-up novel than I wanted to write. But I may be mistaken. Why not try it yourself?

Lewis also wrote in his Letters to Children that Susan may one day reach Heaven by a different path, as well as further explaining why she hasn't made it yet.

The books don't tell us what happened to Susan. She is left alive in this world at the end, having by then turned into a rather silly, conceited young woman. But there's plenty of time for her to mend and perhaps she will get to Aslan's country in the end... in her own way.

What (either in or out of universe) was the motive for giving Susan such a hard lot without even explaining what led to it?

Lewis wanted to show that one could fall from grace. It had to be a character that we cared about. Let's look at the lead characters from the series:

• Peter - the natural leader of the group. It would have felt out of character for him to fall.
• Edmund - he had already fallen and been forgiven/redeemed.
• Eustace - he was fundamentally changed by his time in Narnia. It would have been silly for him to fall.
• Digory - he was portrayed as wise and still a believer in his old age. He couldn't really fall.
• Polly and Jill - were both only covered in one book. It could have been one of these two but they weren't quite as central to the story. One may have assumed that Lewis accidentally omitted them if he were to have picked one of these two.
• Lucy - was obviously Lewis' favourite. She had already been tempted by vanity and jealousy and had resisted the temptation in Voyage. It may have been more shocking but less in character for Lucy to have fallen.
• Susan - central to two of the books but not as important as Lucy. The obvious choice to portray as the "fallen" one. Susan was never quite as well fleshed out as her siblings and so despite her prominence, she was the most malleable major character for Lewis to do this to. Note that in the books she was never as close to Aslan as Lucy and was always the most sceptical of the children.

Susan's sin was not "discovering sex" or womanhood (as some have suggested), rather, her sin was vanity. Being more concerned with appearances and socialisation than with Narnia (remembering that Narnia was an allegory for the Christian faith).

We are never really told the conclusion of Susan's story. It's clear from the texts that she was not with the others at the time of the train accident. She could well be alive now (albeit in her 80s). One of the themes of the Narnia stories was that of redemption. I think Lewis wanted the reader to think on this and picture themselves as Susan, essentially posing the question "are you going to let frivolities consume you or will you focus on your Christian faith?" The fact that Lewis left Susan's fate open was intentional. There was still hope for the character if she turned away from her vanity.

Per @ImperatorHelevetia's answer above, The Problem of Susan takes this and runs with it. One thing that Gaiman points out is not just the cruelty in the afterlife, but the cruelty here on Earth by forcing a child to identify her family's mangled bodies in the aftermath of a train crash, as a clear outcome of the decision by Aslan that she was not fit to come back to Narnia. This is always the question that a skeptic asks a "believer" – namely, if there is a God, why does He/She/It allow these things to happen?

Lewis's own answer (repeated several times in "Shadowlands") is "God wants us to grow up", and all the suffering and pain is how he gets us to grow up, so that ultimately it's for our own good. Many skeptics (e.g., Philip Pullman) would see this as the god being actively evil and immoral. Believers may say that those judgements are beyond human comprehension. YMMV.

A separate possible answer is that God does not intervene, and that's how many people of faith reconcile their faith with the obvious problems in the world. Lewis did not take that approach himself, and it clearly isn't applicable to Aslan who does intervene in the affairs of Narnia.

It's worth pointing out though that this is a children's book. If you're old enough to work out second-order problems like this, then you're old enough to look beyond children's books for philosophy! :) He wrote plenty of books which explained his viewpoint in much more detail. I happen to disagree with him, but it's instructive to follow his reasoning, even if you can see the holes in the arguments.

none of the others seem to spare a thought for their sister and friend whom they now know they will never see again.

Lewis addresses this point in The Great Divorce in two ways.

The first is the time between the train crash and whenever Susan dies later. People arrive in Heaven at different times, and (if they get there and accept Heaven) are happy together regardless. Lewis tells us that time is not relevant to Heaven, because all times are the same. Heaven is guaranteed for those who accept it, so Susan may well get there another time, and it's all the same time, so there's no reason to be upset about a delay which doesn't really exist.

And the second way is that if someone chooses not to accept Heaven, then their family and friends in Heaven can't be brought down by them. Lewis is clear that God doesn't refuse people access to Heaven; instead each individual chooses whether to accept Heaven, and if their soul has been corrupted enough by evil habits then they will actively choose to reject it. So happiness in the afterlife is a choice. From that, Lewis argues that you can't allow someone who refuses to be happy to spoil things for others (else, as Lewis says, "ye'll make a dog in the manger the tyrant of the universe") because Heaven is a place of happiness.

The entire series exists in a state of tension between the story as a story, and the story as a Christian allegory, and in The Last Battle in particular, the allegorical aspects clearly dominate.

In Lewis' Narnian version of Christian theology, it is possible (and even acceptable) to worship Christ (Aslan) without knowing it, but what is not acceptable to have known Christ (Aslan) and have chosen to value something else more. Lewis considers this a real and present danger for a Christian, and so he chooses to dramatize it in this book, as a (rather severe) warning to the reader. The fact that Susan's decision is seemingly both inconsequential and understandable is a part of the warning.

Whether this is good theology, good storytelling, or a good combination of the two, is a matter of considerable debate, but Lewis' motivations seem relatively clear. The series is intended to do real evangelical work, as well as entertain and enthrall, and if one goal has to be sacrificed to the other, it is never in doubt which one Lewis will pick.

CS Lewis was trying to draw a Christian analogy here (a common theme in Narnia). In this case, he seems to be touching on a thorny issue: Christians losing their salvation. I'll try to stay as on-topic for SF as I can for this but it does require some knowledge about Christianity in general.

Jesus once gave an analogy of a man sowing seed. One part of that analogy was seeds falling among thorns/weeds and being choked away. Based on what little we know of Susan's falling away, it seems that is the case here

“Oh Susan!” said Jill, “she’s interested in nothing now-a-days except nylons and lipstick and invitations. She always was a jolly sight too keen on being grown-up.”

“Grown-up, indeed,” said the Lady Polly. “I wish she would grow up. She wasted all her school time wanting to be the age she is now, and she’ll waste all the rest of her life trying to stay that age. Her whole idea is to race on to the silliest time of one’s life as quick as she can and then stop there as long as she can.”

What I've seen Susan as, in the end, is simply left out of the urgency to return to Narnia. I'm going to call the group of people killed in the railway accident the Narnia Club for simplicity. Consider how we got here

1. Tirian appears before the Narnia Club as a ghostly apparition
2. The Narnia Club decides they need to return to Narnia. The wardrobe hasn't worked since book 1
3. Professor Kirke (who owned the infamous wardrobe, made from wood from a tree from Narnia) still had the rings from The Magician's Nephew
4. While trying to convene everyone to use the rings, a railway accident happens, killing the entire Narnia Club

Susan just missed out.

The reasons why seem simple enough. Susan clearly went on with her schooling and likely fell in with the popular crowd. How do you explain to them that there's a real world with talking animals? Oh, and she doesn't know how to go back there either to prove what she's saying is true. I mean, if I told you Middle Earth was a real place I had personally been to, you would demand proof and, if I couldn't produce it, you would think me crazy.

Welcome to the world of faith. It's not easy to believe in things that cannot be empirically proven as real. It's not surprising that Susan got caught up and abandoned her faith in a bid to remain popular. Given Lewis' letter, I don't think he meant to leave her there. But, as he said, her path would have been longer and far more adult. Lewis himself, sadly, just ran out of time to tell it himself.