Should we really be looking at the expense ratio while choosing funds?

Yes.

While the expense ratio is deducted from the NAV, it is still relevant. Imagine you have two index funds tracking the same index. One with 0.1% and the other with 1% TER. Obviously the cheaper fund will outperform the expensive one.

Now we add a layer of complexity and pick active funds. The expected return of a randomly chosen fund before costs is the market average. As high performing funds do not tend to keep their outperformance in the future, this is equal to choosing a well performing fund. Therefore expenses are the most significant predictor of your long term return after costs.

PS: you might no have the impression that funds perform average on average. Check for survivorship bias

Lowering the expense ratio, according to my understanding, only matters when we have another fund with a similar distribution of holdings, and has a lower expense ratio which yields a higher return.

That's a big "only", since investment companies usually offer similar funds.

Past performance accounts for the expense ratio, but is (proverbially) not indicative of future results. In particular, going by past performance and choosing a fund that has recently beaten the market will, according to the efficient market hypothesis (EMH), give no greater chance of beating the market in the future. In fact, it may incur excessive risk.

The expense ratio is one thing that really is predictable -- it is a direct drag on your return and this predictability does not conflict with EMH. So, looking at the expense ratio separately is a way of distinguishing signal (a recurring effect on returns going forward) from noise (past performance fluctuations).