Logging & Monitoring on multiple servers

I lead a small project with 2 groups of servers (servers in a single group are clones with the same microservices, like pods in the k8s), a balance loader, and a DB. Docker & k8s aren’t used because of small scale of the project (but there are plans for migration). Instead, our cloud provider allows to set dynamic servers creation/removal depending on the load using filesystem image. This is fine for now, but there is a lack of logging and monitoring, I tried to structure the requirements:

1. A dashboard with server parameters such as CPU, RAM, disk space, network, etc usage for each server. Some dashboards with such technical information are available by our provider, but a provider-independent solution is desired.
2. A dashboard with numbers of responses by HTTP status code, for each server group in total and each server separately. We use NginX and it provides a status page, but it works on separate servers only.
3. A logs analysing & digging window, with support of multiline messages (like exception tracebacks).
4. Programming language independence. Our server apps write logs to several different files with a date in filename, probably this will have to be somehow changed.
5. Ability to set email (or even launching a shell script?) notifications with custom triggers.
6. Maybe there are some other important features?

Additional wishes:

7. Logging/monitoring setup should be free to use.
8. Simplicity is preferred. I hope a single solution can be flexible enough to cover all the needs of logging & monitoring.
9. Solution maturity / popularity and availability of docs,examples is important.
10. The solution should persist when migration to containers and orchestration will come.

I imagine the final result to be 1) a server that accumulates all the logs, provides dashboards, and logs digging panel, and 2) a script that is installed on each server, is run by SystemD timer, and periodically sends local logs to the server. But how the script will know which exact log files are presented on the current server install, using some config? Though I can be wrong about this algorithm.

I’ve heard about LogStash, ElasticSearch, Grafana, Kibana, Zabbix, Loki, Prometheus, etc, tried to google everything out, but found a total mess of recommendations. I’d to see modern explanations of different technologies, stacks, what software can be used together or is analog of another, and a comparison by the mentioned needs.
I also hope that the answers will be very helpful to many other developers and beginning project leads for the topic is highly important.