Is there a job title for someone that sets up a company's development infrastructure?

The role is called DevOps and requires both complete developer background and lots of additional expertise in build tools being used (also scripting, etc). DevOps is not an architect of course but not a junior role either and you should not feel disrespected for getting this assignment.

In the game development industry, I have often heard and used the term Build Engineer or Build Engineering Team to identify this kind of role. In other areas this may not be as common.

who is usually responsible for setting all this up

In a small startup, whoever can, does. There is no special title required.

This goes for any tasks that need doing - from cleaning the kitchen to setting up infrastructure. In a small startup, you usually don't have the luxury of over-specialization.

In one startup where I worked, I was hired to build the QA team. I spent part of my early time there setting up cubicles and running ethernet cables. I also cleaned the kitchen when it was my turn to do so.

If your company is that small I would suggest you don't hire a new employee to fill this role at all. It generally takes a lot less work to build and maintain an infrastructure than it does for developers to utilize it; so, you will just have a guy sitting around on his hands most days costing your company a paycheck (or stealing your job outright).

Instead you want to find an IT consulting firm. These are companies that specialize in doing IT work for small businesses; so, they can afford to hire experienced and specialized labor then spread the cost of those employees out to all the small businesses they do work for. The IT firm (if it is a good one) should actually have multiple specialists. First you will have a Project Manager evaluate your needs, then based on his recommendation you might have a Network Engineer come in and set up your server room, then you might have an IT Technician setup standard systems like backup routines, anti-virus software, virtualization, etc., then you might have a Build Engineer, Backend Engineer, or Full Stack Developer come in and set up your tech stack.

Unlike hiring a single DevOps person for the job who is probably not going to know everything, you are actually hiring a team of very experienced people who together can do the same quality work you would expect out of a very large and well established company's IT team. When you do need support, it typically costs more per hour to use an IT consulting firm than do it in house, but in all likelihood you are probably looking at a job that will take less than 20 hours to set up and just a couple of hours a month to support; so, even if they charge you $100-200 per hour, you are still looking at a smaller cost than a full time guy at $25-50 per hour.

I'm a little reluctant to use "buzz-words" such as DevOps ... which are "right-now in vogue" and therefore sometimes hard to anticipate.

If you're looking for someone to, for example, set up the computer hardware, troubleshoot the networks, take care of software updates, then I typically see terms like "IT Operations Manager." This person's responsibilities are usually fairly broad.

However – in the short run, at a startup, I daresay you need to be prepared to "get your hands dirty" for a little while.

There are levels of specialization here.

A person who is completely dedicated to building and tending a CI pipeline is traditionally titled a build engineer. In most places, however, a more generalized DevOps engineer (what the system administrator job has largely morphed into in 2020) does a mix of setting up the CI/CD pipeline, automating infrastructure, monitoring and incident response, and other things. Usually once the pipeline is set up there's a modest amount of changes many of which are made by the developers, so only large places have an actual dedicated build engineer (or the business equivalent, a release manager).

Of course, since this is a "side gig" in many small/startup organizations, it'll be a developer, or QA engineer, or someone that does the initial setup, just taking on that role for a limited time. It's usually not a lot of work - for mobile apps, search around on what you want to do, and solutions like Ionic Framework that have a custom CI system for mobile apps will show up. In your case any engineer that's done it before should just do it.

You can also call in time-limited consulting help, though then you get to use a pipeline someone else made, which is suboptimal. Only do this if existing staff really just has no idea what to do.

Typically such an environment grows naturally over time instead of one person setting everything up that you are used to from more mature companies. Assuming you know what infrastructure you need, in principle you could look for a system administrator who will take care of setting up server infrastructure (hardware or virtual servers or cloud solutions etc.), install standard tooling and the like. You might need separate roles to evaluate licenses from a business perspective, especially at the start this might fall into the role of existing managers that are responsible for other expenses already.

If you don't know what tooling you need you might need a senior developer or architect too.

Note that especially at a startup, it is often common to take on some tasks outside your core role. It's in the nature of the startup to not have people of every speciality and to get your hands dirty in areas you're not so familiar with. So it might be your boss expects you to take care of setting up at least a minimal tooling infrastructure for the time being. At least don't be surprised if your boss assumes you can do that.

It's called "DevOps". It's a mixture of development and systems admin blended in with security and site reliability. There is an enormous amount of stuff that the title of DevOps Engineer conveys, but setting up initial infrastructure, environments, security and the automation to maintain them primarily falls into that category.