Efficiently generating mesh for self-generated voxel grid

I’m working on an app that lets users construct a 3D house, with the ultimate goal of it being 3D printable. I use various materials in the form of a height map constructed from an image, like this brick texture:

I map the brightness of each pixel to a height, like so:

But when you start combining wall panels side by side and at different angles, it can quickly get messy and you can end up with a self-intersecting mesh, not to mention many more polygons than are really necessary, all of which contribute to something which the 3D Printer slicer can’t cope with. It looks like this but the geometry isn’t clean enough to print it:

Now, currently everything in the app itself works directly with polygons. Since all of this ultimately is generated from grid-like data (the source images) it seems to me that some kind of voxel-based approach would be very suitable, and could potentially yield much better quality results. I have tested the approach of populating a grid directly from the source images, and that bit can be done very efficiently. But I’m having a hard time finding the right algorithm to then turn that grid data back into polygons ready for 3D printing.

Most algorithms for turning voxel data into a polygon isosurface assume two things, as far as I can tell:

1. That you have no prior knowledge of the underlying geometry
2. That the geometry is a scalar field, rather than a "boolean" grid where the grid lines genuinely do correspond to the final geometry. There could be fine scaled details that fall between the grid lines.

Whereas in my case:

1. I have huge empty patches inside the building, but I can easily ignore those because I have a list of bounding boxes of where each wall panel exists
2. I would like to take advantage of the fact that my data actually is gridlike in nature. Note that I can have ‘diagonals’ – this simple brick texture shown here is probably a little too simplistic, e.g. some roof tile textures use a lot more in-between heights:

Obviously something as simple as marching cubes could be adapted to take advantage of point 1 easily enough (just skip over sections of the grid which aren’t contained within any of my bounding boxes) but I feel like marching cubes will yield WAY more polygons than necessary.

Can anybody recommend any specific algorithms that might suit this scenario (where we have prior knowledge about the underlying geometry) or give me some search terms that I can use to research? I’ve been searching algorithms for generating isosurfaces, but my instinct tells me that what I’m working with here is actually subtly different because of the grid-like nature of the problem – e.g. are these even really "voxels" at all?