Extending (buffering) set of polygons into empty space only using Shapely?

I have a set of polygons of which I would like to extend the boundaries a bit into empty space, using Shapely. They are not allowed to overlap each other at all. For example creating buffers and then removing the overlapping areas is not an option, since this will remove a part of the original areas.

The following image (partially) indicates how I would like the result to be like:

I have retrieved the data through a WFS service, and parsed it into Shapely polygons. I’ve also created buffers. Like so, in a loop:

# in this case, the feature's geometry only has an exterior boundary
if len(bag_polygon) == 1:   
    bag_polygons.append(Polygon(bag_polygon[0]))
    bag_polygons_buffered.append(Polygon(bag_polygon[0]).buffer(1, cap_style=3, join_style=2))

# in this case, it has interior boundaries as well
elif len(bag_polygon) > 1:
    bag_polygons.append(Polygon(bag_polygon[0], bag_polygon[1:]))
    bag_polygons_buffered.append(Polygon(bag_polygon[0], bag_polygon[1:]).buffer(1, cap_style=3, join_style=2))

Putting them in a .shp with Fiona, I have added a vital attribute list to the geometries.

shp_schema = {
    'geometry': 'Polygon',
    'properties': {'bag_id': 'int'},
}

with fiona.open("bag_subset.shp", "w", crs=from_epsg(28992), driver="ESRI Shapefile", schema=shp_schema) as bag_shp:
    for i in range(len(bag_polygons)):
        bag_shp.write({
                "geometry":mapping(bag_polygons[i]),
                "properties": {"bag_id":bag_ids[i]}
                })

with fiona.open("bag_buffered_subset.shp", "w", crs=from_epsg(28992), driver="ESRI Shapefile", schema=shp_schema) as bag_shp:
    for i in range(len(bag_polygons)):
        bag_shp.write({
                "geometry":mapping(bag_polygons_buffered[i]),
                "properties": {"bag_id":bag_ids[i]}
                })

Lastly, I’ve thought about taking the difference between the shapefile that includes the buffers and the shapefile with the original geometries, to subsequently join this difference with the original geometries. However, there is still the problem that the buffer zones overlap themselves.