Calculating bounding box with padding

You can achieve this with the following steps as a kind of algorithmic description (pseudo code) that you can implement whatever software you use. For an actual implementation, I will add a solution how to do it in QGIS using expressions. Also see the screenshot at the very bottom:

1. Measure the distance AB
2. Measure the distance AD
3. Get the higher value of (AB, AD) to get the longest side of the original bounding box
4. Multiply the result from 3 with percentage value P to get the padding distance
5. Create a line from A to B
6. Extend the line from 5 at both ends with the distance from 4 (red line from point 1 to point 2 on screenshot)
7. Get the start and end point of the extended line from 6 (screenshot: points 1 and 2)
8. Get the x and y coordinate for the points from 7
9. Repeat steps 5 to 8 for a line from A to D (screenshot: points 3 and 4, connecting red line)
10. Get the 4 extreme coordinate values (x max, x min, y max, y min) from 8 and 9
11. Create 4 points with the values from 10: they correspond to A', B', C', D' (in fact, 3 points are enough to create a rectangle in the next step: A'=x min/y max ; B'= x max/y max ; C'= x max/y min)
12. Create a rectangle with the points from 11: this is your new, padded bounding box (red rectangle on screenshot)

To implement this in QGIS: Let's presume you have a polygon layer rectangle and a point layer vert for the vertices of the rectangle: A, B, C, D, whereas these four letters are stored in an attribute field name. You can use QGIS expressions either for visualization purpose only as an additional symbol layer or create actual geometries using Menu Processing / Toolbox / Geometry by expression. This is a step by step guide to show how the expression is built according to the steps above for better understanding. However, you can directly jump to 10 to use the expression there as it includes all the steps from 1 to 10.

1. distance (geometry (get_feature('vert','name','A')), geometry (get_feature('vert','name', 'B')))
2. distance (geometry (get_feature('vert','name','A')), geometry (get_feature('vert','name', 'D')))
3. max (a, b), whereas a is the value (or expression) from step 1, b the one from 2
4. c*p, whereas c is the value or expression form 3, p is the percentage for the padding; implementing the whole expression from 1 to 4 with percentage=20% would look like (change 0.2 accordingly for other values):

max (
        distance (geometry (get_feature('vert','name','A')), geometry (get_feature('vert','name', 'B'))),
        distance (geometry (get_feature('vert','name','A')), geometry (get_feature('vert','name', 'D')))
    ) *0.2

5. make_line (geometry (get_feature('vert','name','A')),geometry (get_feature('vert','name','B')))
6. extend (geometry, d, e), whereas geometry is the expression from 5 and d as well as e is the value or expression from 4 (screenshot: red line connecting points 1 and 2)
7. start_point (geom) and end_point(geom), whereas geom is in both cases the line or expression from 6 (screenshot: points 1 and 2)
8. x(point) and y(point), whereas point is the point/expression from 7
9. repeat 5 to 8, replacing B in step 5 with D, everything else remains the same; to get all four points at once, you can create a multipoint using collect_geometries(1,2,3,4), whereas 1,2,3 and 4 are the start- and end-points (step 7/8) of the two lines (step 6/8) All steps from 1 to 9 together looks like this. To keep it straightforward, here I would create a new layer for these four points (1,2,3 and 4 on screenshot), using Menu Processing / Toolbox / Geometry by expression, than set the rectangle layer as input layer, and point as output geometry type and than paste the following expression (here I used with_variable() to define once a variable padding for the padding distance and insert it as @padding afterwards to avoid repeating expressions). After that, use menu Vector / Geometry Tools / Multipart to singleparts to get a separate feature for each point. In the Geometry by expression dialog, use this expression:

with_variable (
    'padding',
    max ( 
        distance (
            geometry (get_feature('vert','name','A')), 
            geometry (get_feature('vert','name', 'B'))),
        distance (
            geometry (get_feature('vert','name','A')), 
            geometry (get_feature('vert','name', 'D')))
    ) *0.2,
    collect_geometries (
        start_point(extend (
            make_line (
                geometry (get_feature('vert','name','A')),
                geometry (get_feature('vert','name','B'))),
            @padding,
            @padding)
        ),
        end_point(extend (
            make_line (
                geometry (get_feature('vert','name','A')),
                geometry (get_feature('vert','name','B'))),
            @padding,
            @padding)
        ),
        start_point(extend (
            make_line (
                geometry (get_feature('vert','name','A')),
                geometry (get_feature('vert','name','D'))),
            @padding,
            @padding)
        ),
        end_point(extend (
            make_line (
                geometry (get_feature('vert','name','A')),
                geometry (get_feature('vert','name','D'))),
            @padding,
            @padding
        )
        )
    )
)

10. Now you should have a layer with four points (red points 1,2,3, and 4 on the screenshot). Get the max and min coordinates for x and y using this expressions: For x: maximum(x($geometry)) and minimum(x($geometry)) For y: maximum(y($geometry)) and minimum(y($geometry))

11. and 12. can be done in one step. Use this expression:

    make_rectangle_3points (
        make_point (minimum(x($geometry)), maximum(y($geometry))),
        make_point (maximum(x($geometry)), maximum(y($geometry))),
        make_point (maximum(x($geometry)), minimum(y($geometry)))
    )

And here you are, you get the red rectangle: