Wavy Edge on polygon in QGIS using geometry generators

I think I figured it out. In the geometry generator for your polygon layer set the type to Polygon/Multiploygon and use this expression.

make_polygon(smooth(make_zigzagline_rand( boundary( $geometry),1.5,.1,3),6))

In order for this function to work you need to define a custom function called make_zigzagline_rand using this code that I modified from @christoph answer for wavy lines https://gis.stackexchange.com/a/354873/94350 to make the wave offset varied instead of static.

import random
from qgis.core import qgsfunction,QgsExpressionContextUtils,QgsExpression,QgsProject,QgsPoint,QgsGeometry

@qgsfunction(args='auto', group='Custom', usesGeometry=False, referencedColumns=[])
def make_zigzagline_rand(geom,dist,min_offset,max_offset,feature,parent):
"""
<style>
span { color: red }

</style>
<h2>converts a linestring to a zig-zag line</h2>

make_zigzagline(<span>geometry</span>,<span>distance(s)</span>,<span>min_offset</span>,<span>max_offset</span>)<br/>

<table>
    <tr><td><span>geometry</span></td><td>linestring geometry</td></tr>
    <tr><td><span>distance(s)</span></td><td>linear point distances (single number or a string of comma separated numbers)</td></tr>
    <tr><td><span>min_offset</span></td><td>minimun perpendicular offset</td></tr>
    <tr><td><span>max_offset</span></td><td>maximum perpendicular offset</td></tr>
</table>
<br/><br/>
Examples:
<ul>
    <li>make_zigzagline_rand($geometry,'15,30',5,15) -> zig-zag line</li>
    <li>make_zigzagline_rand($geometry,15,5,15) -> zig-zag line</li>
</ul>

Use smooth function to create wavelines:<br/><br/>
Example:
<ul><li>smooth(make_zigzagline_rand($geometry,'15,30',5,15),3)</li></ul>
"""

if not type(dist) is str:
    dist = str(dist)

dist = [float(n) for n in dist.split(',')]
l = geom.length()
dist_sum = 0
distances = []
while dist_sum + round(sum(dist),2) < l:
    for d in dist:
        dist_sum += d
        distances.append(dist_sum)

# interpolate points on linestring
points2d = [(lambda g: (g.x(), g.y()))(geom.interpolate(d).asPoint()) for d in distances]
vertices = geom.asPolyline()
start = (vertices[0].x(),vertices[0].y())
end = (vertices[-1].x(),vertices[-1].y())

points2d.insert(0,start) # prepend start point
points = [QgsPoint(start[0],start[1])]
i = 0
n = 0
b = -90
for point in points2d[1:]:
    pt1 = QgsPoint(points2d[i][0],points2d[i][1])
    pt2 = QgsPoint(point[0],point[1])
    a = pt1.azimuth(pt2) + b
    pt = pt2.project(random.uniform(min_offset,max_offset), a)
    points.append(pt)
    i += 1
    n += 1
    if n == len(dist):
        n = 0
        b = -b

points.append(QgsPoint(end[0],end[1])) # append end point
return QgsGeometry.fromPolyline(points)

This will give you this output. One issue I have found is that if your geometry has interior rings (holes) the expression will not work. Using exterior_ring instead of boundary in the expression will fill in the interior ring instead of making the entire feature disappear.