Plot a discrete raster over a hillshade using ggplot2

I’m having a hard time fixing a somwhat simple ggplot. I have a raster of a hillshade and I would like to overlay it with a classified prediction raster with the viridis color scheme.
My code at the moment looks like this:

  # hillshade dataframe
  df.hsd = rasterToPoints(cropped_hillshades[[i]]) %>% as.data.frame(.)
  b.hs = seq(min(df.hsd[[3]]),max(df.hsd[[3]]),length.out=100)
  
  plot = ggplot(data=df.hsd, aes(x=x, y=y)) +
    geom_raster(aes(fill=df.hsd[[3]])) +
    scale_fill_gradientn(colours = grey(1:100/100), breaks=b.hs, guide="none") +
    geom_raster(data=df.pred, aes(x=x, y=y, fill=.data[["class"]]), alpha=.8) +
    scale_fill_viridis_c(name="Class") +
    coord_equal() +
    theme_minimal()

When I plot only the hillshade I get somehting like this:

When I only plot the susceptibility raster I get:

Both kind of look ok, but when I execute the lines above I get:
Scale for 'fill' is already present. Adding another scale for 'fill', which will replace the existing scale.

And that’s not really what I want as the Hillshade isn’t visible anymore… I’m really done with trial and error in ggplot and would be grateful for any help.