Slaves won't fill moats in stronghold

I’m playing Stronghold Extreme. I built a fortress in the mountains with a narrow way to get down. I dug some moats as a way to keep assassins out, but when I tried to build a gate to pass through I found out that there’s no space left. I intend to fill the moats back in, but my slaves won’t move at all. Now I’m stuck on the mountain. What should I do?

I used the shovel with a red sign on it to fill moats. Should I use another unit?