sql, delete duplicate records using an ID

if employeeId 1 for example has 5 files with the same filename then delete the 4 just remain 1 file

One approach uses a delete statement with a self-join:

delete d1
from documents d1
inner join documents d2 
    on d2.id < d1.id 
    and d2.filename = d1.filename
    and d2.employeeid = d1.employeeid

This removes duplicates on (employeeid, filename) while retaining the row with the smallest id.

You should be able to work around the safe mode error in MariaDB by moving one condition to the where clause, eg the filter on id;

delete d1 from documents d1 inner join documents d2 on d2.filename = d1.filename and d2.employeeid = d1.employeeid where d2.id < d1.id

I am not sure if you're trying to select only distinct items or find out those that have duplicates. In case you only want to select distinct / unique values :

 SELECT distinct id, employeeId, filename FROM DOCUMENTS GROUP BY filename

I'd suggest you to move the selected unique records into a temp table and then delete the original table records and then insert the unique records from temp table back to original .