Can I use 'out of this world' negatively in a sentence

Yes, that is proper usage (But it still can be rude to say to someone). You could also say something along the lines of "Their stupidity astounds me".

It can be used in a negative way, but it's unusual. "Out of this world" generally has positive connotations, and that is the meaning most smaller dictionaries give. But it is occasionally used in criticism or to mean very strange without any strong positive or negative connotation. If you use it negatively in some contexts you might be misunderstood as offering praise. But in your example it is clear what you are saying.

Merriam-Webster, Collins COBUILD, Cambridge, and Oxford Lexico all give only positive definitions: "Extremely enjoyable or impressive", "extremely good", etc. The OED as well as a positive definition also allows it to be used negatively or neutrally. It says "P36. b. Frequently with hyphen. In neutral or derogatory contexts: unworldly; quite remarkable; (also) incredibly bad or repulsive. Also in attributive phrases."

One negative example is from the crime writer Andrew Garve in 1951: "They hate our guts, and the way they behave is out of this world." And another from sociologist Peter Wilmott in 1963 is not positive but maybe just implies extreme datedness or oddness: "The L.C.C.'s wallpapers...are very antiquated, out of this world." Another from 1999 is perhaps neutral, meaning incredibly strange (although it might be positive): "The super cinema...was really something out of this world to the people of the period who'd been used to the local cinema or the old converted theatres." So it clearly can be used to mean extreme or weird, without a positive connotation.

• https://www.merriam-webster.com/dictionary/out%20of%20this%20world
• https://www.collinsdictionary.com/dictionary/english/out-of-this-world
• https://dictionary.cambridge.org/dictionary/english/out-of-this-world
• https://www.macmillandictionary.com/dictionary/british/out-of-this-world
• https://www.lexico.com/definition/out_of_this_world
• Oxford English Dictionary, world, n.