Why is it considered bad practice to use the 'new' keyword in Arduino?

I'm adding this not so much to add to the answer as to add some real world implications for those that may be down this particular rabbit hole. It's nice to talk about what could happen theoretically, but a new programmer may still be tempted to think that he can out-think these limitation and still do something useful. Here is some real world talk about why that is foolish even if you are capable.

Let's say we're developing code on an Arduino UNO. We've got 2K of RAM to work with. We have a class that loads a list of names, maybe it's a building access device or something. Anyways, this hypothetical class has a name field to store someone's name. And we decide to use the String class to hold the name as a String.

Let's say after our program is all there and doing what it's doing that there is 500 bytes left for this list of objects, each with a name field that could be of varying length. So we run along nicely for years with a crew of 14 or 15 people with an average name length of 30 characters or so.

But one day a new guy signs up. And his name is really long. Let's say it takes 100 characters. Even if we're smart coders and only have one copy of this String in memory, we've got it there and suddenly it doesn't fit. Now a program that has worked for years suddenly fails. And nobody knows why.

So the solution is easy right? Enforce a maximum limit on the length of the name. Some simple code that checks the name length and we can write a great piece that still allows the variable name length and won't let you create a new user if there's less than that much left. Seems simple enough, but then Becky in accounting gets married and her last name changes from Smith to Wolfeschlegelsteinhausenbergerdorff and suddenly our program breaks again for no reason.

So the solution is simple right? We'll enforce a maximum length and we'll be sure to reserve enough memory for each object that it can afford to have the maximum length name. And that's something that we do most efficiently without dynamic allocation since we already know the size of the object.

I hear what you're saying, "but Delta-G, if we have all those short names in there, why are we going to waste all that memory on them when we don't have to. Let's only save space for a long one if we have a long one." And I like your thinking, this is good thinking. But it doesn't help anything to save that memory. If you do save some, what are you going to do with it? If your program uses it, then there is no longer room for the longer case and suddenly you have to enforce an even shorter maximum length to accommodate that usage.

Say for instance we have 500 bytes of room and we enforce a maximum length of 50 bytes for 10 users. And let's say that when the names are short we want to let the program use some of that saved space. If the program can encroach 100 bytes into that space when the names are short, then why wouldn't the same situation happen with long names? So really since the program can use all but 400 bytes then there's really only 400 bytes of room and we have to enforce a 40 byte maximum for 10 users or 50 bytes for 8.

Since you're going to have to make that sacrifice anyway, then it only makes sense to remove the work of dynamic allocation and just make things fixed sizes or use fixed size buffers.

If we had a PC with gigabytes of memory, we wouldn't even be thinking about this. But on an Arduino UNO with 2K bytes of memory it can become a big problem really fast.

The other problem is that these bugs are so insidious. If out of memory bugs just caused a simple crash and the thing no longer worked then it wouldn't be nearly so scary. But that's not how out of memory bugs work on a microcontroller. It all depends on how things are arranged in memory by the compiler.

These bugs often manifest as something that seems to work most of the time but has some funny bugs nobody can explain. Maybe there's only a problem if someone has a name that is exactly 26 characters long and tries to open the door on a Wednesday. Or maybe the issue will only arise if Becky logs in immediately after Bob. Maybe it just garbles three letters on the screen but other than that everything works. Maybe with a different name changed that suddenly turns into our lock opens for anyone. There's no guessing or explaining out of memory bugs. So we have to be super careful to avoid even the remote possibility of running into one.

And that's why we avoid using dynamic allocation on small microcontrollers. At the end of the day, there's nothing you can save with it, and even if you could the consequences of getting something a little wrong are terribly frustrating. With these types of programs you almost always have to end up enforcing some sort of limit on everything and once you're enforcing limits there's no use for dynamic allocation anymore.

For starters, fix your library

As noted by @crasic, dynamic memory allocation is generally not recommended for embedded systems. It may be acceptable for embedded devices which have a larger amount of free memory - embedded Linux is commonly used, for example, and all Linux apps/services will tend to use dynamic memory allocation - but on small devices such as an Arduino there simply is no guarantee that this will work.

Your library illustrates one common reason why this is a problem. Your enqueue() function creates a new QueueItem() but does not check that the allocation succeeded. The result of failed allocation may either be a C++ bad_alloc exception, or it may be returning a null pointer, which when you reference it will give a system memory access exception (SIGSEGV signal in Linux, for example). It is nearly universal in Linux and Windows programming to ignore memory allocation failure (as encouraged by most textbooks), because the massive amount of free RAM and the existence of virtual memory makes this very unlikely, but this is unacceptable in embedded programming.

More generally though, as @crasic says, memory fragmentation can leave even non-buggy code unable to allocate memory. The result will be a failure to allocate memory, but the code will at least know this has happened and will probably be able to continue.

But better, use a fixed-size FIFO queue instead

Your code relies on dynamic allocation to add and remove elements in a queue. It is perfectly possible (and equally easy coding-wise) to create a fixed-size array for the queue, so the various failure modes of dynamic allocation simply do not apply. An item to be queued is simply copied into the next free queue slot, and a queue slot is marked free when it has been used. (Don't forget to use a mutex when adding and removing items from the queue, because adding and removing will often be called from different places.)

The queue can be made whatever size you feel is appropriate (allowing for how much RAM you have). With a fixed size, you are forced to make a design decision on what should happen if the queue overflows - do you delete the oldest data to make room for the new value, or do you ignore the new value? This may seem an unwelcome new feature, but it is a good thing, because the third option which you've currently got is that your code goes "Aaaarrggghhh I don't know what to do!" and crashes fatally, and we don't really want that.

Dynamic allocation is generally discouraged in embedded applications because you cannot guarantee that you do not exceed (attempt to allocate more than) the available memory. Static allocation will generally have this guarantee although out-of-memory bugs may still be possible.

Additionally, far fewer services or tools are available to automatically manage and mind the memory for you. Any service that does so will consume computational resources.

This means that you inherently create a mechanism in your device that would cause a memory (heap) overflow and possible undefined behavior (UB). This is true even if your code is bug-free and has no memory leaks.

In non-critical, exploration, learning, and prototype applications this may not be important.

Consider that without careful consideration undefined behavior can result in hardware failures and unsafe performance, for example if the device reconfigures GPIO through an errant write to the correct registers during a crash.

Most Arduinos (like the Uno or Nano) have very few RAM, thus you first need to make sure that you never allocate too much memory. Dynamically allocating memory can also lead to heap fragmentation (heap being the part of memory where dynamic allocation happens).

In most cases you would want to allocate memory of different sizes (for example arrays of different sizes) or just different objects (with each having it's own size) (!!! This is the key point here). Then you are going to delete some of these objects which will create holes inside the memory. They can be filled again with objects with the same or less size. As time passes and more allocation and deleting happens these holes tend to get smaller, up to the point where none of your new to allocate objects can fit in there. That memory then is unusable. This phenomenon is called heap fragmentation.

These holes appear naturally also on a PC but there are 2 key differences:

1. The Arduino has such little RAM that the holes can fill up your memory very very fast.

2. While the PC has an operating system, which manages the RAM (defragmenting it or putting unused stuff away into a paging/swap file), the Arduino does not have an OS. So noone keeps an eye on the real available RAM and noone tidies up the memory once in a while.

That does not mean that you cannot use dynamic allocation on an Arduino, but that is very risky depending on what exactly you are doing and how long the program should work without failing.

Considering this big caveat, you are very limited on how to use dynamic allocation. Doing it too much will result in very unstable code. The remaining possibilities, where it might be safe to use it, can also be easily done with static allocation. For example take your queue, which is basically a linked list. Where is the problem with allocating an array of QueueItems at the start. Each item gets a way to determine if it is valid. When creating a new item you just pick the first element in the array, which has a non-valid item, and set it to the desired value. You still can use the data via the pointers, just as before, but now you have it with static allocation.

You might find that the code looks uglier that way but you need to adapt to the platform that you use.

Note that this does not apply when you are going to create only objects with the same size. Any deleted object will leave a hole where any new object can fit into. The compiler uses that fact. So in that case you are safe. Just every object that you dynamically create in your program needs to be the exact same size. That of course also includes objects that are created inside different libraries or classes. For this reason it can still be a bad design choice, as you or others (if you want to publish your code), may want to pair your library with other code.

Another way to be safe is to only create and delete objects in closed cycles, meaning, that a created object needs to be deleted before the next object is created. Though that is not fitting for your application.

'Bigger' microcontrollers, for example the non-Arduino boards with the ESP32, have much more memory. Thus the use of dynamic allocation is not that bad on them. Though you still don't have an OS to manage the RAM.