Is this mold in our windows?

The answer is - it doesn't matter.

Whether the debris is grime and dirt or mold...

1. It won't "hurt" you as there is a glass barrier in between.
2. There isn't a different way to handle it.
3. As a typical home owner without industrial tools, you really have no way of resealing your windows.

So basically a seal has failed on your window. You can either replace that part of the window (frame and all). For that part you will need to know manufacturing details to assure a proper guide fit. More than likely only that manufacture (not all cases) can reproduce.

The other option is take that part of the window out, and either take it to a window or glass shop. They can take window apart and reseal it. Just FWIW I usually tend to order a replacement as I have found that when getting a window resealed for some reason I don't see the quality compared to the window going through the normal "manufacturing" process. Usually the costs are close and obviously this is locality and opinion based. If there was a glass shop down the street that would offer a 5 year warranty I would go with that - mine won't.