What kind of wall material produces orange dust, and how should I work with it?

I live in a townhome that was constructed in 1890 in the US and remodeled sometime in the past 20 years. Most of the walls are drywall, but the walls that separate my home from the units on either side of it are a different material. It’s harder than drywall, and does not produce the same "hollow" sound when you knock on it.

I assumed that it was plaster, and drilled four holes to mount coat hooks. The wall was definitely more difficult and slow to drill into than drywall, but I was able to drill about 1.5 inches through and noticed that the wall was solid material for that entire depth.

Two of the holes produced a beige dust, but the other two holes in the same wall produced a bright orange dust (pictured below). The only material I could think of that might be orange is brick.

1. Is the wall brick, plaster, or some other material?
2. Why did half the holes produce orange dust but not the other half?
3. Do I need to make any special considerations before drilling into these walls or using screws, nails, or other tools or hardware in them?