Trying to identify gray crumbly filler material

Almost anything "could be asbestos" and without lab testing you don't know.

That eyeballs to be Homosote®, a soft cellulose sheet product (can be used as an inexpensive bulletin board - tacks go in easily) as opposed to the more common hard asbestos-cement boards, but it "could" be some softer grade of asbestos for insulation purposes. Or it may be harder or softer than it looks in a picture.

What's the history of the house?

Looks like concrete fill (a gypsum fiberglass product). As Ecnerwal stated you can't really be sure without sending it to the lab. Usually affordable ~$50 to test a sample. Bag it, drop it off and wait for the results for piece of mind.

It could be Cellulose* insulation someone packed into the crevices.

*Shredded paper made to be fire proof.