R {spdep} Multiple y observations per region (pooled cross-sectional data) - How to construct weights matrix?

I’m in a situation where I have a shapefile of N districts (provinces and sub province level) across the globe. I also have a dataset of S survey responses where each individual is tied to a district. I want to account for spatial autocorrelation in a regression but multiple individuals are tied to the same district. For instance, I might have Colorado in the US with 45 respondents.

I want to construct a weights matrix using spdep. I can do this easily with the shapefile data however that matrix is going to only be NxN. The regression functions expect a DxD matrix though (or D long list). I could hard code such a matrix/list from the NxN matrix however I’m wondering if there’s a way I could do this in spdep instead, where each individual in Colorado gets the same row from the NxN matrix automatically.

I assume that assigning a polygon (sf or sp packages) to each Survey respondent is not the smart thing to do?