Are there drawbacks to using solar panels as a roof without asphalt shingles underneath?

You need a WRB on that roof somewhere

One thing all roofs need is some way to shed water. Asphalt shingle roofs use the shingles themselves as the primary water-resistive barrier (WRB), often backed up by a secondary barrier (roofing felt). Other roof technologies, such as metal roofs, have their WRB underneath the roof cladding instead. However, this depends on the roof cladding being a full-coverage sort of thing, which isn't true for solar panels as panels need air circulation for cooling and wiring space behind them for connections and equipment.

You can get solar roof products that serve as a roof cladding, but they still require a WRB underneath them, and such building-integrated photovoltaic systems require more care in wiring as well due to issues with the potential for wiring damage from roof incidents. They're also harder to replace than a conventional panel if they do get damaged, say by hail, and pose challenges under the current NEC rapid shutdown rules.