Does ISP know I have an onion server if I set up a onion website at home?

It is generally easy for an ISP to tell that you're connecting to the Tor network. Tor onion service traffic looks similar to Tor client proxy traffic, so it shouldn't be immediately obvious to your ISP that you're running an onion service (compared to acting as a Tor client). They may be able to infer it's an onion service based on traffic patterns. For example, web servers typically send much more data than they receive, so if they observe that you're sending large amounts of traffic to the Tor network, they may assume you're running an onion service. Using a bridge may help hide that you're accessing the Tor network.

So to answer your question: yes they can (or they can at least infer it), but it depends on if they're actively looking for people running onion services.