Train 3D object detection model for custom object

I am trying to train a model that can detect a 3D object and give me a 3D bounding box around it. For this, I have a RGBD camera and a 2D LiDAR.

Most of the research is done for cars/cyclists/pedestrians and I ponder how to create a model for a custom object (a toy for example).

If anyone has experience with this, I need a helping hand.