

No, it is when not charging you need to worry. When charging odds are you are there and so if it starts to burn you will smell it and take action before it starts a larger fire. When you are not charging you might not even be home and thus the fire will spread.
When things are normal the charger will not burn. When something goes wrong you have to worry. Most of the time you are not charging.
Unplugged vs plugged in is moving the goal posts. I agree that a device that isn’t plugged in is less likely to start a fire (while not impossible, it is very very unlikely), but that is a different situation.