Hi Tomas,
at least I can say: yes, you are not alone.
I also had this issue with my older PC...very annoying!
These days I set up a new PC and at least this (small) issue is gone for me...
The strange thing is that my audio interface is still the same as before, just the machine has changed but anyhow it seems to work fine now.
There are soo many bugs in PD, it can really drive you crazy...!
Best wishes,
Rob