I have this problem occasionally and it was actually on Dr. Oz not too long ago, so a lot of people more than who are willing to admit, have this issue, so we are not alone.
I used to have this problem a LOT. Now I wash my belly button with the orange Dial soap and make sure to get it good and dry after I shower and it hasn't been an issue in a long time .