If you notice that your follower count is dropping (in some cases by the hundreds) do not fear. Twitter is flushing out spammers and sending them in to cyber oblivion.
The follower count has been inconsistent on Twitter for a while now and it has been an issue that they definitely needed to address. So don’t take it personally. It’s not that you have scared people away. It’s a good thing that the followers you have are real people. 
Don’t get so hung up on the follower number anyway.