Am I right to ignore him after he swears at me?

Open your eyes, he is never going to change, and in the long run, you may end up dead with his violent tenancies.