Refactoring is something that should be constantly done in a code base, for every story. As soon as people get scared about changing things the codebase is on the road to being legacy.