I remember the rule as if someone is passing you slow down a bit if it doesn't impede traffic or otherwise cause issues-basically, be polite if possible.