Having the inbox/outbox on my desk is distracting, whether full or empty. I've tried putting it on a side table/cabinet, but since it was out of site it was also out of mind and I never used it.
Yes. It's an important distraction. The inbox is one of the few things that should be distracting, in my opinion, because it represents an open loop that needs to be addressed. Not too distracting, of course; thus the need for helpful placement.