In my experience, writing things down reduces much more stress than the extra work my cause. Also, getting into the habit of writing things down, I have noticed that I remember things better even when I don't look at my notes.
I use Thinking Rock to manage my tasks. This has built in feature to mark something as "someday/maybe". If I am at my computer when I think of something, it is just as easy to input it into TR as writing it down. If I do have to write it down, I physically mark a lot of things as NTH ("nice to have") when I write them down. Then can then go into a folder without having to spend any more time with them. Eventually I get back to them, but I have not spent any great amount of time on them.
"Be more concerned with your character than with your reputation. Your character is what you really are while your reputation is merely what others think you are." -- John Wooden