In my experience, writing things down reduces much more stress than the extra work my cause. Also, getting into the habit of writing things down, I have noticed that I remember things better even when I don't look at my notes.

I use Thinking Rock to manage my tasks. This has built in feature to mark something as "someday/maybe". If I am at my computer when I think of something, it is just as easy to input it into TR as writing it down. If I do have to write it down, I physically mark a lot of things as NTH ("nice to have") when I write them down. Then can then go into a folder without having to spend any more time with them. Eventually I get back to them, but I have not spent any great amount of time on them.