I have a slightly different way of looking at it:
My Downloads folder is, essentially, my inbox. I process things out of that into their final homes. Those homes can be anywhere on my computer; I have one folder at the root level of my hard drive for music, for example.
Thanks to Spotlight, I don't care much where these files exist; I can search for them and find them quickly.