I'm going to think/type out loud here for a bit so bear with me as I ramble on...
You've nailed the 3 main classes of info and also that it's the middle class that is giving me fits.
I must be able to look up the same information within 30 minutes for any animal we have ever had in our flock as well as the status, died, butchered or sold and where sold too. (This is a requirement of a federal program we are in BTW) This also has to be portable or at least with me all the time as I usually need this data when I am not at home but sitting in some government office answering questions or filling out forms.
I should be able to answer questions like how much hay did we use during X time frame and what seeds got planted in field Y and what date did we start irrigating 3 years ago. Such questions should take no more than a half an hour to locate and collate the data. Much shorter if it's just a case of finding the date something happened but longer if it's a calculation such as how has the death loss changed over the last 10 years or how has the inbreeding coefficient of the flock as a whole changed since we imported semen from the UK. These information requests come from all places, government entities, organizations and groups needing info on how policies, weather or other factors are affecting agriculture, my own curiosity and wanting to compare our progress with others. I never know in advance what the question de jour will be. Once I find the answer, particularly if it's a complex calculation one, I need to archive that answer so that in a few years when the same question comes up again I can just update with the latest data. I'm finding that a lot of these things do show up again within a decade. I wasn't keeping my calculations originally but now I am.
This is the long term reference information and my system is getting tweaked with better filing categories but in general I can do this now with what I have and use regularly. Some of the times take longer but as I improve my archive of previous questions answered I am reducing the time it takes to answer the difficult calculation questions.
I'll skip ahead to the
In an ideal system I review on some appropriate schedule everything. That review may cause some things to be triggered into active work but I also need automatic triggers to bring items to my attention when external conditions are correct even if I haven't planned on doing that item then. I need to document the detailed reasons for the project, historical actions and future plans as well as concrete next actions and the conditions that are necessary for those actions. All that data should be readily available and easy to use. I don't need it to be portable in my Treo until it becomes an active project again but once it is active I do need at least summary data and next actions to be incorporated into my current GTD system that I carry with me at all times.
I must be able to pick up something I haven't looked at or worked on for several years and within an hour or less be up to speed on where I was when I stopped, what the goal was, what conditions I was waiting for and be ready to move forward. Ideally that get back up to speed time would be much less but realistically I may have to read a lot of back notes to remember what was going on. I also need some sort of master list of all these things so that if I vanished or was incapacitated others could pick up from where I left off with minimal hassle.
Right now I can and am slowly building files both computer and paper, that hold the details of the why, the history, the future plans and the next actions for all these types of long term projects. It's a start but what's missing are:
1) An easy way to bring it forward or active when I review it and decide it's time to work on it
2) Integration with existing tickler and next action system when it does become active
3) No automatic triggers to start action when conditions are right
4) No good emergency backup list of all projects that someone could use if I wasn't available.
The part of the scientific notebook that is missing from the calendar or scrapbooks is the thoughts on where to go next. The now that that experiment is done what's the next step to take. That should become part of the files for the long term projects I would think.
Long and rambling but it's helping me clarify my needs which will I hope eventually result in a good working solution.