WHY DO I GET THE “CRUD” WHEN I WORK OUT?

I have an anomaly I would appreciate feedback on. I have noticed a pattern in my life. As I get older and more sedentary I try to muster up the enthusiasm to work out  and get back in to good physical shape that I had during my youth; I am middle age now. The pattern I keep seeing is this: When I finally get motivated and go work out on a consistent basis and push myself a little, I start to notice a slight improvement in my shape and feel of my body. When I get serious about it, I can see instant results. The problem I have is that as soon as I see and feel ANY results, I get a sore throat and feel run down and get very sick with a respiratory infection of some kind. I can be sick in bed for a week or two with a hacking cough for months afterwards. This stops all of the progress I had accumulated (which was only a small beginning). I am sick as I write this. This scenario keeps playing out in my life over and over and over and I can’t get back in shape. If I don’t exercise outside of my somewhat physical job, I don’t get the “crud”. I can be exposed to very sick people and my own family coughing and hacking on me and I don’t get sick. It is only when I work out and push myself to get in shape that I get the “crud”. What is happening? Should I go to a nutritionist? My doctors have no advice, and I don’t feel like they listen to me. Does anyone else ever notice this health phenomenon or is it just me? It seems to pop up like when one gets a fever blister on their lips when they get a fever or a cut on the lips. Could it be some kind of unknown viral thing that blooms when I get my body hot and sweaty? I don’t think I start working out that hard. I have the will power to get in shape but am very frustrated that I always get the “crud” when I try. The “crud” stops me. I have a job to keep with a very strict attendance policy and I can’t be off sick. I will be traveling to southern California this summer and would like to swim with the dolphins and fit into one of those wet suits and swim in the ocean without looking and feeling like a blob. When I was more youthful, working out keeps my appetite at bay and I have a desire to eat right. Any suggestions out there? Please no rude comments.