I know the easy answer to this one is FOX's carelessness towards MITM in general(With football/time changes/bad promotion), but would anyone be willing to offer some different theories?
I really think that is the number 1 reason. Also the fact that it was even on Sunday with Fox's animated line up in the first place really hurt it. That only added to the (false) perception it was a kids show. However popular Simpsons is, and however long it has been on, most people over 30 still simply are never tuning into Fox's Sunday lineup. Lots of times it was up against 60 minutes, one of the top rated news shows, or against sports. Malcolm should have been on a weeknight for seven years like other sitcoms that adults watch.
I think it is quite possible for some people (and kids themselves) to watch Malcolm and after an episode or 2 conclude it was a kids show. They may have only watched because they like the overt funny things that happen, as if Malcolm was like the Threee Stooges (bee shooting robots, giant sling shots, Malcolm and Reese fighting ect). On top of the wackiness and physical comedy that is first apparent, there is a whole other layer of intellectual comedy that goes right over many people's head's. I think what happened is over time those who just liked the superficial elements, that there were funny kids ect... well those fans got bored and didnt watch after a couple seasons. In fact the kids grew up so if that was the main reason you are a fan you begin to say things like the later seasons were not as good...
If you don't see the intellectual comedy, psychology ect.. then you will lose interest over time. Teenagers being obnoxious is naturally not as cute as Malcolm and Reese rolling on the floor in the pilot. The fans who only liked the zany antics of some kids in the first seasons would not even really comprehend the premise of episodes like Billboard, Zoo, Pearl Harbor or Hal Greaves. I mean the season 2 episode Bowling won an emmy. Do you think anyone under about 16 even understands what is going on in that episode? I doubt it. I almost think you really have to have taken at least psych 101 to really understand some of the Malcolm episodes.
Basically, Malcolm in the Middle got screwed by being grouped with or as a kid or kid friendly show from the begining, when actually much of the jokes kids wouldn't even understand. These fans jumped off the Malcolm train as the family became less cute and more grown up and they never realized there is a whole other layer to the show. Even after seven years on the air there are lots of people who thought it was just a silly show about kids and never tuned in.