I think for adults point of view they grown up and were no longer the little cute troublemakers. I think that is why my parents didn't really watch it anymore after first or second season.
Also, maybe, for those who watched Season 1, then somehow didn't pick up watching it when the next season started, and then at a later time they tried to watch it again, they would realize that the show is no longer mainly focused on Malcolm, and it didn't really have that breaking the 4th wall anymore. That might have turned off some people from picking it up again. But for those of us that continued with the series throughout, we appreciate the changes happening on the show, and really that is what got us to keep watching the show, every character had their own in-depth story.