Yeah, Seagal was good early on.  Norris had that “avg American Joe” quality and also good fighting skills.  Van Damme was the “everyday hero” too and was in great shape.  Both were action stars AND martial artists. 

My guess at why Seagal took a dive in popularity had to do with him not staying in shape.  He went from martial artist to action star only.  It limited his roles.  He could only do cop/crime movies and started teaming up with rappers to stay relevant.  Some of those early 2000s movies were good but not much fighting.  Just guns/cops.  That limits your audience, so he started getting less movie offers. 

Norris and Van Damme were always making B movies but they were always decent and still hit theaters.  Seagal started making C movies.  Like direct-to-video quality.  He just needed $.  No real love for telling a good story.