Somewhere in the 80s, RPGs changed from fight the "dungeon" to fight the "DM." The adversarial nature of gaming became more and more apparent. Now, I missed this era because my group didn't play this way, but when I started working in a game store, I met lots of groups that played this way. They took on the anti-DM approach to gaming and from this thinking sprouted an entire subculture of RPGers (not unlike Knights of the Dinner Table and so on).
As gaming grew from books to video games, those seeking a more DIRECT approach to fighting monsters left the game table and those who needed the socialization of the game table remained in the RPG arena. I don't know the exact connection, but many gamers became more and more interested in the MINUTIA of the game. FASA was particularly gifted at making games filled with page after page of data so thick, only the most afflicted asperger sufferers could master the Street Samurai combo for 15 actions per round.
Through the 90s, RPGs suffered, as fast, easy-to-learn CCGs took center stage of the gaming industry. Those who tired of the 4-hour character creation process and the endless number-crunching of more obvious power-game RPGs (cf. Rifts, Earthdawn) turned their attentions to games they could play quickly, still capturing the tone and theme of the fantasy/sci-fi/horror/nerd gaming industry.
Soon, however, the bottom fell out as "casual gamers" were pushed away from play. Number-crunchers and college kids with too much time on their hands created unbeatable decks, exploiting the otherwise fun and loose games at their disposal.
As I see it, casual gaming lost a lot of ground in the 90s and it wouldn't be until the German Board Game wave hit, that casual gamers could collide again with competitive play. But in the mean time, D&D was hemorrhaging. Years ofSCA-focused 2nd edition releases had pushed fans of the game away from the center. Crappy rules made crappier by kit books only weakened any hold the company might have had on perennial gamers.
Then 3.0 hit and we all know what that did.
Theories abound, but an uncontrolled/unpatroled, free-license is among the highest reasons the industry was glutted. And 3.5 did a lot of damage in SPLITTING the fan-base in THREE. With board game sales climbing and rpgs on the way out, there was only one place left for RPGs... its eventual return to a hobby-form.
Pricing aside, with every recognizable genre ground into pounder by every 2 cent a word kid looking to get his name in a game book, the glut had spilled out of d20 into every corner of the RPG market.
The players were done.
The only route left, as I see it, was smaller, faster, simpler games with genres, topics, and themes people had done poorly before or had never touched before. Dogs in the Vineyard, Burning Wheel, and Primetime Adventures certainly stand at the top of the heap of games that not only did things well, but opened doors of "gaming communication."
I don't have a finale here. I wish I did. What started as an attempt to reconcile (for myself) how the industry got HERE, became a history lesson.
For me, the future of my RPG enjoyment is games like Warhammer Fantasy, Dogs, my own personal designs, and some new Indie product (like Agon) that promises fun with less prep time. My buddy Josh is working on something that COULD be pretty bad ass... but right now the game takes a little too long to explain. I'm hoping he/we can streamline it over the next few months... it could be as cool as Dogs.
Gaming should be whatever you need it to be. It's a hobby, like model trains. Attempts to make it an industry always fail and always leave people sad... "What happened to Game XYZ? I loved that." Because after all, it's not that XYZ isn't profitable... hell in the age of PODs, selling 10 copies is profitable... it's a matter of expectations of profit. And people are curious how to make millions doing this (not pennies)... and I think those days are over. There's no quick-start route to success anymore, because so many people are looking in so many directions for the perfect combinations, the true formula has been lost to postmodernity...
make a good product.
Let some bean-counter worry about demographics and thresholds and trend patterns and every other nonsensical marketing paradigm jargon and leave good game design to good gamers.