Because the franchise fell from being a rival to Mario in the market of well-designed mascot platformers to a series that's largely pretty bad and only held up by SEGA being very good at marketing the series to children. Aside from a few anomalies last gen, SEGA stopped giving a shit about Sonic as anything but a guaranteed moneymaker after the Dreamcast died. At this point they don't even bother halfassing the games themselves and outsource them instead.
It's kind of the opposite situation from the Mega Man series, which failed due to blunders in marketing rather than poor game releases outside of the later PS1 X games and two of the Network timeline games.
I haven't been particularly into Sonic for years now. I still love the Genesis/SCD platformers, and have a liking for the Rush games and Colors/Gens, but other than that I don't have a big interest in the series other than hoping that Sonic Team's next time at bat will turn out more like Gens than Colors.
I still love the buttrock the other 3D games produced though.
X is not going to get big numbers. It's a niche game on a failing console, and Nintendo is probably not going to advertise it at all. Nothing is going to save the WiiU, it's a huge mess and not much can be done for it. Some sales will trickle in for the first party games that come out every couple of months, but regardless of quality, that's not enough to turn it around now that the systems that the entire rest of the industry is going to focus on are out now.
I can't think of a good reason for a single player game to have 40 (or more) playable characters. Your idea sounds like one of those overambitious fangames that either never gets finished, or is terrible.
I don't think Eggman is getting stale since he's only been the opposing force for the entire game in two 3D titles so far. To be honest, I'd like more games where Eggman is just the villain, and nothing more complicated than that.