You hit one of the points: app bloat, because everybody and everything demands a separate app. I'm looking forward to being able to set up folders on my screen to try to organize the mess, and I'm far less afflicted with it than many people.
A second thing is that nearly every app requires to you set up an online account with somebody, whether you need it (because of a need to control something when not there) or not. If the app were truly a remote control, communicating only with the device(s) being controlled, I'd be fine with the transition. But having everything go over the Internet is a deal breaker. Too much data to be mined, personal info thrown into the wind, and more avenues for malware to infect my home network. I'm willing to be pleasantly surprised, though: recently, I discovered that a "smart plug" device somebody game me actually could be controlled entirely locally with an app, without generating an online account with the vendor. Great!