It sounds like you’ve added both the light and the button to the devices in the scene, but only the light should be added.
Scenes set controllable devices to specific states and a button is generally not a controllable device (it’s the device that does the controlling). Scenes are also not directly activated and need an automation to activate them.
You’d need to create an automation and specify that ‘when’ the button is pressed, ‘then do’ activate the scene.
Happy tinkering!
I found when changing the Zigbee channel with Z2M that I needed to reflash the firmware on the Sonoff Zigbee 3.0 P dongle to erase its list of device addresses, then delete the coordinator_backup.json so that the list of device addresses is not reimported, and then start pairing my devices from scratch.
Z2M was automatically selecting channel 11 which in my case is the same as Wi-Fi access point nearby which was causing interference leading to significant delays or losses in signals (as well as Wi-Fi performance issues on the 2.4GHz SSID), which seems similar to your issue
You are very correct that it is user-unfriendly, but I haven’t tried with Hue products (I find that IKEA Tradfri bulbs work well with Z2M though)
Glad I could help!