For one without restraint and refinement, sāmya-yoga is hard to achieve. But for one with self-control, it can be attained by proper effort and discrimination.
Krishna is categorical in stating that yoga is not hard at all to practise and perfect. The question is merely of understanding it properly and taking up the right practice, with sufficient discipline and discrimination.
The practice may seem hard for those who lack sensory regulation. Excessive indulgences greatly hinder the pursuit, a point Krishna had emphasized earlier too (6.17). Moderation is the watchword for human life. It is even more so for spiritual life and yoga pursuit.
For the immoderate, says Krishna, yoga is hard to practise and perfect, a point spiritual seekers must keep in mind without fail and act upon earnestly.
For one with his senses and mind under sufficient moderation, sāmya-yoga is quite attainable, with the right understanding. All worldly perceptions are through our five senses. These get reduced further to the mental two, sukha and duḥkha. With evenness towards both, one will have won over the world! (5.19)