My hardest battle was the first one, I did defeat Cress in my first battle with him but he took out almost all my Pokemon. Not wanting to win like that I reset the game and went training like crazy. Lenora I suppose places second as hardest due in part of her actually knocking out my Musharna Cait-Sith ;_;
The one that gave me some worry was Elesa. She kept paralyzing all my Pokemon but we pulled through without a hitch. Same can be said for Brycen, he worried me but we were able to pull through. Not sure about the last gym leader Iris though, haven't gotten to her yet. Well here's hoping it will go well.