With so many variables how can you get an accurate assessment? Pro vs amateur....first time shooting something vs 50th time....different gear - 1DX or Mark IV vs a Rebel T2i or fast glass vs f4 or higher? Full unrestricted access vs shooting from the stands, bleachers or outside the fence.
I don't think you can get an accrate reading for what you are asking unless you specify a sport, location, skill level, gear etc...
No worries, and I wasn't trying to be terribly scientific with control variables and such. The general question is totally a matter of opinion and as you point out many other factors influence what makes it "hard".
I have taken photos of collegiate division I-A sports down to my friends playing pick up games with cameras ranging from rebels to a 1D3. I firmly believe that sports at the collegiate and higher level are easier to photograph regardless of their being indoor/outdoor just because more money goes into illuminating the stadium/arena for national broadcast. At any other level, the terrible lighting in gyms makes volleyball/basketball very difficult to photograph.
The hardest sporting event to shoot is the one you are least familiar with. Now with all things equal, you have field size, action and lighting. Bald eagles and hummingbirds are harder to shoot!
I honestly think it is different from shooter to shooter, sport to sport, level to level. I know a guy here who shoots volleyball mostly, and complains how hard other sports are to shoot, but how easy vball is for him. I, on the other hand, think vball is the hardest for me, just because of getting clean angles, lighting, the net, etc.
Competitive Allstar Cheer is the one for me. Easily the most challenging thing I shoot. I find myself bored shooting some of the other sports after doing Cheer the last couple of years.
At single day competitions you get 2mins and 30secs to get as many girls as possible on a team with as many as 30 kids where half of them never see the front of the floor. In baseball/Football there is always another at bat, another pitch, another play. Sometimes in Cheer you may only get one shot at a kid....One.
There is not only a horizontal plane but also a vertical one, they throw girls..and boys as high as 30+ ft in the air and will most of the times throw as many as 6-8 up at the same time flying into the sky.
The parental/coach element. A lot of times you have coaches, parents, other cheerleaders almost on top of you during the routine. They jump, clap, pound the ground around you and lord do they scream. Inches from your head do they scream. The volume level at times can be absolutley deafening. There where times this weekend at Worlds where my earplugs where useless and the crowd was so loud you couldnt even hear the routine music.
The lighting is not always ideal. In some scenarios you'll have lighting setups where you swear the man that set it up was looking to draw a portrait of a dalmation on the mat it's so spotty. +/- a stop or two sometimes. And oh by the way you have 30+ kids moving around all over the mat in 30+ different directions.
All that being said....
I love it and it is by far my favorite thing to shoot bar none. I am already having Worlds withdrawls waiting till next season to start.