It's not so much the extent of the winter freeze or the summer heat that can spoil certain types of fruit tree, but the fact that British spring is notorious for feeling like summer in late February and then plunging back to arctic temperatures/snow/late frosts in March-April.
It's the late frost when blossom is open that is the main problem, but also a cool spring can mean very few pollinating insects available.

I was also under the impression that the UK's notoriously dull and damp climate are the main cause of fungal attack (such as leaf curl) on peaches and almonds.