As PaulW said, it's only an offence if the nest is in use or being built. The
Defra website says:
"It is an offence under section 1 of the Act to kill, injure or take any wild bird; take, damage or destroy the nest of any wild bird while that nest is in use or being built, and take or destroy an egg of any wild bird."