More Rosemary going downhill

[Marb67]

Well, every year I just can't seem to grow Rosemary despite heeding advice. One has completely died, one planted in the soil is yellowing and the other in a pot has mottled, sickly leaves and some dead stems. They are not left in waterlogged compost. The pots are on gravel and the raised bed it gritty, well drained soil. The garden is walled and sheltered from harsh weather so what can this be ? I just can't seem to be able to grow this at all