do you have some flowers around? I noticed that aphids like better everything that smell sweet!
I had the same problem last year, but as soon as I put my pot close to the sweet peas, they vanished! Either they went away, or they have been eaten by something lurking in the flowers...