I had the same problem last year and then all of a sudden I had reasonable sized butternuts! The same has happened this year - at the moment they are the size of a golf ball - so wait a while to see what happens.