I'll leave mine out until it gets close to frost time. Some of the plants were grown from pinched out side shoots so will be the last but I reckon letting them ripen at their own pace means I get a steady supply rather than a big glut.
Any still green before the frost will be popped on a sunny windowsill and some will be used for battered fried green tomatoes.