I hardly ever shoot at F4 with the 17-40, but that's because I mostly take car pictures and I always want to be sure I won't end up with a blurry rear wheel
Yet I never came across a situation where it was so dark than f4 wasn't enough (I'd just compensate with the ISOs instead), you may ask TT about this as he has much more experience with everyday street sightings.