When the whole town comes beating on your door you sign them all up for the militia.

There is a big difference between being covert or being in the open, if you are staying in your neighborhood and defending it people will obviously know.

Another thing to keep in mind is the need to know basis, if you have a militia formed ad-hoc they would only be capable of very simple tasks and for their own safety and yours they need to know very little.