The main information security group I network with is my local chapter of ISSA (Information System Security Association). There are chapters across the world.
I am also involved in a local group called HackMiami.
I am also member of my local ISACA and IAPP chapters, but these groups are NOT mainly infosec.
You might find local, less formal groups in many areas. Some may be tied to Defcon. These are usually called "DC<somenumber>".
Hope this helps.
---
Michael Brown, CISSP, HCISPP, CISA, CISM, CGEIT, CRISC, CDPSE, GSLC, GSTRT, GLEG, GSNA, CIST, CIGE, ISSA Fellow