After spending at least 20 minutes pulling my hair out today, I stumbled across some cryptic post that made a whole lot of sense.
When setting up wordpress to accept posts via a google apps email address (or any gmail address I would assume), you enter the host as ssl://pop.gmail.com instead of just pop.gmail.com.
Funny what a little change like that will do to your sanity.