There's no two ways about it: that's creepy. Although technically no better or worse than using your search history as a template for targeted advertising, the idea of Google literally listening to phone or VOIP conversations is downright unsettling, even if it is just through a computer proxy. The patent covers "environmental conditions" detected by any of a smartphone's sensors, including location, temperature, humidity, or physical movement. Just as an example, an engineer could write a set of parameters looking for the keywords "football game" in conjunction with rapid shifts in a person's speed and direction to target people playing a game of pick-up, then serve an ad for Jamba Juice when they search for a drink.
Other sections of the patent point towards contextual advertising beyond a mobile device, including nearby ad vectors. In the football game example above, a digital billboard might be alerted to particular users and begin flashing a Jamba Juice ad with directions, not waiting for the user to activate his or her phone. While some of the more passive methods of tracking in the patent are much less disturbing than the voice keywords, it's all a considerable step above what Google's already doing with search and email.
Google's been under some pretty intense privacy scrutiny in the last year or two, and it's at a head at the moment. To be blunt, they'd be hard pressed to implement a system like the one described in the patent in the current climate. The documentation goes back to 2005, so someone's been working on it for quite a while, but there's no evidence that a voice or sensor tracking system is going in to effect any time soon.