There's more to how human understands conversation from others.

Eye contact, visual, context. You understand another person in a noisy environment because you can recognize his/her voice, so you know to pick that one up and to isolate the rest. You also have the visual assistance of looking at that person's face to see his/her lips moving, so you know when to expect to hear something and when not to. Then there is also the factor of context. You know that person best, and what that person would be talking about.

Now let's try theory craft here and assume how would one be required to program a voice recognition software to work in the same manner. It's not going to be easy at all. It's not as straight forward as, "Jarvis, please schedule an appointment with Pepper tomorrow at 3pm, and organize for a new Aston to be at my door tomorrow for that. Oh, and make sure to send some nice flowers to her too."