Pet Peeve #002 : The listener of the rude interruptor

I try to be overly considerate not to talk in group settings where there is someone presenting or a clearly established meeting moderator.  I had an incident today that I want to draw attention to so that hopefully others won't cause such a situation themselves and/or for you to tell me how I could of or should have handled the matter.

While I was in the audience of a meeting, the person to my right decided to share personal anecdotes about the topic being discussed.  He was speaking in a normal tone of voice, not a whisper.  I found his volume and timing to both be inappropriate.  I tried to do a quick "yeah, yeah" and noticeably turn my attention back to the meeting moderator - but he continued to talk.

While others started to notice the rude talkers (plural from their perspective, as in he and I) over on that side of the room, I was getting the scowls as well as the talker for being an accomplice - It seemed to me that to others, the listener is just as wrong as the talker.

Should I have more actively 'hushed' the talker ? - Is being a passive listener to a rude interrupter an offense ? - Let me know, comments welcome below...

Best Regards,
Dave