Some neighbors have expressed annoyance with the throngs of media and crowds that have gathered in recent days