The neighborhood around Anthony's Pier 4 was a wasteland of rotting piers and dirt lots when the restaurant opened in 1963.