The Patrick administration has decided to restrict work on a high-profile $285 million bridge project over the Merrimack River to contractors who agree to use only union labor, a rare move that has the state’s nonunion construction firms accusing Democrats of election-year pandering.

Advocates for the nonunion firms, some of which have poured thousands of dollars ­into preparing bids for the work, say the administration had given them a clear indication that the Whittier Bridge project would not have any such limitations.