A number of Secret Service officers and agents sent ahead of the trip of President Barack Obama to Cartagena, Colombia have been relieved of duty and sent home. There was alleged misconduct involving prostitution by the agents and officers. Close to a dozen Secret Service officers and agents are under investigation due to the findings that say they returned to their hotel with a number of prostitutes in Cartagena.

None of the officers or agents under investigation was on the personal protective detail for the president. The allegations overshadowed the beginning of the Summit of the Americas. The president arrived on Friday for the event and will focus on energy, trade and security in the region.

Before the president arrived, a number of agents from the Secret Service were sent home, an agency spokesperson said. Edwin Donovan a spokesman for the Secret Service said, “There were allegations of misconduct against our agents in Cartagena. Because this happened, those personnel were relieved of their current assignments and returned to their prior place of duty.”

At least one Secret Service agent had a dispute with the woman he brought back to the hotel, about a request for payment, said source in the U.S. government. One of the Secret Service agents refused to pay one of the women and she in turn complained to local police.

One inside source said it could amount to being the Service’s largest scandal in history. Twelve agents were alleged to be involved in one way or another, from interfering in the subsequent investigation to being actually involved in the misconduct.