Theresa May is under pressure to get an agreement from the Democratic Unionist Party (DUP) on the status of the Irish border when the UK leaves the EU.
The prime minister pulled out of a deal with Brussels that would have kick-started trade talks after meeting fierce resistance from the DUP.
The party said it would not accept a deal which saw Northern Ireland treated differently from the rest of the UK.
Mrs May is due to update ministers at a cabinet meeting on Tuesday morning.
Former Brexit minister David Jones described Monday as a “very difficult day” and said the prime minister now had “a lot of talking to do” with DUP leader Arlene Foster.
He believed the DUP had been spoken to beforehand about what was being proposed, but that the precise wording had not been explained and the Northern Ireland party “clearly thought twice”.
Mr Jones, one of the Eurosceptic MPs on the Conservative benches, said he too was not happy with the key phrase the DUP objected to in the draft agreement, which apparently guaranteed “regulatory alignment” between Northern Ireland and the Republic of Ireland.
The prime minister needs the support of the DUP, which is Northern Ireland’s largest party and has 10 MPs at Westminster, because she does not have a majority to win votes in the House of Commons.
Former top Foreign Office official Lord Ricketts said the row was “damaging” for Mrs May, adding that it was “pretty extraordinary that this wasn’t all stitched up with the DUP beforehand”.
“We’re used to prime ministers going to Brussels and having a row with the EU and coming back without an agreement, but to go to agree with the EU and then have a row on your own side is inconvenient,” he said.
The UK is due to leave the EU in March 2019 having voted for Brexit in last year’s referendum.
Mrs May wants to begin drawing up a deal on post-Brexit trade relations between the UK and the EU but first needs to get an agreement on “divorce” issues before European leaders meet on 14 December.
The three issues that need to be resolved are the Northern Ireland border, citizens’ rights and the amount of money the UK will pay as it leaves.
She will return to Brussels as early as Wednesday to try again.
Talks with European Commission president Jean-Claude Juncker broke up without agreement on Monday, although both sides said they were hopeful of getting a deal by the end of the week.
Mrs May had been expecting to make a major statement to MPs about a breakthrough in Brexit talks on Tuesday, but that has now been cancelled.
Instead, it is thought she will hold meetings with the DUP, as she attempts to get them to support her.
wise people got already engaged