Conservative MPs said retaining their seats at the next general election will “no doubt” be difficult following the double by-election loss.
The party lost their former stronghold of Tiverton and Honiton to the Liberal Democrats and surrendered Wakefield to Labour at the by-elections on Thursday night.
Meanwhile, Conservative Party co-chairman Oliver Dowden quit, saying he and Tory supporters are “distressed and disappointed by recent events”, telling Mr Johnson: “Someone must take responsibility.”
Tory grandee Sir Geoffrey Clifton-Brown said there is “no doubt” it would be “difficult to hold” his seat if there were a by-election in his constituency now.
Speaking to BBC Radio 4’s Today programme, the MP for the Cotswolds and treasurer of the 1922 committee said: “I think, factually, if I were to run under a bus today it would be difficult to hold my seat. There’s no doubt about that.
Conservative MP Jesse Norman, who wrote to Mr Johnson earlier in June to say he was withdrawing his support due to the partygate scandal and his government’s lack of “a sense of mission”, reiterated his concerns over their impact on the electorate.
Mr Norman tweeted a quote from that letter on Friday morning, writing: “For you to prolong this charade not only insults the electorate, and the people who support, volunteer, represent and campaign for our party; it makes a decisive change of government at the next election much more likely.
“That is potentially catastrophic for this country.”
Veteran Tory MP Sir Roger Gale said the Prime Minister has “trashed” the reputation of the Conservative Party.
He said Mr Johnson is choosing to “hang on to the door handle at No 10” but “it can’t go on forever, and it certainly won’t go on until the next general election”.
Asked if he sees Mr Dowden’s resignation as a trigger for more expressions of discontent from the Cabinet, he told BBC Breakfast: “It is possible that that may happen, but it is up to my colleagues in the Cabinet to decide whether they can go on supporting a prime minister who, frankly, has trashed the reputation of the Conservative Party, my party, for honesty, for decency, for integrity and for compassion.”
Senior Tory MP Tobias Ellwood tweeted that Mr Dowden’s resignation was “honourable”.
Meanwhile, Tory peer Lord Barwell said the Tiverton and Honiton result was “catastrophic” for the Conservatives and that other seats in the south are “vulnerable”.
He told Sky News: “It’s one of the safest Conservative seats in the country. It’s a strongly Leave-supporting constituency.
“So, for the Liberal Democrats to be winning there, and winning comfortably, it means that there’s a whole swathe of seats across the south of the country that are vulnerable.”
Sir Robert Buckland said he has told Mr Johnson he needs to “look in the mirror and do better” as he told Sky News the Conservative Party is “about more than one man”.
“What is frustrating for those of us now on the side, if you like, is that lack of focus and a real sense of a co-ordinated message here about what the Government is doing and what it needs to do,” he said.
“The Conservative Party is a broad coalition of people who have different views across the centre right in politics. We need to reflect that far, far better – we’re not a sect, we’re not some iconoclastic tribe trying to overthrow the state.”
The by-elections, triggered by the resignation of disgraced Tories, offered voters the chance to give their verdict on the Prime Minister just weeks after 41% of his MPs cast their ballots against him.
In the rural Devon constituency of Tiverton and Honiton, the Liberal Democrats overturned a 24,000 Tory majority to win, while Labour reclaimed Wakefield.
Mr Johnson sought to deflect from the by-election defeats being about his leadership as he spoke to broadcasters from Rwanda on Friday morning.
He said: “I think that what governments also have to recognise is that I don’t want to minimise the importance of what voters are saying, but it is also true that in mid-term, governments, post-war, lose by-elections.
“I think, if you look back to last May, the truly astonishing thing was we managed to win Hartlepool in very different circumstances.”
Mr Johnson said the country is facing pressures on the cost of living, adding: “We have to recognise that there is more that we have got to do and we certainly will; we will keep going addressing the concerns of people until we get through this patch.”