The chairman of the Parliamentary Labour Party (PLP) says the party has “failed to address anti-Semitism”.
John Cryer said procedures had improved, but told BBC Radio 4’s Westminster Hour the party “needs to act much more quickly”.
He also criticised the growing threat of de-selecting MPs within the party, including against his own wife – Ellie Reeves – who is five months pregnant.
Labour MPs must confirm if they wish to stand at the next election by today.
A number of backbenches have already announced their intentions to step down, including former farming minister Jim Fitzpatrick, Kevin Barron and Ronnie Campbell.
Vauxhall MP Kate Hoey has also announced she would not be seeking re-election as a Labour candidate.
The PLP are also due to meet later amid increasing calls for the party to change its stance on Brexit.
Shadow chancellor John McDonnell echoed calls from other senior party figures for Labour to “get on with” backing a further referendum, and said he wanted to campaign for Remain.
But he defended his leader, Jeremy Corbyn, to the BBC’s Andrew Marr for “rightfully” trying to build a consensus.
Mr Cryer said under the new general secretary, Jennie Formby, procedures to deal with anti-Semitism in Labour had improved, but they did not go far enough.
“We’ve failed to address anti-Semitism,” he told the Westminster Hour. “The bottom line is, are we kicking people out of the party who are anti-Semitic? In some cases yes we are, but in some cases no.
“If you’re a racist, you shouldn’t be in the Labour Party.”
He claimed that in some cases, “you get a suspension but then the case just drags on and on, and it’s those cases where we need to act much more quickly”.
And he called the launch of an investigation into Labour by the Equality and Human Rights Commission “pretty embarrassing”.
‘Tolerance of different views’
When it came to the party’s MPs, he said there was “a lot of worry” about trigger ballots and possible de-selections.
He called for “tolerance” of different views at the top of the party, suggesting Mr Corbyn accepted differences, but others around him did not.
His wife, Ms Reeves – the MP for Lewisham West and Penge – had faced calls for a motion of no confidence by one member of her local party, but it was never moved.
However, a number of other MPs have faced such a vote, including Joan Ryan and Chris Leslie – who ended up leaving Labour to found the new party Change UK.
Mr Cryer said: “To do it with a woman who’s five months pregnant, who’s a relatively new MP and has clearly worked really hard – I think it’s beneath contempt.”
He added: “Across the party, whether you’re elected or not elected, you’ve got to have a tolerance for people with different views.
“If it is the case that people are going to be driven out on the basis of differences of opinion then it’s not going to be the Labour Party that I’ve always known.”