Centrelink has sent out almost 200,000 debt notices to current and former welfare recipients since the start of the financial year.
The automated data-matching system is part of an effort by the government to retrieve an estimated four-billion-dollars it says is owed.
But reports continue to emerge of wrongly-issued notices, with the government conceding around 20 per cent of those issued have been to people who didn't owe any money to Centrelink.
The Federation of Ethnic Communities' Councils of Australia says many migrants and refugees have been scared by the letters.
Spokeswoman Emma Campbell says its especially difficult for those who have come from countries with intimidating and threatening governments.
"If English is your first language the letters are confusing and distressing enough. If English is not your first language then it can be almost terrifying"
Federal Opposition M-P, Anthony Albanese, says he has received many complaints from his constituents.
Mr Albanese says one of the complaints was from 29 year old Sydney man Tony Barber.
Mr Barber received a Centrelink debt notice for $4500 before Christmas.
The cancer survivor says he had taken time off work in 2010 for chemotherapy treatments and received welfare payments before returning to work in January the following year.
Mr Albanese says it appeared Centrelink had taken Mr Barber's income from the periods when he was working and applied it over the whole year, not taking into account that he was sick and unable to work.
He says it's not good enough.
"I can tell you that people have been ringing elected officers around the country, who feel very vulnerable. If you are someone where English isn't your first language, or you are elderly, or you are just not dealing with bureaucracy. But, I must say, as a politician, who is used to receiving a lot of correspondence, these letters are pretty brutal. They don't provide a great deal of detail. They just immediately threaten people with debt collections."
But the government remains unapologetic, saying $300 million has been recovered through the controversial scheme.
Deputy Prime Minister Barnaby Joyce has told the A-B-C, the government has confidence in Centrelink's debt recovery program.
"We make no apology for the fact that we are trying to make sure that we are more proficient, have a wider grasp of those who might have received payments in error. We are doing that because we are not getting people to check, we are getting a computer to say, 'These are the details you told the Taxation Department, these are the details you told the Social Security Department. They don't match up. We need you to explain the difference."
The Auditor General says it is assessing Centrelink's data matching system.
