Pastor Ray Minniecon, one of the organisers of Sydney's Invasion Day rally, said Mr Abbott was 'an idiot' after the former Prime Minister said the arrival of the First Fleet was a 'good thing' for Indigenous Australians.
"He's an idiot, he doesn't understand Aboriginal people or the history of this country," he told NITV News.
Mr Minniecon, a Kabikabi and Gurang-Gurang man from Queensland, said Mr Abbott will "not accept the fact the arrival of the First Fleet meant the massacre and genocide of First Australians."
Speaking to 2GB radio, Mr Abbott said the landing of the First Fleet was a 'good thing' for First Australians.
"What happened on the 26th of January 1788 was, on balance, for everyone, Aboriginal people included, a good thing because it brought Western civilisation to this country, it brought Australia into the modern world," he said.
The former Prime Minister's comments praising Australia day followed his Sunday tweet that stated: "This Friday I will gladly join millions of my fellow Australians to declare my faith in what, to us, is surely the best country on earth."
During his 2GB interview, Mr Abbott said that January 26 is for celebration and that there are other days of the year to consider the country's issues.
"There are pressing problems and we should address them. We are nevertheless a great country and Australia Day I think is a wonderful opportunity to celebrate all of the things we have achieved given that there are 364 other days where we can lament the things that are yet to be done," he added.
Co-Chair of National Congress of Australia's First Peoples, Rod Little, said he was disappointed by the comments saying Indigenous Australians have little to feel good about towards the day.
"Do you think it was good for our ancestors to feel as though they were disposessed of their lands, there were killings, there were massacres," he said.
"To have the circumstances we have today in our education outcomes, our unemployment, our incarcaration rates, our health rates, really what good has come from that first landing?"
Mr Abbott concedes that for a long time 'white Australians did much better than Indigenous Australians,' but efforts to improve the lives of Aboriginal and Torres Strait Islander peoples are progressing.
"Even there I think we are making progress. We are starting to see a significant middle-class, we've got I think six Aboriginal people in the Parliament, and this is progress," he said.
"So I think that all of the things that we know and love about modern Australia are the lineal descendants of the attitudes that came ashore with the First Fleet on that day back in 1788."
Prime Minister Malcolm Turnbull responded to Mr Abbott's comments saying he won't engage in debate about Australian history of European settlement.
"Other than to say obviously it had tragic consequences for thousands of Indigenous Australians and the wrongs that were done in the past we are setting right," he told reporters in Townsville.
Abbot's view of the legacy of Indigenous Australians
Despite his comments, Mr Abbott says Australians should not underestimate the contribution the Aboriginal ethos has made to the wider Australian ethos.
“That easy going, stoicism, that laconic style that so characterises Australians is typical of the spirit that pervades Indigenous Australia,” he said.
"It's very rarely a one-way street, so to speak, but if you look at the Australian achievement over the last couple of centuries it is overwhelmingly a positive story, and increasingly it is a positive story for Indigenous Australians as well."
Arrente woman and writer Celeste Liddle took to Twitter to label Mr Abbott's comments as 'ignorant' and 'racist'.
"You really are an ignorant, elitist, racist little jerk, aren't you?" she tweeted.
Local Greens members from inner-city Sydney suburbs of Petersham and Newtown said Mr Abbott 'will never get it,' and asked him to join them and First Nations people at a march on January 26.
Track record of 'racist' remarks
Mr Abbott's latest comments come after a long list of controversial remarks. The former self-proclaimed 'Prime Minister for Indigenous Affairs' has built up a long record of insensitive comments toward First Australians.
Just last week, Mr Abbott tweeted January 26 is the 'best available day' to celebrate Australia Day after the Greens announced their renewed campaign to change the date of the national holiday.
"There are 364 other days a year for the Greens to be politically correct. Why can’t they just accept that Jan 26 is the best available day to celebrate all that’s good about life in Australia," he tweeted.
In 2015, Mr Abbott, who was Prime Minister at the time, commented on the plight of First Nations people in Western Australia facing community closures referring to Aboriginal people living in remote communities as a 'lifestyle choice.'
"What we can't do is endlessly subsidise lifestyle choices if those lifestyle choices are not conducive to the kind of full participation in Australian society that everyone should have," he told a Kalgoorlie radio station.
"It is not the job of the taxpayer to subsidise lifestyle choices. It is the job of the taxpayer to provide reasonable services in a reasonable way."
His comments came after he backed a plan in the state to close more than 150 remote communities putting approximately 12,000 Aboriginal people in those communities on notice.
At the time, Cape York leader Noel Pearson slammed the former Prime Minister's comments.
"I think it's a very disappointing and hopeless statement by the Prime Minister, quite frankly," he told the World Today.
Former Aboriginal and Torres Strait Islander Social Justice Commissioner Mick Gooda said he was "baffled" by Mr Abbott's comments, saying his 'lifestyle choices' remark was 'poorly thought out.'
"I think they will cause offence in the Aboriginal and Torres Strait Islander community," he told the ABC in 2015.
Mr Abbott's 'lifestyle choices' comments reached the attention of the United Nations with UN special rapporteur on the rights of indigenous people saying Australia had 'regressed' and that a remark by the Prime Minister about remote communities "smacks of racism."
"This whole issue of racist kinds of pronouncements doesn’t really speak well of how governments are supposed to be complying with their human rights obligations," UN rapporteur Victoria Tauli-Corpuz told the Guardian.
A year earlier, in 2014, Mr Abbott was accused of believing Australia was a "terra nullius" before British settlement, after his remark that Sydney was 'nothing but bush' prior to the arrival of the First Fleet.
Speaking at a breakfast for former British Prime Minister David Cameron, Mr Abbott made a speech about the "extraordinary partnership" between the two countries since the arrival of the First Fleet in 1788.
"As we look around this glorious city, as we see the extraordinary development, it's hard to think that back in 1788 it was nothing but bush," he said.
Mr Abbott's former chief Indigenous advisor, Warren Mundine, responded saying the comments were 'silly' and 'bizarre,' while former Labor senator Nova Peris said it was "highly offensive."
"British settlement was not foreign investment. It was occupation," she said at the time.
