One US serviceman died and four others were wounded in the attack on Sunday, the first major US military action since Donald Trump became president.
After the raid a Yemeni official said the attack left 41 Al-Qaeda members dead, plus eight women and eight children.
The International Crisis Group, an independent group that analyses conflicts around the world, said the raid had left many civilians dead including at least 10 women and children.
The US military's Central Command said it was "likely" that civilians died in the raid, which targeted members of Al-Qaeda in the Arabian Peninsula in the Yakla region of Yemen's Baida province.
"When you think of the loss of life throughout America and institutions and in terms of the world, in terms of what some of the individuals could have done, I think it is a successful operation by all standards," White House spokesman Sean Spicer said, referring to the people targeted in the attack.
He added that it was hard to talk of success when an American soldier was killed, and praised the soldier's sacrifice in fighting what Spicer called people who posed a threat to America.
Spicer made no mention of civilian victims.
Trump has vowed to fight Islamic extremism relentlessly.
The president traveled to an air base in Delaware Wednesday to receive the body of the US Navy SEAL killed in Yemen, identified as Chief Petty Officer William "Ryan" Owens, age 36.
The US military is investigating the reports of civilian casualties, while accounts which have appeared in the and the detail a protracted firefight in which some of the women killed had taken up arms.
President Trump departs the White House to honor the NAVY seal killed in the president's first military raid. February 1, 2017. Source: Getty Images North America
Among the children killed in the raid was Nawar, the eight-year-old daughter of assassinated US-born preacher Anwar al-Awlaqi, who lived with the family of her maternal uncle, a relative said.
Awlaqi was killed in September 2011 in a drone strike under the Obama administration.
His sixteen-year-old son Abulrahman was killed two weeks later in a similar US attack.
The International Crisis Group has said that the use of US troops and the high number of civilian casualties are "deeply inflammatory and breed anti-American resentment across the political spectrum."
The fallout could encourage support for Al-Qaeda in Yemen, the group says.
Yemen has been engulfed in a bloody civil war since an attempted coup by Houthi rebels in 2015 forced the government to flee the capital to neighbouring Saudi Arabia.
Saudi Arabia - an ally of the overthrown government - imposed a punishing naval blockade and has led air-strikes in an attempt to push back rebel forces.
Almost 3,000 civilians have been killed, according to the UN, many in Saudi airstrikes which have hit hospitals, schools and factories.
Amid the chaos, Islamist terror groups such as Al-Qaeda and Islamic State have attempted to expand their reach.
