AJO, Ariz. — Some people here call it Sesame Street. Others call it Legoland.

They're referencing the boxy-shaped, brightly colored houses built by the government for U.S. Customs and Border Patrol and agents who work in and around this small former copper-mining town near the Mexican border.

The houses, some bright blue, others the color of salmon, are lined neatly along two rows on a hill near Ajo's historic plaza. They have been a source of contention both for locals and now for federal investigators, who say the government spent nearly $700,000 per house in a city where the average home costs less than $90,000.

The report by the Department of Homeland Security's inspector general found that U.S. Customs and Border Protection overspent by about $4.6 million on new houses and mobile homes. The agency spent about $17 million for land, 21 two- and three-bedroom houses and 20 mobile homes. Construction was completed in December 2012.

Critics are calling it a classic case of government waste.

"My personal opinion is that they spent too much money on the homes. It's ridiculous," said Ernie Crowell, a retired, lifelong resident who was one of the few who stuck around after the mine was shut down in the mid-1980s.

The agency began the project after anticipating a need for more housing for agents as the Border Patrol doubled in size from 2004 and 2008. But opponents questioned the need to build new homes in Ajo at a time when existing homes in town were vacant.

Linda Sharp, a real estate broker who splits her time between Alaska and Ajo, says the government decided to build houses not because there was a shortage, but because the current structures were not up to standard for agents. Many of the homes in Ajo are older and don't always have the amenities that newer housing offers.

"There's just always been more than enough houses in Ajo, always," Sharp said. "I'm the taxpayer and you're telling me the house is not good enough for your workers?"

Customs and Border Protection says that that while the agency agrees with recommendations made in the report, it disputes the way the inspector general calculated the cost of each house and mobile home, calling the method "comparing apples to oranges."

In a statement, the agency said it normally lets workers rely on the private housing market but called Ajo an exception to the rule given the isolation of the town and the lack of suitable homes. The agency said the homes were built in accordance with government standards and guidelines.

Spokesman Jim Burns said the agency remains committed to providing quality, cost-effective housing for front-line border security personnel and their families. Burns declined to say on Friday how many of the homes are occupied.

The town of less than 4,000 southwest of Phoenix has few jobs and relies in part on tourists headed to Mexico for revenue. Ajo also draws a lot of so-called "snowbirds," or residents who only live here in the winter.

Bety Allen, the executive director of the Ajo District Chamber of Commerce, says that with more housing options available, the community will be better off. When she and her husband, a border agent, moved to Ajo about three years ago, the couple struggled to find a home that would allow their five dogs.

"That was a very, very stressful time," she said "If those (new CBP) homes had been available for us, it would have been a lot easier."

Standing outside Olsens Supermarket off the highway that leads into town, Kim Cubillas said she has always heard the rumblings against the housing, but she's not directly affected.

Still, she added that the money spent could have benefited local schools.

"They should have made them available to Ajo residents as well, not just the Department of Homeland Security," Cubillas said.

Candace Gregory spent three-quarters of her retirement savings to buy four houses in anticipation of the Border Patrol growth. Now, they are empty and "falling into disrepair," she said.

Gregory says she has had to go back to work because she's been unable to rent the houses she and her husband bought.

"It's all just going down the drain," she said.