Barack Obama, under attack for his "condescending" remarks about small-town America, told Pennsylvania voters on Tuesday that it's just "silly" to call someone who was raised by a single mother and who was on food stamps an elitist.

Both Hillary Clinton and John McCain have assailed Obama for saying that small-town voters are clinging to guns and religion because they are frustrated with Washington. Clinton, who has repeatedly broached the issue with voters, launched an ad in Pennsylvania Tuesday featuring five residents criticizing Obama's remarks.

Asked about the flak he's been taking, Obama said the charges make little sense to him.

"I am amused about this notion of elitist, given that when you're raised by a single mom, when you're on food stamps for a while when you were growing up, you went to school on scholarship...." he said, also talking about his wife Michelle's money troubles.

"So when somebody makes that argument, particularly given that I spent my entire life working with workers, low-income communities to try to make people's lives a little bit better, then that's when you know we're in political silly season," he told a veterans forum in Washington, Pa.

"Hopefully it'll come to an end fairly soon and we can start focusing on the issues that the people of Pennsylvania and the American people care about."

The individual who asked about the comments suggested the charges of elitism had racial overtones.

"As a white person, this term, the way it is being used against you, it isn't far from 'uppity,' OK?" the man told Obama angrily. "And I think the Clintons are getting away with something that they must be called on. They will continue to do it until somebody states, ‘Mrs Clinton, you're really close to prejudice here. This is wrong.'"

Obama said it was "nice to say that," but he didn't think racial overtones were shading the debate.

"I think that, you know, it's politics," he said. "When we start getting behind in races, then we start going on the attack."

In a counter to Clinton's ad, Obama's campaign release an ad Tuesday that showed footage of Clinton being jeered for bringing up Obama's "bitter" comments at a Pittsburgh event the day before.

"There's a reason people are rejecting Hillary Clinton's attacks," the narrator says. "Because the same old Washington politics won't lower the price of gas or help our struggling economy. Barack Obama will represent all Americans. He offers a new approach"

Obama made the controversial remarks at a San Francisco fund-raiser a week ago, saying voters facing economic hard times in rural Pennsylvania and elsewhere "cling to guns or religion or antipathy to people who aren't like them or anti-immigrant sentiment or anti-trade sentiment as a way to explain their frustrations."

His opponents, and his opponents' surrogates, say that Obama's words showed his true disdain for small-town voters, expressed in an unguarded moment among friends.

But with just one week remaining until the April 22 Pennsylvania primary, and with little movement in the polls as a result of the flap, Obama has said that voters are justifiably angry over high gas prices, the loss of manufacturing jobs and other examples of economic insecurity, but can still appreciate a message of hope.

"Just because you're mad, just because it seems like nobody is listening to ordinary Americans, that's not a reason to give up hope," Obama told the Building Trades National Legislative Conference.

"You get mad and then you decide you're going to change it. If you're not angry about something you're going to sit back and let it happen to you. If you're only angry, you don't feel hopeful."

FOX News' Aaron Bruns contributed to this report.