Washington (CNN) - CIA Director Leon Panetta told House members Tuesday that any way you look at it, Pakistan's role in Osama bin Laden's whereabouts was troubling.
According to two sources in a closed door briefing, Panetta told lawmakers "either they were involved or incompetent. Neither place is a good place to be."
Panetta was responding to a question from a member of Congress about Pakistan, which was the first question of the hour-long classified briefing, the sources said.
He made clear that he and other administration officials are trying to get to the bottom of which it was -involvement or incompetence.
Bipartisan frustration had been boiling over from both parties when it became clear bin Laden's compound was in an urban area less than a mile from a major Pakistani military academy.
"It had everything except a neon sign sticking out there," Democratic Sen. Frank Lautenberg of New Jersey, told CNN.
Freshman Republican Allen West of Florida, who served in the army in Iraq and was a civilian officer in Afghanistan, told CNN "there is no way that people in the ISI [Pakistan's Intelligence Agency] and military did not know that Osama bin Laden has been living there for quite some time."
Lautenberg and West are among several lawmakers who say Congress should cut off funding immediately, to try to force Pakistani officials to answer questions about what they knew about Bin Laden's location.
Over the last eight years, the U.S. has given Pakistan nearly $20 billion dollars in foreign aid, mostly to try to convince the tenuous ally to help combat terrorism.
The State Department requested $3 billion more for next year. The Defense Department asked for $2.3 billion just for Pakistani counterterrorism efforts.
Despite calls to withhold funding now, many senior lawmakers say that's premature and that Congress should wait until the U.S. gets answers from Pakistan.
"Was this just benign indifference, or was it indifference with a motive? I don't know the answer but we need to find out." said Democratic Senate Intelligence Chairwoman Dianne Feinstein of California.
Members of congress who are warning against a rush to judgment note different factions of the Pakistani government offer different levels of cooperation. The relationship is often tense, and quite complex.
"They've lost thousands and thousands and thousands of their soldiers fighting terrorists. Now this doesn't mean we don't need more oversight, and I'm willing to do that," said Senate Majority Leader Harry Reid.
"It's complicated because it's not just all security. They are a country right in the middle of where everything is happening. They do work with us when they choose to," said GOP Rep. Sue Myrick of North Carolina.