SEDONA, Ariz. -- CBS 5 News has learned that the crude sweat lodge constructed at the Angel Valley Resort that killed two people should never have been built.

Yavapai County building safety manager Jack Judd said there was no record of an application or permit for a temporary structure at the Angel Valley Retreat Center.

Self-help expert and author James Ray hosted the event. His publicist said the sweat lodge was built by the workers at Angel Valley; however, Yavapai County Sheriff's Office spokesman Dwight D'Evelyn said authorities believe Ray's staff either participated in erecting the structure or oversaw the construction.

Ray's spokesman, Howard Bragman, declined to comment on the permit, but said the resort's staff was under contract to build a sweat lodge to accommodate up to 75 people.

Resort owner Amayra Hamilton said Saturday the sweat lodge, built with a wood frame and covered with layers of tarps and blankets, had been taken down.

Judd said no inspection of the structure was conducted before it was dismantled.

The owners of the resort have not disputed that Angel Valley helped build the lodge, and have said that Angel Valley has been building sweat lodges for Ray and his groups for seven years. Last year, Ray asked to make the lodge even bigger so it could accommodate more people, the owners said.

Angel Valley representatives said resort personnel specifically told Ray it was a bad idea to build the lodge, and that cramming that many people into that small a space wasn't safe. Others who have used sweat lodges before said they agree.

Those who have experience with sweat lodges said the structures are meant to contain a very small group. On Thursday night, however, between 55 to 65 people attending the "Spiritual Warrior" program were crowded into the 415-square-foot space during a two-hour period Thursday night, sheriff's officials said.

Critics of the oversized lodge have decried the use of a sweat lodge for financial gain, saying the practice should be reserved for spiritual enlightenment. Ray charged as much as $9,000 per person to participate in this one.

Sweat lodge user Lina Rueda said, "It takes a lot of responsibility to run one and I think that some of those things must have gotten overlooked, unfortunately, but when they're done right it's just a beautiful experience."

Officials at Verde Valley Medical Center said all but three of the people initially taken to the hospital have been released.

Two are listed in fair condition, but one is still in critical condition.

Verde Valley Fire Chief Jerry Doerksen, whose department responded to the initial 911 call of two people not breathing, said his department regularly inspects the Hamilton's property.

An inspection of the fire area used to heat cantaloupe-sized rocks that were taken into the sweat lodge determined it "would have been a legal fire," he said.

"Where they had the fire, they had a big area outside of it that was cleared," he said. "There wasn't a risk of it spreading someplace."

The sheriff's office continues to investigate whether to determine if criminal negligence played a role in the fatalities.

Kirby Brown, 38, of Westtown, N.Y., and James Shore, 40, of Milwaukee died after being overcome in the sauna-like hut, which was built specifically for the five-day retreat.