In shooting the documentary ‘Nick Nolte: No Exit’(available On Demand via Sundance Selects since December 30), director Tom Thurman concocted a rather existential ploy to bring out the burly, intimidating sixty-eight year-old actor’s vulnerable side: he assigned Nolte to literally interview himself. Hopefully, this oddball tactic would uncover the true Nolte, who in the past notoriously relished in embellishment.

“I’m known for being a liar,” Nolte said. “I lied for a living and I lied in my interviews. It got boring after twenty years. People have said to me, after interviews, ‘I really wished you answered that motherfucker’s question!'”

To avoid that escapism, Thurman and Nolte, who became friends while filming Thurman’s ‘Buy the Ticket, Take the Ride: Hunter S. Thompson on Film,’ developed a series of deeply intimate questions, several of which would probably catch Nolte off guard if an actual journalist posed them. In one day, Thurman shot footage of Nolte as slick reporter, sheepishly grilling Nolte the actor; the very next day, he filmed a wearier-looking Nolte answering his own questions, sometimes wistfully, sometimes sarcastically. Nostalgic slides of Nolte in his salad days were projected at random onto a computer throughout the “interview,” further derailing him.

“We both wanted the interviews [in the film] to be prison-like, with no escape [from myself],” said Nolte. “We shot in low light. And if I ever got out of line with myself, Tom would shut the camera off.”

Not that this challenging, self-reducing stunt on Nolte’s part comes as any surprise to his peers, a cross-section of whom–Rosanna Arquette, Powers Boothe, Ben Stiller, Barbara Hershey and others–add unflinchingly honest commentary to the film. Even his approach to Method acting, no doubt culled from his idol Marlon Brando, has taken him to dark, unseemly places. Paul Mazursky, who directed Nolte in the 1986 comedy ‘Down and Out in Beverly Hills,’ recalls how the actor lived briefly on downtown L.A. streets to nail down the lead role of a filthy bum.

“I was hanging out around this homeless section, it was like Whorehouse Alley,” Nolte said, “and I was sleeping there for two days before I was supposed to get a bed in a mission shelter. It doesn’t take more than seven hours or so outside to turn into a space cadet. Your mind just gets nuts. And then I couldn’t get into the shelter. I didn’t have seniority.” After that experience, Nolte delved further into the part by refusing to bathe for weeks. “[Co-star] Bette Midler was disgusted,” Nolte chuckled.

But even at the start of his film career ten years earlier, long before he had the industry clout to freely experiment with roles, Nolte’s choices hardly catapulted him into quite the Hollywood star he could have become. He bloomed late in Hollywood at thirty-five, after over a decade of theater work, and despite his dashing, golden-haired looks, the expertise he’d picked up gave him the foresight to avoid limiting—if potentially stardom-boosting—matinee idol casting. He has sex appeal, but it’s unkempt, and while he often gives off a brutish aura, the machismo in his roles often rises out of turbulence—perhaps stemmed from his quietly haunting, Omaha, Nebraska upbringing by a WWII vet father.

“When these soldiers came home, they wanted only peace, no controversy,” remembers Nolte. “The 1950s were a time of great secrecy and fear, with no questioning of authority. I was in conflict with everyone from that generation—coaches and teachers especially.”

That tension led to some early acting-out, a rebellion easily identified in Nolte’s often impetuous on-screen demeanor. “At football camp, some friends of mine and I shit in a bag, tied it real tight so no one could open it, and left it in the dormitory,” he fondly recalls. “The coach came over to my bed that night and said ‘I know you did that, and I’ll make sure you never play football again.’ I’m sure he made some phone calls.”

It’s a miracle football didn’t pan out, for that lifestyle clearly wouldn’t have suited Nolte’s refusal to conform. To this day, Nolte still opts for difficult roles, often as ornery, tortured men, prone to outbursts of rage—as in his Oscar-nominated turn as the drunken antihero of Paul Schrader’s 1998 drama Affliction. And he’s often threatened by an overabundance of success, comforted more by a string of failures.

“If you’re successful all the time, you don’t learn anything,” Nolte explained. “You get no introspection. It’s impossible to assimilate without some disturbance to your personality.”

“Studios try to triple your salary when you have a bunch of successes,” Nolte continued. “The self-pressures are immense. It’s an idiotic game, just for short-term profit. And you lose your anonymity. A failure can offset that.” Plus, to most Hollywood studios, “success has to come in only three weeks. ‘Hulk’ (2003) was considered a failure because it was ranked seventh or eighth in the box office. But it made money back. Almost every film does.”

Perhaps it’s that very attitude that has led Nolte to work repeatedly with directors not known for box-office hits, most notably Alan Rudolph, who cast Nolte in four of his films, including the critically savaged 1999 Kurt Vonnegut adaptation ‘Breakfast of Champions.’ In general, he is more scintillated by filmmakers and writers that refuse to pander to audiences.

’When Mother Night’ (1996) came out, I saw the film with [screenwriter] Kurt Vonnegut, and afterwards he said, ‘The audience seemed perturbed. That’s good.’ [John] Cassavettes was like that. If a test audience liked a part, he’d cut it.”

Nolte will next appear in Gavin O’Connor’s Warrior, about a family of martial artists, to be released in the fall of 2011.