Torture and Rape isn't touched on much in games, and for good reason, gaming is about fun escape and dealing with such horrific concepts always lies outside people's comfort zones. More than that though, gaming is fantasy, and normal sane people don't fantasize about such things. To do so harkens back to the early 1980's during the height of gaming paranoia and Tom Hanks starred in a disgustingly misinformed movie called Mazes and Monsters. Rest assured there are dozens and dozens of wrong ways to game out such horrific deeds, methods that will inevitably cause player alienation and more often than not kill a game, however I've experienced games where such storytelling devices were used to create compelling and provocative stories. These stories were certainly not comfortable, but they were extremely memorable.
To me the most memorable sessions were the ones where the trials were unexpected and actually challenging, they were also the ones that deeply impacted my character. Often I finished the game session upset, annoyed, or spitting mad, and occasionally I left a game in shock. Most of this was because of one female GM who has a cruel streak. Her secret? Make the players care, and then sunder that which they cared about in the most graphic way possible. Nothing was sacred to her, no NPC was immune to her destructive whim, no law of reality could not be bent or broken to shatter the PC's illusions. Sometimes she took these shocking tactics too far, but sometimes too far was best.
I've experienced the horror my character sitting helpless in a prison cell nearly overcome by his personal demons while his soul mate in the next cell over - a pillar of support in a trying time - was gasping incoherently because her tongue was cut out in a torture session she'd just been returned to her cell from...
Another session my character was restrained and forced to watch another friend become drugged and sexually assaulted by a lecherous god because this was his "process" which he used to instill her with oracular ability. This oracle would go on to help us find out pivotal information that we needed. But my character had to come to terms with the fact that the information recieved was gotten at such a cost. He wanted to discard it, but that would have made his friend's sacrifice all for nothing...
Sessions like those were far from the most fun I've ever had in role-playing, but they taught me things about myself and developed my character in unexpected ways. When I started, the character was someone I envisioned like one of the badasses I'd seen in gritty movies like Sin City, while those experiences turned him into something that was completely different than that vision, the result was far more legitimate.
I can't say that I was always ready for the sorts of horrific scenarios that I've been subjected to in some of these games. I don't think anyone is really 'ready' if they actually do care about their characters. This is why Game Masters who do these things need to know their groups. There needs to be balance in the tone, players don't come to care about their characters if good things never happen to them, and if they don't care then the horrifics won't mean anything. A game should never be horrible all the time or there simply won't be any fun to draw players back.
One element I cannot stress enough is flexibility when dealing with this kind of storytelling. Absolutes and extremism mixed create very bad things. An example of this is the Carcossa Supplement V which was a blogger-made game that featured Cthonic gods and black horror. One of the controversial points of the game was that there were spells written with listed prerequisites involving Podiatry, sacrificing babies and similar heinous acts. I've examined what was behind the writer's motivations for creating such disgusting content, and while from a setting purist's stance I do see some degree of justification, everything I know about gaming tells me this sort of thing will not play out enjoyably. Even if players and Game Masters alike can get past the distasteful nature of the content, the gameplay will never end up as intended because the system utilizes D&D's inflexible alignment mechanics.
I'll tone down the subject matter to something more acceptable and relatable as I explain. We're all familiar with Star Wars' light force dark force concepts, where the dark force offers great power and thus tempts the Jedi to use it and fuel himself by doing dark acts. This is great compelling story and character development material, but is wrecked when one includes inflexible morality mechanics such as the ones that appear in D&D.
Alignment mechanics as they sit in D&D begin to break immersion when a player goes to the extreme ends of them. You get players that start doing stupid things "because my character is evil" rather than looking at the simple question of why is that character evil? Morality isn't black and white, especially when you consider that ethics and morality aren't the same thing. World of Darkness takes a different approach and makes morality a fluid stat that can be raised or lowered based on the PC's decisions, thus you can see a slide into darkness or a quest to redeem himself. The mechanic offers incentive to RP correctly but doesn't force anything because it's flexible. An inflexible mechanic forces the Player to contrive a reason why his character acts the way the Alignment says he does, rather than letting the player decide how to play his character on his own. In the case of the light-force/dark-force temptation an inflexible mechanic offers no incentive to stay good, or feel the strain of that slip into darkness, the player just makes a conscious decision to become a dark Jedi because they get the cool powers he wants and that's that.
Certainly role-playing horrific deeds can be accomplished independent of subtle mechanics, but absent those mechanics introducing heinous acts as prerequisites for great power suddenly becomes a personal issue and is in fact enforcing certain behaviors where the only check or balance is the players and GM's own distaste for such material. Absent flexible morality mechanics, a game that has a power or spell requiring a horrific act - be it something to appease dark gods or tap the dark side of the force - you have a number of reactions occurring:
• The GM will disregard the requisites as distasteful and choose to run the game without such content.
• A player will either be disgusted and refuse to play, or apathetic and if he wants to cast that spell he'll make a character that "has to be this evil bastard, because that's what I need to cast the spell".
• Or the player is somehow intrigued by the concept of such a thing and wants to play it out.
All of these reactions miss the mark on what the intent of the component was meant to play out like. It doesn't instill the proper mindset and inevitably breaks the immersion of the game as both players and GM's alike have to decide not only what is appropriate for the game table but how their characters should feel. No game designer aims for this scenario, and it's certainly not a game mechanic I would ever want to play in, yet I've said before that I've found compelling stories based around similar concepts, so what's the difference?
The difference comes from allowing that internal conflict to flourish and become part of the character one is role-playing. This is accomplished by having variance in the mechanics showing a slipping slope of moral certainty, but also by granting benefits and drawbacks to a particular moral stance. In the world of Star Wars, wielding the power of the dark force was theoretically set to decay the Jedi's body, in the World of Darkness, monsters deprived of their humane side become more frightening and harder to blend in. In Exalted a character's virtues don't necessarily orient his moral compass but allow him to excel in the things that are backed by the strength of his convictions. These sorts of systems of flexibility combined with mechanical incentives and drawbacks allow games to survive horrific stories without breaking immersion and hopefully without making anyone so uncomfortable that they leave the game.
Ultimately that's what this is all about: the story. In basic literature we're taught that among the key conflicts is man vs. self. Horrific acts are stimulated by man vs. man conflict, but the character must resolve the implications of such acts in the theatre of his own mind. Perhaps in his struggles he will fail and become the villain. Or hopefully he will be raised by his own convictions and the aid of his allies to become a hero once again, thus making him that much greater.
As an advocate of balance in all things, I think that characters that accomplish great deeds should sometimes endure great hardships, both physical and emotional, else their greatness might become hollow. So maybe if you're a GM that has a strong repoiré with your players, maybe try going outside of everyone's comfort zone and see what the results are, they might surprise you.