Some really cool answers.
Now here's a question, since I'm done with my big schpeel on story in games and I like discussions on this kind of stuff:
I've noticed that a lot of people comment on wanting more story in games. Do you feel that many games don't have very good writing or implement story poorly, or do you want more story-centric games?
Providing some examples of what games have done it well or poorly would be good for discussion so we can critique what's been done.
Well for instance if you look at the simplest games there is some vaguest attempt to include some kind of backstory which serves little purpose as it doesn't enhance the gameplay. Remakes of Atari classics and old school games (like Tetris) try to invent a reasoning as to why you are playing but the characterisation of what is essentially blocks personally puts me off- especially if the style isn't too good to begin with. On the flip side is highly stylised games like Wip3out (and others). The game is sooo coool, and reading the well written manual text about the future of the industrialised world really does implant the stylistic vision of the world in your head, enhancing the games mood- especially combined with its distinct aesthetic.
In the PSX reincarnation of Pacman the game was re-orientated into a whole different game retaining only the main character in a genre where plot plays an important role. This is a valid use of plot since the whole gameplay mechanics changed (moving through a world environment). This brings to mind games where plot is the central gameplay provider in the form of quests and travelling (like FF), and these games remain the very memorable since their plot events have a key impact.
Do you remember Aeris death?

The role that story plays in the game is really dependent on what you want the core experience to be. If the whole point of the game is physics puzzles, a strong emphasis on story during the game would likely get in the way. An explanation of why you are doing silly physics puzzles is a good idea, but aside from that the player might not need any more incentive to continue playing than simply advancing though each stage and improving their time/score.
I think unless that reasoning is awesome then I need not care either. If the game is trying to set a specific setting by the use of story then it is a reasonable allowance. Often through the use of silly characterisations or stereotypes along with a commonly recognised theme you can subtly hint at some undercurrent of whatever so if its a party game, the plot is essentially Pickup & Play. Usually the problem with plot seems to be when it tries to rationalise ludicrous gameplay.
And there are of course some games that are all about telling a story, where developing a strong milieu, compelling characters, and engaging plot are critical. But it really depends on what the core experience of the game is. Thinking of it in terms of what we want the player to experience is important to putting the right emphasis in which devices you are going to use to entertain or teach, such as story, graphics, sound, gameplay, and everything else that makes a game.
Half-Life 2's story was there simply to support the pacing of the gameplay. You got new weapons, vehicles, and environments just before you would have started getting bored of them, and they were introduced in such a way so that you'll know how to solve problems in the future (the Zelda games do this really well too, everything you learn in the dungeon and the items you get are so you can defeat the boss at the end). That is what they focused on, and they used the story and settings to make those transitions from gameplay mechanic to mechanic make more sense. That pacing of the gameplay is what gave me those strong emotional responses while playing the game: that warm fuzzy when I finally got to take down that chopper with a newly acquired add-on to my air boat, after it had been shooting at me for the past hour was a welcome experience. As for the story, I didn't really care that Eli got captured, except that I thought, "Sweet, I'll get to go to the Citadel and blow more stuff up." The core experience was really about combat and using the physics in clever was to defeat foes or advance. Everything else was just to support it.
All the ingredients that go into a good game seem to be good components. Good story, graphics, sound, and gameplay help achieve that fulfilling experience and increase immersion.
"Chrono Trigger was supervised by a group referred to as "The Dream Team", consisting of Hironobu Sakaguchi (producer of the Final Fantasy series), Yuji Horii (director of the Dragon Quest games), character designer Akira Toriyama (of Dragon Ball and Dragon Quest fame), venerable producer Kazuhiko Aoki, and Nobuo Uematsu (music composer of Final Fantasy fame)."
Chrono Trigger is considered by many as a classic game, and each department of that game was led by a notable.
I disagree with your analysis of Half Life that the plot took a backstage to the action and was only a simple device for advancing gameplay. Half Life 1 was an innovator in narration using no cutscenes in which the controls where locked or camera moved away from the player (no cinematics) and no time discontinuities. This limits the amount of backstory you can find out, but I guess Valve expects people to play Half Life 1. Also there is a secret tramp Vortigaunt who makes future predictions.
The plot if anything in Half Life is taking central stage with the saga of the G-Man playing out. Certainly the gameplay is an amazing experience and it is a very well done polished game. But to say the plot is insignificant is a faulty analysis in my opinion considering the amount of effort that seems to have gone into the story writing there. Notice that the tutorial sections are integrated into the gameplay itself without breaking the third wall in order to keep the immersion there.
I can't comment on Zelda since I never really played it, but indeed storylines are quest givers when they exist- otherwise these commonly remixed quests (they are essentially all the same gameplay mechanic with some minor variations) would just be boring and dull. When its a immersive virtual reality setting seemingly real events are happening... this is when the adrenalin starts to pump and it becomes thrilling. Would you find some cubes damaging you, as entertaining as zombies eating you?
I've been working on a project that I originally intended to be a series of video games. After about the first year into the project, I realized that what I was creating would be better experienced as a show, rather than a game. It was a shame because there where some events in the story that would have a very strong emotional impact if you were playing the main character, but most of what I created was humorous dialog and interaction between the five main characters. I started the project with the purpose of creating characters that you would love to watch interact with each other, create unusual circumstances that they would have to deal with, and in short create a really good story.
It seemed that it just wouldn't work out as an interactive experience. Yes, at first it was to create interesting gameplay scenarios, but the story evolved into something else. The story was too character driven to become a video game with our current knowledge of devices we can use to make in an interactive experience. We didn't want to use cinematics, and to make the story work we had to use a lot of them. We also didn't know how to handle character development in an interactive way, since the focus of the story is how that main character changes. The way the project evolved dictated that it was supposed to be a show, and not a game.
Yes well, film has been around for a century and as such has perfected and developed to a high degree its devices. Theater has been around much longer. Story telling even more so. Too many times, trying to mix gameplay and movie just ends up being interactive video (which is dull!). Just because games are young, and the techniques to use story telling aren't there (as well as all the technology!), this doesn't mean that it shouldn't exist. Games have the potential to be a much more powerful medium encompassing story telling, visual art, music, interaction and motion video to create an encompassing engaging reality. Creating an attachment with the main player and the main hero in which feedback from the game goes to the player is interesting. Some RPGs classically have had silent protagonists for this reason, even with empty dialog where the player can fill in their own gaps (Chrono Trigger, Suikoden, ...).
As for your game, maybe it just didn't need objectives? Like The Sims for example.
A lot of games do start out as stories. Tim Schafer (Psyconauts, Full Throttle, Monkey Island), says that "games are wish fulfillment". His process is to make a really cool world, and then create the coolest person in it; that is the person that you are going to play. Then the story is created so that it evolves around you. The core experience to him is wish fulfillment. For my story, it was different. I like making things that "you can enjoy with you friends and family at the same time". That project I spoke of earlier sort of did that, in that it's fun to watch and multiplayer was planed, but to really stick to the core value of what we want people to experience, it had to be a show, because that is something that everyone can enjoy together. Even though we currently believe that it's not going to be a video game, our process still worked and it's going to be a better as a show than a game.
In short, you as a game designer are trying to communicate something for the player to experience. Story isn't everything. Graphics aren't everything. Heck, even gameplay isn't everything. Final Fantasy has the most boring gameplay of any series I know. But all of that combat and exploring is just a bridge to the wonderfully detailed world and cinematics--the device they primarily used to tell the story. So, decide on what you want the player to do: explore, fly, blow stuff up, rhythm exercises, quest with friends, race, do silly things with a Wii-mote infront of your family, whatever. Then realize that everything else that you do, from the gameplay, visual representation, and story are all part of that communication and aiding in the core experience. If any of those things distract from the core experience, then it needs to be taken down a few notches. In the case of my project, the gameplay interfered with the story and so we chose to dump the gameplay.
Thinking about games in this way also helps you creatively. Don't think of games that you want to create as platformers, FPS, RTS, RPG, and so on. Think of what you want the player to experience--preferably something that they can't experience in real life and especially something that isn't available in the gaming market today--and then use story, and everything else, in such a way that will support it.
I don't think theres any ABC to constructive game design as with most design. All these things needs to be considered in their interweaving details. Genre classification is not as evil as people always make out to be- there are a few well established genre's out there with a body of knowledge that has been developed incrementally over the years and to start from that is not stupid. All the new ideas are simply just remixes of old ideas. It also helps the gameplay designer/developer who can examine the task analytically starting from a genre.