So you want to design a board game? Hooray! It's a wonderful, rewarding experience. Here's 5 ways to make it unique (and hopefully innovative), while remembering that is has to be fun to play.
1. Don't just clone other games.
If you think a deckbuilding game is fun, then play more deckbuilding games and decide what you can bring to the genre (if anything). You don't really want to make another clone of Dominion, perhaps what you really want to do is tweak or house-rule or re-theme that game. That's awesome! But if you want to make a unique game, dig deeper and find inspiration. You should avoid adding a paint job to an existing game. Even if you make improvements your game probably won't achieve much notice or replace a popular game. Most games use the best ideas of other games in new ways, so seek something new, or a new spin. Maybe be inspired by a movie or a book or a family story, and find mechanics to fit.
Most games exist in a pseudo state where the only thing limiting their design is your imagination. Don't think of the gameplay like a rigid set of rules that must remind you of other games. Your gameplay foundation should be more like gravity... there are ways to supercede gravity, but it still exists to prevent us from flying up into outer space.
Dune. Machinations and treachery and hoarding spice! There are some light area-control elements (from many other games) and variable player powers (from Cosmic Encounter, the designers' previous game), but the rich theme and player interaction, and combat system, was a revolution. The design team was inspired by ideas of political backstabbing, and one of their favorite books, Dune.
Unique ideas don't have to be epic. There's a game I saw recently that you play with a balloon and an iron-maiden style headpiece with spikes that goes around the balloon. You take turns slowly screwing the spikes deeper and deeper into the balloon until the balloon pops. This is pretty new, but it seems clearly inspired by Jenga and similar games. The game doesn't look fun to me, but it definitely evokes a real-life feeling of dread.
Evoking emotional reactions is one of the most powerful things a game can do. Lots of people make fun of Pretty Pretty Princess (a game where you dress up in plastic jewelry and tiaras). There's a reason that game has lasted: it's because some people find it fun.
Which leads us to:
2. Find true inspiration from your own life.
Games are more powerful when they are rooted in the human experience. I've read in game design forums many times "Should I pick a theme before or after I design the mechanics?" Those are both bad places to start. There are plenty of great games alreday published, more than you could ever play in your entire life. Why would you want to make "just another game?" If you want to design a game about pirates, stop thinking about pirates in terms of what they DO (plunder, hoard, sail around, fight) and start thinking about what YOU love about pirates. Was there a movie from your childhood that spoke to you? Do you love the sense of freedom pirates enjoy? For me, Merchants and Marauders is a good board game not because it perfectly captures the pirate "theme" (although it does admirably) but because it offers you a large amount of choices on your turn.
3. Game rules are like city ordinances; things get fun when you break them.
Once you are inspired, you need to not limit the game by the basic rules (or at least playtest the possibility of breaking them). I personally like when each player has unique game-breaking abilities, but there's also other ways to break rules. Sometimes entire games are about breaking rules (Fluxx is a game in which the victory goal of the game keeps changing). Some great ideas for games can pop up because of an idea to break another game, but I would caution against making a new game that doesn't inspire you emotionally.
4. Have fun designing it, and keep at it!
For me, a game is most fun when it's in its early stages. Playtesting games can be amazing because the initial inspiration that sparked the game hasn't been accidentally squashed yet, and it may still be possible to find some amazing way to break the game. I think part of the fun of Magic: The Gathering is that there are so many possible combinations of cards that players know it is still possible to come up with innovative decks, and uses for cards that will eventually get errata'ed. It's a glorious journey you are about to undertake, and you step up to the challenge and enjoy it.
It's really fun to add a bunch of new cards or rules and then try to "break it" or figure out ways to make characters or card combos overpowered. For that one perfect moment, your game became something more than just a game. It became a puzzle that you unlocked! Now you can fix the mistakes and test it again. If it isn't fun to test your game, it isn't fun to play it. And if you would prefer to play another game more than playing your game again, your game probably isn't ready for primetime. It should be AMAZING. If you designed the game with true passion, then you should love playing it. Most people will never like it as much as you (even if it's a great game) because it came from within you-- it's your baby, so cherish it!
5. Pay attention to your playtesters, but don't listen to them too much.
Playtesters will be very supportive and offer all kinds of helpful suggestions. Write it all down and look over the notes from time to time. But don't listen to most of it. First time playtesters don't necessarily know what type of experience your game strives to convey, so they will automatically equate the experience of playing it with other games they have played. This is why my game Chaosmos gets compared to Clue and and Capture the Flag and Battlestar Galactica. Yes, those are VERY different games. No one really knows what Chaosmos is like because it's something new. What you should do is pay attention to what the playtesters enjoy about the game and how they subconsciously try to make it more fun. If they play a rule wrong and have more fun than a different playtest group, figure out WHY it was more fun to play it wrong. Interpret what they are saying in the context of their misunderstanding of the game.