I tend to start on medium, since most games refer to it as "normal" difficulty, so why not start off by playing the game in its normal mode? To me, medium/normal is like a default difficulty of sorts, almost like the programmer's personal recommendation for the difficulty of the game, so I generally just go with that. If I find the game fun enough to play through again, I jump up to hard mode, or any other difficulty beyond hard.
Two games stand out as the exception to this rule: Doom and Ikaruga. Doom I play on Ultra-Violence, which is the second hardest difficulty mode behind Nightmare, and anyone who has played Doom will tell you that Nightmare is absolutely retarded - the default difficulty is Hurt Me Plenty, which is the third hardest difficulty and sits just behind Ultra-Violence. As for Ikaruga, I err towards easy, since Ikaruga as a whole is hard as a brick sandwich regardless of your choice in difficulty, and the medium and (God forbid) hard modes only make an already hard game Nightmare-level crazy.