There may be a lot going on . . . as I may have mentioned in a previous post, we cannot control someone else's behaviour . . . what I would suggest is for you to consider some of the following:
1. did he approve of you and his Mom getting together or did he feel that his position as the male in his Mom's life has been displaced? Depending on how long he has been the focus of his Mom's life may well account for his attitude towards you? Perhaps he does not feel that you are "worthy" of his Mom and, quite frankly, and this is what my brother was at 17, perhaps he is just a pain in the a*& because of his age, not being happy, not having control, etc. He is 17 . . .will he be employable and out of the house or is Mom intent on keeping him at home . . . to a great degree, Mom is the key here . . .
2. right now, the Son knows how to push your buttons . . . you cannot control what he does but you can control how you react . . .difficult, yes, by all means . . . when I stopped reacting to my brother's sarcasm and put downs, it was less fun . . . my favourite tactic was to pretend that I did not hear him and continuously say "oh, sorry, I wasn't paying attention to what you were saying" and if he repeated it I would say "oh, okay, fine". Eventually, the game became boring for my brother and he stopped. Of course, this does not address the issue of forging a positive relationship. I would suggest that you approach your wife and say something along the line of "You know, I would like to have a better relationship with XXXX. I am not sure how to go about doing that. Do you think that he feels that I am trying to replace his father or his position with you? What would you suggest that I try to do?" -- - now, I will warn you, you may make a million attempts and they all get rebuffed, but just take it in stride . . .yeah, I know that it is difficult but start with baby steps . . . and if the step son does not respond, well, think about maybe how you were at 17 . . . change is difficult especially for 17 year olds whose brains are still forming (up to around 25), who may not have the critical processing skills of the life experiences to process what is going on, etc.
3. Given that he does not show respect for you and your wife can't see it, then I would suggest the indirect approach . . . perhaps the three of you could establish some "house rules" as to acceptable behaviour . . . .but, instead of discussing what is wrong, look at what is good . . . . oh, it might be very very small things on everyone's part, but then see how to build from the small things . . . .perhaps as simple as you recognizing that among all the crap that is directed towards you, you recognize that the Son has done something good for his Mom and that you noticed it . . . it is a game but it is a game of reenforcing the positive.
4. As for you, don't try so hard . . . you are healing yourself right now; don't play into the game that the Son has established; if he says something nasty, don't feed into it ., . .your stepson is a lot like dealing with anxiety . . . if we accept that anxiety makes the rules, then anxiety controls us but the way to manage anxiety is to put it up against reality . . . if we try to control reality, we usually lose; if we let it wash over us and manage it, we usually make advances . . .
Look, none of this is easy at the very best of times . . . but do what you can without feeding the situation and keep working on yourself . . . it is nasty when someone takes out their own frustrations and disappointments, etc. on someone else especially when it is a close environment, but you can manage your reactions . . . if nothing else, convert it into a game of his action and your non-reaction if just for the time being until you get a foothold . . .
Sorry I couldn't be of more help . . . take care, kc