I understand your brother's frustration, but the reality of the situation is this. Unless he forms a long term relationship with the mother of his child, there will always be other men in his child's life. He can either accept that, or make things more difficult for everyone (including his son). Look, the situation is not ideal, but he picked her (the mother).
And I also disagree that he is not the stepdad. If he lived with the child for three years, and continues to play an active (and, by your description, loving) role in your nephew's life, then he is the stepdad. He would still be living with J if it was up to him.
I know this is hard emotionally, but your nephew will remember who was always there for him, even if mom's boyfriends change. Your brother will be the constant, stable force in his son's life. He can live up to that by letting his son know that it is okay to love his stepfather. I wish my daughters' "mother" would have given them the same message. Instead, they are in turmoil about their feelings about me and the loyalty they feel to their "mother".
After all, this should be about what is best for J, and not about hurt feelings or wounded egos in the adults involved.
My family has always been a mixed family. Both my parents were married prior to getting married to each other. My dad had four children (one deceased) before marrying my mom - ranging from 14 years old to 6 years old. My parents ended up having two more afterwards (I'm the baby) and none of us were treated differently. My dad raised them to believe there was enough love for two moms, but their biological mom wanted no part of that, so they always called my mom by her name.
My brother is also a step-dad. His first marriage, his wife had a 1 year old baby when they married. She wasn't his, but he treated her as though she was. She didn't know my brother wasn't her biological dad because they didn't want her to know (I have my own issues with this, but alas.. not my dealings). Then, when she turned 16, my brother and her mom divorced and all of a sudden her mom tells her my brother's not her dad and she can't call him dad, nor can she go see him. My brother ended up with custody of his other two kids but he didn't even get visitation for my niece. Even now he considers her his daughter, but they haven't spoken in probably 8 years. She has since found her biological dad and brothers, and no longer considers our family to be part of her family.
When my brother remarried, he already had teenagers (19, 16, and 13). His new wife had a daughter before she married my brother. Her first daughter is still my brother's daughter, even if it's not by blood. He's also had another daughter since then, and not a single one of them is treated any better or worse than the others.
I guess my long-winded post is just to say that family is what you make of it, and titles don't really matter to kids in the long run. As long as they are loved and taken care of, it doesn't matter if they have a dad, daddy, pop, father, etc.
While I agree that adoption is a different issue, I think at the boy's age and level of development, having a "daddy" in the house, even a not-so-hot daddy is probably better than not. "Just the guy mom lives with," is going to be a harsh reality soon enough (from what it appears).
My concern is more about what the arguments about the name issue is doing to the child. Bringing him into the argument (and even discussing the issue in front of him, or where he can hear) is only going to be traumatic (and because kids always do, he'll find a way to blame himself). We can argue all day about what the rule "should be," but that's rather pointless, as it's already been said that the mother is not willing to negotiate the issue - so it's probably best not to worry about the semantics, and instead focus on the relationship.
It would be lovely if all the adults could agree and cooperate (which is why I recommended family counseling), but ultimately it's the child's perspective I'm considering. Making a fuss over what the child calls the stepdad (he's functioning at least part-time as a stepdad), is only going to make the child feel that one way or another he is hurting or betraying someone - blaming himself for a situation that the adults set up.
Before he met me, my husband lived with his then-fiance for four years, and treated her three kids as his. When they met, she was newly divorced, and pregnant with the third child. Long story, but in a nutshell, she and two of her three children were deaf, and in some deaf subcultures there's a major push not to date/marry "outsiders" she dumped hubby for a deaf man as her friends had pushed her for years. For several months after the split, my husband, even though he was not bio-dad or stepdad, still shared informal custody. He would take the kids on weekends and continue to buy them gifts and clothing, and generally continued to act as a father.
... then the "daddy" issue came up. The ex's new live-in wasn't happy that the kids were still calling my husband "daddy," so the mother suddenly, and with no notice put a stop to my husband seeing the kids. He had no legal rights to see them, so there wasn't anything he could do.
I met my husband several months later, and we married 13 months after our first date. Ironically, shortly after we married, his ex contacted my husband offering to let him see the kids again (emotional blackmail really, as she told him how much the kids missed him... although according to the rumor mill, she was impressed by the high-paying job I had at the time and thought we could buy the kids stuff she couldn't afford). Luckily, my husband's not a moron, and he asked me what he should do. I know that in his heart, he really wanted to reestablish contact with those kids, so I told him I would support anything he chose to do, but that I didn't recommend it. After two years without seeing him, I thought it just wasn't worth the risk to the emotions of my husband and the children. It would be too traumatic for the children (and my husband too), if they were allowed to reestablish contact with my husband only for the mother to snatch it away again when it was inconvenient - and with my husband having no legal rights there would be nothing we could do about it. It was a difficult decision, but my husband agreed.
My husband still feels like "Dad" to those kids, even though he only did the "job" for four years (but never had any legal "right" to the title). If his ex had not been so worried about "confusing" the children by allowing them to call two men "Dad," my husband could have maintained a relationship with the kids, and their lives would be richer for it. "Confusing" the kids wasn't the real issue, the real issue was "hurting the feelings" of the "new Dad" (who as it turns out, didn't even stick it out for two years).
It sucks when children have to live in unstable situations, but it's vital that the adults in their lives do what they can to create as much stability as possible, which is why I think family counseling is so important in a situation like this.
