it's probably because his mother was not as attuned to him as she should've been when he was a baby/small child and he is still resentful about it but probably doesn't realize that's why he's resentful and so he attributed it to you