"Your distance is not due to something they have done, is it? To make you settle so far from them?" Ned knows that the children all grew together, saw no difference between the lot of them regardless of whether their name ended with a Stark or a Snow. He knows Robb and the others considered Jon as much a brother as they had Bran, though they had also been painfully aware of the differences of the lives they would lead based on their name.