"And if you had done as the letter asked, then you would have given some sort of reward?" That bothers him more now that it once would have. He'd made worse deals. So he's not worried about what it says about the people who left the letter; they've already shown how far they're willing to go for whatever their plans are. He's more worried about what it's doing to Moana to have to do something like this.

"You shouldn't feel bad about not doing what the letter asked."