Small rift with neighbours - who do you think is in the wrong?

It's clear that he's the one who's wrong. You should ask him why he was late. If he has a good excuse, then forget about this whole situation & continue helping him with his English (and of course, tell him to leave a notice next time he's gonna be late); but if he doesn't, then tell him that he wasn't supposed to be late & that you weren't obliged to wait for him for about an hour and a half!