Pay Time off in leave at final pay

Hi, I would like to have a question about how to pay out time off in lieu.

I have an employee which is a fulltimer who works 38 hours per week. He isn't paid with overtime rate in public holiday. I pay him with normal rate and he will receive DIL for the holiday he worked.

Now he resigns and I have to prepare the final pay which pay out any unsused annual leave and DIL that he hasn't taken time off for.

But if I pay the DIL in his final pay, doesn't that mean I paid him twice for 1 public holiday?

Could anyone please advise what's wrong in this process?

Thank you so much.