Try this trick if you have trouble remembering details about a character and keep looking back at your notes.

Pick a muscle that you can stretch or tighten up (a toe, finger, fist, etc.) without much movement. Focus on the character details that you keep forgetting, while at the same time tightening or stretching those muscles.

Whenever you are speaking that person's voice, tighten those muscles. This physical link will help you to recall the details. In a two person dialogue, you may find yourself alternating between two tensions. It's like using your left hand and right hand to talk to each other, but less noticeable. Once you can draw on that character automatically, you can stop using this crutch to remember the details.

If this works for you, fine. If not, ignore it.