This is the problem of how to sound when a character is relating what someone else said.

Make sure that you keep track of who is talking, when someone relates a story. Here is an extreme example, taken from the Sherlock Holmes story, The “Gloria Scott”. The story begins in Watson's voice, as usual, but quickly switches to Holmes' voice as Holmes takes over uninterrupted to tell Watson about his first case. We now have a story in a story). In the course of telling the story to Watson, Holmes relates how he was met at the station by one Victor Trevor. While they are traveling by dog-cart back to the Trevor family home, Victor begins to tell Holmes a story, so at that point Victor becomes the narrator (supplying the 'he said' attributions). (We now have a story in a story in a story) If not enough, Victor quotes dialogue between his father and the sailor Hudson, so we hear new voices:

[Victor talking to Holmes] “ ‘That evening there came what seemed to me to be a grand release, for Hudson told us that he was going to leave us. He walked into the dining-room as we sat after dinner and announced his intention in the thick voice of a half-drunken man. [Hudson talking to Victor and his father] “ ‘ “I’ve had enough of Norfolk,” said he. “I’ll run down to Mr. Beddoes in Hampshire. He’ll be as glad to see me as you were, I daresay.” [Victor's father talking to Hudson] “ ‘ “You’re not going away in an unkind spirit, Hudson, I hope,” said my father with a tameness which made my blood boil.

The quotes within quotes within quotes is Victor's father talking to a Hudson, as heard by Victor and being related by Victor to Holmes, who is relating it to Watson, who is relating it to you the reader.

As listener, what is important is that there are enough differences between voices so that it is clear who the narrator is. It is the author's duty to get back to the original speaker. In the Holmes story, Victor has to finish up (as listener, we hear Victor again), after which Holmes has to finish up (as listener, we hear Holmes again), and finally we return to Watson's voice ending the story. (The story could have ended with Holmes making the last statement, but in that case it would be clear from the context that he was talking to Watson.