tropic of confusion

I am having trouble keeping up what is happening in the story. I have always had a preconceived understanding that Tropic of Cancer is in reality just Milton's delusional sexual ramblings, but I don't know where i got that idea from, because this is a legitimate story with characters and something of a plot ( his stay in Paris). There are just so many characters and tangents and time period jumps that I'm am caught in a whirl win of confusion. The interesting thing about this is that overall it has not impaired my interest or  general understanding of the novel (i hope). Is anyone else feel a bit overwhelmed?...for lack of a better description