The context comes from Lehi's Dream found in I Nephi 8 and interpreted in Chapters 11 and 15:
From the standpoint of the Great and Spacious building it must be very comforting to point fingers at a band of grimy transients eagerly eating fruit from a tree while the finger-pointers believe only those things which are convenient at the time. Of course the residents of the building probably agree on one subject only: The people eating the fruit are intolerably zany. Left to themselves, the occupants of the building would scratch one another's eyes out.
The point of the dream isn't that the people eating the fruit were correct in everything they believed as they traversed the path (Lehi actually takes pains to point out that they didn't know what they were doing most of the time) but that they held on, mists of darkness and all, and they stayed on the path found the fruit and were blessed if they didn't fall away.
The gospel has always had this problem: As seen from the outside what is the glory of the Tree compared to the glory of that building? The Tree and fruit seem to have little going for them but the ability to make one happy. The building: the glory and praise of the world. In the end the building has nothing but destruction awaiting it. The tree still majestically stands.Ultimately, politics don't matter. Only the "love of God which sheddeth itself abroad in the hearts of the children of men." The tent pitched in the shade of that tree is big enough for all, even if few ever find it.