Hi All,

I’m developing a Python script to fetch data from Reddit based on a given keyword for market research on a specific topic. The script retrieves the top 3 subreddits related to the keyword and extracts 50 posts from each, totaling 150 posts. I want to generate a summary that highlights key insights, including the issues users are currently facing, the methods they have tried, what worked, what didn’t work, and their major pain points. However, when using the OpenAI API for summarization, I hit token limits. When I break the data into chunks, the summaries don’t effectively capture the overall essence. How can I efficiently summarize large datasets while maintaining coherence and extracting meaningful insights?”