I believe something went terribly wrong by your side. I just checked on my almost 10 years old i7-3770 3.5Mhz with 4 cores 8 threads, the largest RU data set was processed in "threads" mode with debug info for 314 seconds, "top" mode took 231seconds . I believe you didn't try raw dataset, in news/categories/threads/top however it took 2120s for "top" processing 35k files to merge into top topics both RU and EN articles for each category (near the edge). Languages, News and Categories have linear equal speed and "categories" on RU dataset took 78s . Are you sure all of your cores were free for my tgnews? I understand, that python is not the fastest one and an edge is somewhere near to the size of your dataset in threads and top mode (for production needs there's a great possibility for the scalability with databases and load balancers) , but I believe in this case something went from from the beginning. I will highly appreciate your attention to this issue!

I see this but do not understand how could it happen =( I also see the problem with categories in RU dataset and it really exists but I do not understand how timeout could happen with smaller EN dataset that is smaller. Processing speed is linear and very far from the edge at least for news/categories/languages and enough to process threads and top for your dataset. That is why I wonder how could it happen.