Even if this issue has been fixed for a while in opencart, some people stil seem to have problems with disabling the product count next to the category name in the opencart left category menu, on the top menu's dropdown and on the subcategories from within a category's page.
In the most recent versions of opencart there is one simple setting to disable the product count in all those instaces, the setting can be found in admin > system > settings. Edit the main store, go to the option tab and the first setting below the products heading is category product count, set it to no and you easyly solved your issue.
I older versions of opencart the left category menu has its own settings for product count, so apart from the change at system settings you will also need to edit the category module in admin > extension > modules and for each added category module set the product count to off, again only if you have an older version of opencart and you can see this setting.
There are also some 1.5.x versions of opencart that dont have the product count setting at all. If you are in this situation, you can still easyly solve the problem by installing one of the free mods available on the opencart extension directory. Here you can find some free extensions that will do that: http://www.opencart.com/index.php?route=extension/extension&filter_license=0&filter_search=product%20count
I hope this will help you sove the issue you are having with the category product count, be it a speed issue or just an aesthetics issue.