I order to use the filter in categories and products you must first insert them (similar to how you do it for options or attributes).
In your admin navigate to catalog > Filters. Since you have no filter yet, click Insert.I will use as an example a color filter.
At first you will be asked to enter a filter group name, which in this case will be 'Colors', next is the sort order which everyone knows what it is for.
Below the filter group name and sort order is the actual filters form. Click on the 'add filter' button and a new row will appear. The row has 3 columns, the first one is the 'Filter name', then the sort order and on the third a remove button(obviously to remove the filter if you wish to). So in filter name I will insert 'Blue', and sort order 1. Then again I click on add filter (as I want to insert more possible colors), now I inser 'Red' and sort order 2.You can click add filter as many time as you wish and insert as many filters as you wish(of course relevant to this filter group > colors).Now I hit save and go back to the catalog > filters page. Now on the filters page I can see the 'Color' filter group which has inside 2 filters (2 colors), and of course you can edit that filter group and add any new filters at any time.
So that is the way you add the filter groups and filters (you can add as many as you need from the start), next you need to link those filters to products.
Adding Filters To Products
You can add the filters to products by editting the targeted product, and on the product edit page click on the links tab. As you can see, in the new opencart 1.5.5 a new filters area was added under the links tab. Adding filters to a products works trough autocomplete (similar to how you add related products). On the filters row, in the input field type in the name of the filter you wish to add, then it will show up in a dropdown and once you click on it it will be added to that product. Do not try to type in the name of the filter group as nothing will appear (in my color example > if you type in color, the colors will not appear in the dropdown, you need to type in directly the name of the filter > ex: Red). In my optinion would have been better to type in the name of the filter group and then select from the dropdown the actual filters.
Mention: You can add multiple filters to the products
Adding Filters to Categories
Example: You have a product called 'Cool T-shirt' in the category 'Shirts'.
Even if you edited the 'Cool T-shirt' product and in the links tab you added it the filter 'Color > Red' like above , when you navigate to the 'Shirts' category you wont be able to see the filter 'Color > Red' because in order for that filter to appear on the category, it also needs to be added to the category.
So in catalog > categories edit the 'Shirts' category , click on the data tab and there you will see a Filters section same as the one in the product edit page. Here you should add all filters you want to display on that category's page(Example: Even if you just added the filter 'Color > Red' to the 'Cool T-shirt' product, here in the category you should add al filters you might need in that category - So if in that category you will also have a blue tshirt you should also add the 'Color > Blue' filter and any other filter you might need).
Displaying the Filters module
Now that you created filters, you added the to both prducts and categories you can finally display the Filter Module in the store.
In admin navigate to extension > modules. In the list locate the 'Filter' module and click edit (click install first if you didnt allready). On the filters module page click add module. In the first column you select the layout so select 'Category' , next position (best on column left or right), on the third column is the status (make sure is enabled) and last the sort order (set it as you see fits).You could click add module again and also add it to the product layout but would only display on product pages to which you got trough from a category page that has filters (In the example above: If you went to the 'Shirts' category and then to the 'Cool T-shirt' product you would be able to se the filters on the product's page , but if you went to the 'Cool T-shirt' from the featured/latest module or search page, you would not see the filters module).
Mention: The filter module (or 'Refine Search' module) will only display on the pages of the categories that have filters.