You'll want to create custom BBCode tags in your forum.
In the admin section under Forum, there's a Custom BBCode option. You will want to create 3 rules - one for each item
Rule 1: (converts the [list] tag)Tag Name: List open
Regular Expression Search String: /\[list]/i
Replacement test: <ul>
Rule 2: (converts the [/list] tag)Tag Name: List close
Regular Expression Search String: /\[\/list]/i
Replacement test: </ul>
Rule 3: (converts the [*] tag)Tag Name: List items
Regular Expression Search String: /\[\*]/i
Replacement test: </li>
That should do it. Also, you need to add a description to each rule as well.
As an answer to your special note: For some reason li tags don't need closing tags - and this is according to the W3C standard for html