I have added a custom BBCode [pli] to produce a modified format <li> item. The only trouble is that it's adding extraneous <br> elements between each list item anytime I visually space the list elements in the actual post text. Is there a way I can tell it to NOT do this?

You could try to pre-empt it by removing the newlines after the closing </pli>

For example:

/\[pli](.*?)\[\/pli][\n\r]+/si


The [\n\r]+ at the end there will attempt to match all newlines following any closing [/pli] element.