Inactive activities created as queue items when a Case is added to a queue

We are working with Fax/Letter/Case which all have the setting "Automatically move records to owners default Queue" 

The process we follow is to assign an activity or Case to a Team which then adds the Case as a Queue item in the teams default Queue.

This works very well for us except in the following scenario.

We have a QA team with a default queue of QA.

We create a letter activity for an incoming letter, this is converted to a Case and the letter activity is "completed".  So we now have an active Case with an inactive letter activity attached.  No problem.

We then assign the Case to the QA team, and it appears as a Queue item in the QA queue. Nice.  Unfortunately the inactive letter activity attached to it is also created as a queue item in the QA queue.

How can we prevent this happening for completed/inactive activities.

Thanks in anticipation