I'll give it a try.
Every rigger has the responsibility of declaring airworthiness or non-airworthiness when asked to pack a rig. But his decision is not binding on any other rigger. The owner is free to find someone else to do the work. Caveat emptor! I don't think this is what you are asking though. I think you are asking who can permanently ground a TSO'd component.
And I think the answer is "no one." Everything is repairable or replaceable, including TSO markings. When a manufacturer or other expert says something is unairworthy, they are really saying it is economically unrepairable. How should we convey "economically unrepairable" to the owner? As important, how should we convey it to anyone else who comes into possession? The most concise way is to obliterate the TSO marking, by removing it or by writing "unairworthy" or "condemned" or similar on the label, words that are technically not true because it could be made airworthy if time and money were no object. I'm okay with that, but I'm open to alternatives.
It still belongs to the owner, though, and he should get it back in exchange for the agreed compensation for inspection, shipping, etc.
--Mark