Are Domain-Specific Trade-Offs Undermining On-Device Language Models?