Within a quantum virial expansion, we investigate theoretically the violation of universal thermodynamics for a strongly interacting unitary Fermi gas trapped in a harmonic potential. The violation is caused by the existence and anisotropy of the trapping potential and a finite-range of the two-body interaction. We calculate the second virial coefficient by solving a two-fermion problem in 3D uniform harmonic traps, as well as in anisotropic traps. In the unitarity limit, the universal value of the trapped second virial coefficient is 1/4. We discuss in detail the non-universal correction to the second virial coefficient and to the equation of state.