We study lepton flavor violating (LFV) τ and B decays in models with heavy neutrinos to constrain the mixing matrix parameters UτN. We find that the best current constraints when the heavy neutrinos are purely left handed come from LFV radiative τ decay modes. To obtain competitive constraints in LFV B decay, it is necessary to probe b → X sτ±e± at the 10-7 level. When the heavy neutrinos have both left- and right-handed couplings, the mixing parameters can be constrained by studying LFV B decay modes and LFV τ decay into three charged leptons. We find that the branching ratios B(τ± → ℓ1 ± ℓ2 ± ℓ3 ±), B(Bs → τ± e±) and B(b → Xs ℓ1 ± ℓ2 ±) need to be probed at the 10-8 level in order to constrain the mixing parameters beyond what is known from unitarity.