Exponential convergence towards consensus for non-symmetric linear first-order systems in finite and infinite dimensions