The driving force of the dynamical system can be decomposed into the gradient of a potential landscape and curl flux (current). The fluctuation-dissipation theorem (FDT) is often applied to near equilibrium systems with detailed balance. The response due to a small perturbation can be expressed by a spontaneous fluctuation. For non-equilibrium systems, we derived a generalized FDT that the response function is composed of two parts: (1) a spontaneous correlation representing the relaxation which is present in the near equilibrium systems with detailed balance and (2) a correlation related to the persistence of the curl flux in steady state, which is also in part linked to a internal curvature of a gauge field. The generalized FDT is also related to the fluctuation theorem. In the equal time limit, the generalized FDT naturally leads to non-equilibrium thermodynamics where the entropy production rate can be decomposed into spontaneous relaxation driven by gradient force and house keeping contribution driven by the non-zero flux that sustains the non-equilibrium environment and breaks the detailed balance. On any particular path, the medium heat dissipation due to the non-zero curl flux is analogous to the Wilson lines of an Abelian gauge theory.