The aim of the present study was to investigate the molecular mechanism associated with the traditional Chinese medicine formula Gui Zhu Yi Kun formula (GZYKF), in the treatment of polycystic ovary syndrome (PCOS). In this study, granulosa cells (GCs) of rats with PCOS were cultured and treated with testosterone propionate (TP) alone or with serum from rats treated with different doses of GZYKF. The effect of TP on cell growth was assayed using the MTT method. Expression levels of Beclin-1, light chain (LC)3, mechanistic target of rapamycin (mTOR), tumor suppressor p53 (p53), adenosine monophosphate-activated protein kinase (AMPK), sestrin2 and tuberous sclerosis protein 1/2 were evaluated using quantitative polymerase chain reaction and western blotting. It was demonstrated that TP increased the expression of Beclin-1 and LC3, whereas GZYKF significantly decreased the TP-induced expression of Beclin-1 (P<0.01). Additionally, GCs treated with GZYKF exhibited significant increases in mTOR, phosphorylated mTOR and AMPKα expression levels, and significant reductions in p53 and sestrin2 expression levels were observed. In conclusion, the findings of the present study suggest that a reduction in ovarian GCs in rats with PCOS may be associated with GC autophagy. Furthermore, the effects of GZYKF in mediating the p53/AMPK pathway may inhibit GC autophagy, which suggests a possible novel mechanism underlying the treatment of PCOS with GZYKF.