The small GTPase Rab6 regulates retrograde membrane traffic from endosomes to the Golgi apparatus and from the Golgi to the endoplasmic reticulum (ER). We examined the role of a Rab6-binding protein, TMF/ARA160 (TATA element modulatory factor/androgen receptor-coactivator of 160 kDa), in this process. High-resolution immunofluorescence imaging revealed that TMF signal surrounded Rab6-positive Golgi structures and immunoelectron microscopy revealed that TMF is concentrated at the budding structures localized at the tips of cisternae. The knockdown of either TMF or Rab6 by RNA interference blocked retrograde transport of endocytosed Shiga toxin from early/recycling endosomes to the trans-Golgi network, causing missorting of the toxin to late endosomes/lysosomes. However, the TMF knockdown caused Rab6-dependent displacement of N-acetylgalactosaminyltransferase-2 (GalNAc-T2), but not beta1,4-galactosyltransferase (GalT), from the Golgi. Analyses using chimeric proteins, in which the cytoplasmic regions of GalNAc-T2 and GalT were exchanged, revealed that the cytoplasmic region of GalNAc-T2 plays a crucial role in its TMF-dependent Golgi retention. These observations suggest critical roles for TMF in two Rab6-dependent retrograde transport processes: one from endosomes to the Golgi and the other from the Golgi to the ER.