Seed size shapes plant evolution and ecosystems, and may be driven by plant size and architecture, dispersers, habitat and insularity. How these factors influence the evolution of giant seeds is unclear, as are the rate of evolution and the biogeographical consequences of giant seeds. We generated DNA and seed size data for the palm tribe Borasseae (Arecaceae) and its relatives, which show a wide diversity in seed size and include the double coconut (Lodoicea maldivica), the largest seed in the world. We inferred their phylogeny, dispersal history and rates of change in seed size, and evaluated the possible influence of plant size, inflorescence branching, habitat and insularity on these changes. Large seeds were involved in 10 oceanic dispersals. Following theoretical predictions, we found that: taller plants with fewer-branched inflorescences produced larger seeds; seed size tended to evolve faster on islands (except Madagascar); and seeds of shade-loving Borasseae tended to be larger. Plant size and inflorescence branching may constrain seed size in Borasseae and their relatives. The possible roles of insularity, habitat and dispersers are difficult to disentangle. Evolutionary contingencies better explain the gigantism of the double coconut than unusually high rates of seed size increase.