Sperm whale (Physeter macrocephalus) populations were expected to rebuild following the end of commercial whaling. We document the decline of the population in the eastern Caribbean by tracing demographic changes of well-studied social units. We address hypotheses that, over a ten-year period of dedicated effort (2005–2015), unit size, numbers of calves and/or calving rates have each declined. Across 16 units, the number of adults decreased in 12 units, increased in two, and showed no change in two. The number of adults per unit decreased at -0.195 individuals/yr (95% CI: -0.080 to -0.310; P = 0.001). The number of calves also declined, but the decline was not significant. This negative trend of -4.5% per year in unit size started in about 2010, with numbers being fairly stable until then. There are several natural and anthropogenic threats, but no well-substantiated cause for the decline.