Evolution of complexity in human languages has been vigorously debated, including the proposal that complexity can build in small, isolated populations but is often lost in situations of language contact. If it is generally true that small, isolated languages can build morphological complexity over time, but complexity tends to be lost in situations of language contact, then we should find that forms of language complexity that have evolved multiple times will tend to be associated with population size, isolation, and language age. We test this hypothesis by focusing on one particular form of morphological complexity, polysynthesis, where words built from many parts embody complex phrases. By assembling a global database of polysynthetic languages and conducting phylospatial analyses, we show that languages with highly complex word morphology are more likely to have small population sizes, less likely to occur with many other languages in direct contact, and have a greater tendency to be on long phylogenetically isolated lineages. These findings are consistent with the hypothesis that languages that evolve in isolation for long periods may be more likely to accrue morphological complexity. Polysynthetic languages also tend to have higher levels of endangerment. Our results provide phylogenetically informed evidence that one particular form of complex language morphology is more likely to occur in small, isolated languages and is prone to loss in contact.