Many real-world complex systems are characterized by interactions in groups that change in time. Current temporal network approaches, however, are unable to describe group dynamics, as they are based on pairwise interactions only. Here, we use time-varying hypergraphs to describe such systems, and we introduce a framework based on higher-order correlations to characterize their temporal organization. We analyze various social systems, finding that groups of different sizes have typical patterns of long-range temporal correlations. Moreover, our method reveals the presence of non-trivial temporal interdependencies between different group sizes. We introduce a model of temporal hypergraphs with non-Markovian group interactions, which reveals complex memory as a fundamental mechanism underlying the pattern in the data.