Gravitational instability is an instability in the dynamics of a system instigating changes stemming from characteristics of the gravitational field. A simple example is an object residing at a L1 point between two co-orbiting bodies, which, if for any reason moves slightly closer to one of the two, will fall away from the point.
The term is used for matter spread out, such as dark matter, clouds, or disks, referring to situations where any occurrence of extra density receives positive feedback. It is a consideration in the formation of anything that gathers matter, i.e., galaxy clusters, galaxies, stars, and planets. It can be prevented by sufficient relative velocity, or by a force counteracting gravity, such as pressure.
The gravitational instability model, referring to protoplanetary disks is of interest regarding planet formation. Secular gravitational instability (secular GI) refers to types of instability that occur over long periods of time, i.e., from situations in orbits where overall gravitational force is periodically different, leading to slight changes that grow over time.