Prosody
Prosody is the part of spoken language that are properties of larger-than-single-phoneme units (you'll often see terms like 'non-segmental' / larger than single segments).
Prosody is usually categorized as part of phonetics, though it has tendrils in more than that.
This tends to focus on syllables and the way we lay intonation (in non-tonal languages), stress, and rhythm into them.
Pauses do not fit that description directly, but because they are sort of the whitespace around the content, it is typically considered part of prosody as well - the sound equivalent of "this sentences changes meaning depending on where you put the comma"
People argue whether it is part of rhythm or not - perhaps in part because this while this can clearly be the case in rehearsed speech, in spontaneous speech it is often primarily hesitation, which has little to no contentful information. Also, long pauses are often none of the above.
By that description, pauses are not the most natural inclusion, but
Things like pauses could be considered prosodic boundaries, .
Prosodic unit
'prosodic unit' is a generalized term for something that can be independently pronounced,
In a lot of languages, they are consistently pronounced with the same patterns in pitch, rhythm, and tempo, and you can often tell them apart by consistent patterns in pitch and tempo, and where people choose to pause for breath and reset that pitch and tempo.
It generally refers to splitting of phrases, clauses, and sentences,
and also used to categorize them (in which contexts/ways?).
You can also see differences between prepared and natural speech
in terms of how much content each prosodic unit carries - generally fewer (often one) lexical nouns per prosodic unit,
and if the "um"s are semantically empty.