You have understood one thing correctly: a preposition is combined with what follows it to form a prepositional phrase, and they form a single constituent in the sentence. Generally speaking, words in English govern (ie. control or specify) the words that come after them. In linguistics, we say that English is right-branching, meaning that new syntactic elements come after (to the right of, in writing) the elements that govern them.
Note that there are exceptions, such as adjectives, which precede the nouns that govern them. English is not exclusively right-branching, but it is predominantly right-branching.
But what does this have to do with prepositions? Well, just as a preposition governs the noun phrase that comes to its right, the preposition itself is governed by something to its left. And in many cases, that thing is a verb. English is full of idiomatic combinations of verb + preposition, where the verb requires a specific preposition to follow it, and anything else is an error. To take some obvious examples cribbed from other answers:
I converse with you. [Not to/at/of you]
They rely on the bus. [Not with/to/at the bus]
These combinations are highly idiomatic, meaning that the correct choice of preposition cannot be predicted simply by knowing the general meaning of the words involved. So the people who ask about what preposition follows a certain word are asking a reasonable and intelligent question. The choice of preposition very, very often depends on what came before it.