Strings Are Immutable

You can't actually "modify" a string. Every time you do something that fundamentally alters even a single character in the string, you've actually created a new string. This prevents some awkward behavior that keeps certain object-oriented behaviors from happening compared to if you wrote your own string class. I thought I'd clarify that, since many people don't actually realize that they're not "modifying" strings.

Many Functions Are Useful Tools

Functions are great tools, and many of the methods in the String Class are awfully convenient, including String.toUpperCase, String.toLowerCase, String.escapeXml, and so on. They let you not have to worry about doing specific tasks, because they're solved problems. String.repeat is especially useful when you need large, repeating strings, for example, as it can actually perform thousands of times faster than writing your own loop.

Some Functions Are Merely Convenience Tools

However, functions like String.format are not your friend. String.format does not really "solve" a problem. First, it's a function, which means there's a performance penalty per call, and second, the only thing it's really good at, putting positional arguments anywhere, is rarely so useful that you'd actually "want" to use it instead of just using "+". It can't even accept an array of Object values to allow for formatting Dates, Booleans, Integers, and so on.

Which Is More Flexible

"+" and "+=" handle can handle most primitive data types, and all custom types that have a Object.toString method, but requires loops when you don't know how many elements are involved. String.format can support String values and nothing else, and the format string has to be either hard-coded or built on the fly, in which case, it's useless because you may as well build the string yourself. String.join has problems with custom Object.toString methods, but works very quickly with most primitive data types, and can handle arbitrarily long lists of random data without too much trouble. It's preferred for variable length strings instead of building your own loop.

Which Is More Efficient

The specialty functions that skip the mess of allocating memory repeatedly are some best functions in the library and are literally more efficient than anything you could write in Apex. String.format, while a nice tool to help you visualize string construction, is a function that's nearly useless in terms of performance. It's about 6 times slower than "+", and it can't even handle non-String values, like numbers, Boolean values, and so on. String.format will never outperform "+" and "+=", but it sometimes makes code easier to read.

The functions that are useful for getting new strings efficiently include most of the functions in the string class, even String.join if you have a variable/unknown number of strings to join at once. However, you should avoid using any function that's duplicated by an operator ("+", "+=", "=="), because those are only convenience methods that will slow down your code. For example, String.equalsIgnoreCase is the same as "==", but requires way more code, is slower than "==", and throws NullPointerExceptions. It literally has no purpose in the library that I could find, yet people use it because it's there.

Summary (tl;dr)

Use operators when at all possible. They will always outperform any function call by a large margin. However, do use functions that are purpose-built to do "just one thing", and make sure you keep checking the String Class with every release, as many of those functions are useful. String.format should be avoided in most cases, especially in a loop, but can be useful to make code more legible when you have a large number of values you want to string together (but, arguably, I'd still rather use String.join).