Major edit: while I still personally like (k+1)th, @Mitch has found confirmation for (k+1)st in a math handbook, to which I must concur. Therefore, (k+1)st appears to be the most correct to mathematicians. Barring any future pertinent revelations, I would suggest using (k+1)st.
Kay plus first seems worse to me because one visualizes k + 1st instead of (k+1)th. If only for this reason, I would say "(k+1)th". Adding "orthographic consistency" to the mix pretty much decides the issue for me.
I would say that the only reason why it might not sound natural is because we don't say "(k+1)th" in ordinary life, hence oneth is not a usual word. However, we often use the word first, and therefore it sounds more natural. This does not mean it is the best choice.
As to which is used more often, doing the googling ("(k+1)st" -grade -grades and "(k+1)th") turns up evidence for the inference that each is used about as much as the other. (Wikipedia uses (k+1)th, however, for what it's worth, while our friends at Math.SE seem mainly to support (k+1)st.)
As for spoken usage, any would be acceptable, but all are clunky. Try to avoid such terms.
As for usage difference based on context - there does not seem to be a usage/meaning difference between the two.
Either way, spoken or written, I recommend strongly that you reword your sentence so that facing the issue is unnecessary if possible.
This is a simple case of cluster reduction of /db/ → /b/, combined with assimilation: /n/ is labialised to /m/ before a bilabial consonant /b/ or /p/, and velarised to /ŋ/ before a velar consonant /g/ or /k/.
I expect the cluster is reduced because it’s relatively uncommon. This is often where you encounter reduction and epenthesis. Compare:
- nuclear /klj/ → nucular /kjʊl/
- hamster /mst/ → hampster /mpst/
I’m a native speaker of American English, from New England, and a good portion of the time in running speech, I pronounce the cluster /db/ as /bː/ or /b/:
- goodbye (goobbye)
- birdbath (birbbath)
- bedbug (bebbug)
- deadbolt (debbolt)
And the cluster /ndb/ as /mb/:
- windbag (wimbag)
- sandbox (sambox)
- windburn (wimburn)
English has a long history of cluster reductions involving nasals and stops:
- /kn/ → /n/ in “knight”
- /gn/ → /n/ in “gnome”
- /mb/ → /m/ in “lamb”
So I expect this is little different.
Best Answer
Some native speakers find it difficult to pronounce sixth ( /sɪksθ/ ). It is not uncommon to hear people say 'sikth' ( /sɪkθ/ ). I believe that some are unaware of this mispronunciation.
I personally pronounce 'xth' as /ɛksθ/.
Unfortunately, saying /ɛkθ/ simply sounds wrong so I don't suggest that as an alternative.
If you are reading the expression from a board or a display where the listeners can see it, then you could get away with pronouncing it as /ɛks/. Listeners would tend to fill in the missing sound mentally. (Edit: Another very good approximation is /ɛkst/ )