There are three possible interpretations of the second sentence. All of them are technically grammatical, but none of them make for a natural sentence.
In the following, my use of ✘ is not to indicate that it's ungrammatical, but that it's unusual and not normally used.
1. a1) ✔ Seafood is harmful to me.
1. a2) ✔ English is harmful to me.
These sentence are fine, especially the first one. (Thinking of English as being harmful is slightly odd, but it's still understandable.)
1. b2) ✘ Seafood is difficult to me.
1. b2) ✘ English is difficult to me.
But if you replace harmful with difficult, it no longer really makes sense. Things are not normally difficult to other things.
2. a) ✔ To me, (in my opinion) English is difficult.
This is more of a general statement, saying that English is difficult objectively, rather than only saying it's difficult in terms of your own use.
This also makes sense, but this interpretation is only normal if to me is put at the front of the sentence, typically followed by a comma.
2. b1) ？ English is difficult, to me.
2. b2) ✘ English is difficult to me.
If this interpretation is used, having to me at the end is semantically fine, but unnatural in its order. Although, note how the addition of the comma actually makes this interpretation more understandable.
3. a) ✔ I find it difficult to use English.
This makes sense, and it has the same meaning that's being expressed in the question.
3.. b) ✘ English is difficult to me.
But the phrasing in the original version is not something that would normally be used. As the Google Ngram Viewer results show, the use of for is much more common.
What makes the use of to more unusual is that there are other interpretations that can prevent you from being totally sure what is meant, even if this third interpretation is the most likely.
No matter which way you interpret the sentence, the use of to is either uncommon or outright strange.
Best Answer
To tell someone something is to to give information to that person; to tell someone of something is to give information about something to that person. In (1) he said that there was a story, that a story existed; in (2) he said the whole story to you.
Tell of
Tell