We did this, and the favicon used by the site config page in salesforce works for both public and secure URLs...nothing extra to do there.
I didn't add the favicon on the site template.
I'd suggest giving it a few days. Is your site new, or just the favicon?
For the Favicon to work, I had to upload my image to the orgs assets library:
Under Branding>"Select an image to generate a palette" :


Only then was I able to reference my icon, which I had to upload as a 16x16 png image.
Finally, I was able to reference it using the file-asset's relative path as follows:

Notice that the icon name when I reference it has a versioning appended to it,
<link rel="icon" href="/file-asset/coveofavicon?v=1" type="image/png" />
You can use your browsers dev tools to identify the relative path for your community assets, for reference purposes, here is another link provided that helped resolve this:
<link rel="icon" href="/sfsites/c/file-asset/Favicon_FileName?v=1" type="image/png" />

I did not find this documented anywhere =(, hope this helps other people.
This will change some of your community stylings, since it will detect the colors your image/favicon uses, however, this is easily reversible by re-selecting the original colors you had under the branding option.
Best Answer
This may not be exactly what you are looking for, but it is a Chrome browswer extension that will override the default Salesforce icons:
https://chrome.google.com/webstore/detail/salesforce-colored-favico/peohlnebahcddpmfaplmilpkgbkkcdho?hl=en
According to the description:
Also, it may be possible to customize the solution that the author has provided.
UPDATE: I am editing the answer in response to a comment. I have no affiliation at all with the author of the Google chrome extension I cited.
I did confirm that changing the "favicon" in any html page requires a modification to the HTML tag. The Salesforce platform is never going to allow code that manipulates their generated HTML markup.
https://www.w3schools.com/tags/att_link_rel.asp
A browser extension to customize the icon may be the only way to do this, and this browser extension is fairly highly rated. A potential downside is that it requires access to read and modify data from websites that you visit.
Also, you may want to take a look at some of the comments on the Chrome extension's "Support" page. If Doug Ayers has installed this extension and got it to work in SF Lightning experience by tweaking a javascript file, that's something to consider.