You need not change anything about the default setting in order to have people "left out" of the benefits of clerical magic.

I don't think that clerical spellcasting is as easy to come by as you make it out to be. In other words: you can easily have your harsh and gritty world. In what follows I'll always lean toward the more-utopian interpretation of things, lean on the rainbows-and-unicorns end of numerical ranges, and I think you'll see there's still plenty of room for gritty/harsh living conditions. Obviously, then, if we dial back any of those happy-happy-joy-joy assumptions we can make things even harder for people.

Demographics

You've told us you want a harsh, uncaring setting. Luckily, much of our history on this planet has been conducted in that setting so we've got lots of data to go on. I'll point you toward my favorite such tool--Medieval Demographics Made Easy--which helps us imagine D&D's pseudo-medieval implied setting using the tax rolls of 13th-C France.

This gives a baseline of one "clergy" per 40 population, and one "priest" per every 25-30 clergy. For your purposes I'm going to interpret "clergy" as an unleveled acolyte, and a "priest" as a leveled cleric.¹ That means that we find a leveled cleric every 1000-1200 population.²

Villages/farmlands/wilds

Villages (according to MDME) top out at about a thousand people, so let's assume we've got a 1000-person village with one L1 cleric serving as its priest. That's two first-level spell slots per day for a thousand-person village.

In the case of illnesses remember that we can only cure two people per day. I don't feel like running a full S-I-R model here,³ but we don't need too-extreme of infection and recovery rates to overwhelm the "bonus" recovery from spellcasting. And curing an illness requires a second-level spell slot, requiring a third-level cleric!

In the case of extreme injury, just remember we only need three people seriously injured to overwhelm our poor country cleric. I've never erected a barn or taken down a tall tree, but I can easily imagine three simultaneous injuries. If a cleric can heal two but has to wait until the next day to heal the splinted broken ankle, there's your limper.

Accidental death. Let's talk size: sticking with medieval France we get a population density in arable lands of about 100 ppl/mi\$^2\$. So this village and its lands occupy about 10 square miles, or about a 3-mile (diameter) hex. Recall that without help you may only have 21 seconds of life left once you hit 0hp. The odds are not good that this accident happened within 20 seconds of our cleric =(

Cities

Cities (again, using MDME's classifications) range around 10,000 people.⁴ Using the same numbers as above we see that a 10K-person city should enjoy only ten leveled clerics.

But notice that the underlying numbers don't really change all that much. Travel times may go down, and with more casters we'll have some of higher levels (and more slots). But there are just so many people...

Again being generous, let's assume one cleric of each of levels one through ten, and that they use fully half their slots on daily petitioners. That's 45 spells to cast for ten thousand people, or 93 spell-levels for hoi polloi. All you need is more than one percent of the population to "need" something on a given day and you'll overwhelm the daily petitioning system. 2% would overwhelm the entire clerical-magic capacity of the city. One disease, a fire, even just a day when one of the high-level clerics is tied up in bureaucratic argle-bargle will leave some unfulfilled.

Turning the dials

Let me briefly summarize the assumptions made, and how changing them will change things:

• "Clergy" are not casters, only "priests" are. They are as abundant in D&D-verse as in medieval France. Obviously, in a world where the gods walk the earth there might be more priests. But keep in mind (a) there's also the lure of arcane casting; (b) clerics are also needed to deal with the undead hordes/curses/magically-fueled religious wars inherent in such a world; and (c) somehow the D&D-verse hasn't figured out any better agricultural technology than the moldboard, so we still need 90% of people to be involved in food production.

• Farmland is really rich so the region/kingdom/world is pretty densely-populated. Drop population density down to 30 ppl/mi\$^2\$ (like 13th-C. England) and one's proximity to a cleric drops commensurately.

• City-based clerics spend half their spellpower helping "commoners". This one I see as a fun dial to twist. If I know anything about bureaucracy at least one or two of them definitely aren't getting their hands dirty. Perhaps some orders are martially focused and so are embedded with a standing army and none of their clerical spells go to (directly, immediately) help common folk. Perhaps some order devotes 100% of its divine power to such works. Perhaps some order has grown world-spanningly-huge and spends much of its time maintaining its own bureaucracy. These are huge cultural differences that can really change how the world feels.

• The cleric in a village is a L1 caster. But one can imagine that a village where a L15 cleric has retired would be a little green patch of paradise: he attends all childbirths, so mother and infant mortalities aren't a thing. Droughts/blights? Create food and water'll sustain people and seed crops well enough to get through. Village festivals (to the cleric's god, obviously) come with a Hero's feast.

Bringing it back to reality:

The situation you describe isn't fundamentally different from our own world. There are people with physical troubles, and there are some with the power to alleviate those troubles. Why, then, do we see people in difficult circumstances? Proximity, access, cost, &c all play their roles. The same is true in the D&D-verse.

In some places charities step in. In some places social structures embody greater sympathy/antipathy toward the problem. But spell slots are a limited resource and so will evolve into their own economy, just like anything else.

^{1 - This, effectively, is exactly what @timster argues for in his answer.}

^{2 - If it seems I've queered the deal from the outset by declaring "priests" to be leveled clerics rather than "clergy" (and thus dialing back by a factor of 25 the number of available clerics), I urge you to consider the alternative: 2.5% of people are leveled clerics. Given twelve classes and assuming clerics are one-twelfth of leveled characters we see 30% of the population is leveled.}

^{Sticking with pseudo-medievalism we've got to have ~90% of the population involved with agriculture, though. So not only could you have every non-farmer a leveled NPC, one out of every five (roughly) farmers would still be, too! That's not really the setting implied by most published material. (But it's kinda a cool one, methinks. Every time you stop for dinner you've got to listen to the barkeep's old campaign stories, then the stablehand's old campaign stories, then the lamplighter's old campaign stories....)}

^{3 - okay, that's a lie. I really want to run an SIR model incorporating a clerical healer and I did. Assuming the cleric (a) doesn't get infected ever, (b) can get to and heal two people per day, (c) healed people now have immunity and can't re-infect, and (d) the same epidemiological parameters as the SARS outbreak in Hong Kong in 2003\$^5\$, we get a peak infected population of ~15% the total population; 150 people have this disease on the outbreak's worst day. The cleric, healing two a day, just doesn't do very much for those people; there's plenty of room for some disfiguring illness to scar people. (The cleric does have a huge effect on ramping down the "long tail" at the back end of the outbreak.)}

^{Admittedly, the underlying SIR parameters may be badly constructed for this setting: people are less-densely packed and don't travel nearly as much. But then again, hygiene and sanitation are much worse and the population may be starting at a less-healthy baseline. In short, I'm not a medieval epidemiologist, just an amateur mathematician with a copy of MATLAB.}

^{4 - This may seem small to us--it's the size of the suburb I grew up in, for instance--but "a typical large kingdom will only have a few cities in this population range." (MDME)}

^{5 - Razum, Beecher, Kapuan, Junghauss. "SARS, Lay Epidemiology, and Fear." The Lancet, May 2, 2003 + analysis of these data in one of my modeling classes in grad school.}

Very few feats published in official supplements or Dragon touch lay on hands:

• Action Healing (Faiths of Eberron): spend an action point to lay on hands with 50% extra healing, lay on hands as a swift action, or lay on hands from up to 30 feet away. Even if we assume you’re using action points, action points are way more valuable than those effects.

• Azure Touch (Magic of Incarnum): Grants 1 point of essentia, and allows you to basically trade away essentia for the day to gain another use of lay on hands (or wholeness of body). Essentia is way more limited and valuable than uses of lay on hands, so that’s a pretty terrible trade—the bonus point of essentia is by far the more valuable thing from this feat.

• Serenity (Dragon Compendium): Use Wisdom instead of Charisma for all paladin abilities, lay on hands included. Fantastic feat, but because of divine grace, not because of lay on hands.

• Touch of Silver (Faiths of Eberron): Burn a use of turn undead to make lay on hands hurt lycanthropes and evil outsiders as if they were undead. Which is, again, a terrible trade.

In three out of four cases, these feats involve trading away some limited resource in order to get some minor effect: these are terrible feats. Serenity is good, but not particularly because of lay on hands.

Which kind of works out, because lay on hands is a very weak ability. It is a really bad idea to invest feats, which are very scarce and extremely valuable, into it.

I do not know of any third-party feats that significantly change this conclusion, but I am also generally unfamiliar with third-party material.