Note: All of my terms and page references are from my English PDF of CoC 7th Edition. In addition, I have directly dealt with the player behavior described, and I have run CoC, but I have never dealt with this specific problem while running CoC. First I will address the rules as written. Then I have some more general advice based around things I've tried which worked in other games.
The Rules As Written
There is no rule in the 7th edition which expressly forbids players taking turns on certain tasks in order to maximize the chances of success. There ARE rules which can help you remain in the spirit of the setting.
The section When to Roll Dice beginning on page 82, and the section Rolling Dice beginning on page 194 both provide some guidance here. There's too much for a full quote, but the the core of it is "There's no need to roll dice for everything," and "The Keeper decides when to roll dice."
Applying this here, first, you as Keeper will decide when a roll is appropriate. The players can ask, and if allowed and failed, they can attempt to justify Pushing the Roll. But you decide if one is allowed in the first place. More importantly it means you can decide when a roll isn't needed at all.
Player: "We search the shelves for clues."
Keeper: "A quick search reveals one book hastily replaced out of order, its cloth bookmark marking a specific page."
No roll. Just a successful search.
The Rolling Dice section also suggests rolling in full view, even for the Keeper's rolls. I think this creates a different dynamic than rolling in secret.
The rules for Spot Hidden on page 76 address this directly, suggesting automatic success for players performing "a thorough search" might be appropriate.
The rules for "More than one player rolling dice for a skill roll" on page 86 should help you decide when you allow multiple skill rolls from the group. One of those examples expressly states all characters in a position to potentially spot something get a roll. But those rules also give you some idea when separate rolls should be allowed, and even required in some circumstances.
Lastly, if you or your players have access to the Investigator Handbook, the Rules Advice beginning on page 216 has some useful information on how to approach the CoC mindset, including this tidbit on Accepting Failure:
Don’t be disappointed when you don’t win every roll. Accept failure—it
can take the story to unexpected places. Sometimes, in hindsight, you
might be very grateful your investigator didn’t manage to open that
cellar door.
General Advice
The rules for Pushing the Roll (I think this is what you referred to as "forced rerolls;") begin on page 84. These rules DON'T refer to multiple players, but I reference it here because they lay the groundwork for something you can try, which I have done in other games. Consequences.
When Pushing, the player must explain what gives him the impetus to make a new attempt, and then the Keeper must explain the consequences of failure. As Keeper, you could use this even when multiple players are trying. There are possible consequences to multiple attempts which could occur regardless of who's making the attempt.
- "A second complete search of the room will take time you can't afford. If you fail the Cultists will be one step closer to completing their ritual."
- "A second attempt to break down the door will definitely attract unwanted attention if you fail."
- "A second attempt to repair that engine might irreparably damage a vital part, making it useless until you can have it in a shop during downtime."
- "A second attempt to intimidate the guard may make him mad enough to sound the alarm and attack you."
Several of the above examples touch on the second point. Your players should always consider time. Time is a HUGE factor which will work for or against the players. Wasting time on multiple attempts at the same action is certainly one way to ensure time works against them. The enemy may be closing in... or getting away...
In play, I've simply made players aware of the time required to perform repeat attempts. Sometimes I mention a specific possible consequence, but others I just say something like, "Sure, you CAN all search in turns. Are you sure you want to take an entire hour?" Then, after warning them, let them do it if they want. And make it matter. "You find that piece of information that eluded you, but now you hear a key in the lock of the front door!"
Properly used, time can help you.
There's an old text-based computer adventure where all the events are timed. It's literally possible to "wait" through the entire game and the adventure will just pass you by. I'm not suggesting anything this extreme, but if you demonstrate the world isn't static while they're busy, it will encourage players to move along.
Finally, consider the necessity of the roll. The absolute best way to discourage re-rolls is to not require rolls in the first place, except when needed. We touched on this above as it related specifically to the rules, but it's something to consider all the time. Is a roll really necessary?
In play, I generally call for a roll when the result will be interesting no matter how it comes out. Failure can move a story along just as well as success. And failure can be just as boring as success in some cases.
As a corollary to this, I use a rule I borrowed from another game called "Say Yes or Roll Dice." As GM in most games, it's my job to let players have their way, OR to make a conflict out of it. I love to RP and interact, but at some point the players will ask, "look, is this guard going to get out of our way or not?" And then I will say "yes," or the scene will become a conflict and the dice will decide.
As a final note which isn't really GMing advice, just make sure your players are up for the game you're running. Your descriptions make it sound like you have a conflict of tone vs expectations. There is a... pressing darkness... an urgency implied in many situations of CoC. The looming horror isn't going to wait around for your team to make 6 separate attempts to do ANYTHING. So, what are your players expecting out of this experience?
When he:
Contradicts you
Absolutely unacceptable as a matter of general habit. Talk to him, lay down the position that he needs to accept at least the fact that mid-session, you're the one responsible for adjudicating the rules.
There's some subtlety in how you might want to adjudicate disagreements, but ultimately, mid-session, this is your job.
Says something you disagree with or interferes with you running the game
Disagree with him openly, and see what happens. If he rolls with it perfectly well, there's nothing to do. If he contradicts you, see above. If he doesn't contradict you outright, but this interaction rubs him wrong, discuss this situation with him and phrasings he can use to make this pattern easier. Things such as "Ask the DM if you need to make a check" instead of "make a check".
Says something you were about to say or that you would have wanted to say in retrospect
Roll with it. This isn't a problem. You're getting a free assistant.
Best Answer
If you think about it, it won't work to just say: "Oh, you rolled a natural one? Well, there's definitely no secret door there, then!" The players will quickly figure out that low rolls mean you'll tell them the opposite of the truth, and they'll start doing the opposite of whatever you tell them.
When a DM really wants to give false information based on the die roll, they have to do something more complicated: they ask for the player's modifier, roll for the player behind the DM screen, and then give an answer without ever telling the player what they rolled. Some groups like to do this, but I think it's not as much fun when you don't get to roll for yourself.
Here's what the rules say:
"Progress combined with a setback" might mean, for example, that the characters eventually find the door but it takes much longer than they expected. Or it might mean the characters find the door but the ivy turns out to be poison ivy, with some associated penalty. Or it might mean that the characters find the door but all the pounding on the walls as they search draws unwelcome attention.
In many cases, the act of searching might take significant time, so the penalty for a failed check is just that the characters get no reward for the time they invested.
But, if there's no obvious setback to be had, it's probably best to just say "no, you don't find anything" and let the players move on.