Plumbing – I have an exterior water tap for a garden hose that needs new caulking

Water can travel a long distance, and come from lots of sources, so there's no easy answer here. You'll have to start opening the walls and working your way up until you reach the source. If there's any spaces to check (attic, roof, another bathroom) look there before getting too destructive.

Around a bathtub, you can check for any locations water can pool at the edges and ensure it's caulked and sloped into the tub. Check around the faucets, valves, and shower heads for anywhere water could leak and run back into the wall. Fill the tub all the way to the overflow level and make sure the overflow doesn't leak. Then open the drain when it's full to check the drain pipes for any leaks (some of these won't show until you're under pressure).

And realize that you could also be dealing with condensation if it's cold outside and your insulation isn't adequate with a vapor barrier. You could also have a leak from the roof or condensation on pipes above the bathroom. Even uninsulated HVAC ducts can get condensation in the summer from the cold AC, as can dryer and exhaust ducts in the cold winter.

You don't remove it. The silicone is probably just to secure the pipe to the wall - so it doesn't wiggle or so that air doesn't get in.

If you need access to the pipe I would cut the white pipe. This would allow you to fully snake anything below. Chances are pretty high that all of these pipes are glued very well together. They are not meant to wiggle apart. If you do not want to cut them then you can try putting in a snake from a point further up or you can see if you have access from below in your main stack.