The issues with the compression fitting have been addressed.
The brass flange fitting, the rubber gasket, and the threaded pipe is the "toilet spud". Replacements are available if the rubber is perished or the metal parts somehow damaged.
The flange nut looks pretty well bonded to the threaded portion. Soak well with WD-40 and let it soak in. There are splines inside the spud on which one would apply an internal spud wrench to keep the threaded pipe part from turning. You may find some other chunk of metal that will wedge against the splines, which in turn can be held by a conventional wrench. Holding the spud in place, remove the flange nut with a suitable tool.
Once the flange nut is removed, push the threaded pipe part inwards, but ensure it will not fall into the toilet. The part inside the toilet is conical shaped. Pushing it inwards will give enough free play that the rubber gasket can be pulled out. Once the rubber is out, there will be enough clearance in the toilet hole that the threaded pipe part can now be removed.
Examine the parts and evaluate if replacement is necessary. If the rubber is not too far gone, it may be possible to clean it, treat it with some sort of rubber treatment, and reinstall. If the rubber is cracked, hardened, rotted, or brittle, it will need to be replaced. The nut should be tightened firmly enough to ensure the rubber seals well against the toilet, but not so tight the rubber gets pushed out of position or the toilet gets cracked or broken.
I think what is leaking is the drain. Does it look like the object sticking out to the side in the below picture? If so, this is used to drain water from the house side of the supply when the valve is off.
It sounds like you over tightened it in the first place which started the leaking. If you've broken only the cap, you might be able to find a replacement. If the stem and threads are damaged, long-term you will need to replace the entire valve. As a short-term fix, you just need to block it some how. Marine epoxy is handy for emergency plumbing repairs.
Best Answer
Turn the water off, and remove it. Those things are rubbish, and always leak eventually.
If you need a valve there, cut out the damaged section of pipe, and install a proper tee and valve. If you don't need it, cut out the damaged section of pipe and install a coupling.
If you're not an expert solderer, They make push-fit fittings (SharkBite® is the well known brand) that are easy to work with. These fittings are more expensive than typical fittings, but ease of use and not having to buy soldering supplies might make them comparable.