The issues with the compression fitting have been addressed.
The brass flange fitting, the rubber gasket, and the threaded pipe is the "toilet spud". Replacements are available if the rubber is perished or the metal parts somehow damaged.
The flange nut looks pretty well bonded to the threaded portion. Soak well with WD-40 and let it soak in. There are splines inside the spud on which one would apply an internal spud wrench to keep the threaded pipe part from turning. You may find some other chunk of metal that will wedge against the splines, which in turn can be held by a conventional wrench. Holding the spud in place, remove the flange nut with a suitable tool.
Once the flange nut is removed, push the threaded pipe part inwards, but ensure it will not fall into the toilet. The part inside the toilet is conical shaped. Pushing it inwards will give enough free play that the rubber gasket can be pulled out. Once the rubber is out, there will be enough clearance in the toilet hole that the threaded pipe part can now be removed.
Examine the parts and evaluate if replacement is necessary. If the rubber is not too far gone, it may be possible to clean it, treat it with some sort of rubber treatment, and reinstall. If the rubber is cracked, hardened, rotted, or brittle, it will need to be replaced. The nut should be tightened firmly enough to ensure the rubber seals well against the toilet, but not so tight the rubber gets pushed out of position or the toilet gets cracked or broken.
Best Answer
Whether dope is required at all, is dependent on the type of threads. The type of thread is determined by the tap or die used to create the threads, and should be labeled on the pipe or fitting.
Thread Standards
National Pipe Thread (NPT)
This type of thread when mated, may contain slight gaps between the major and minor diameter of the threads. Because of this, a thread seal agent may be required. However, this type of thread is tapered at a rate of 1/16 (3/4"/foot), which often allows a seal to be made without a sealing agent.
May be listed as MNPT or FNPT, for male and female threads respectively.
National Pipe Thread Fuel (NPTF)
NPTF threads are designed so that when the threads are mated, they actually deform to create a mechanical seal. Since the seal is created by the threads themselves, a thread seal agent is not required (though may be used as a lubricant).
May be listed as MNPTF or FNPTF, for male and female threads respectively.
National Pipe Straight (NPS)
This type of thread is similar to NPT, except that the thread does not have a taper. A thread seal agent is required, but should be selected differently due to the lack of taper.
May be listed as MNPS or FNPS, for male and female threads respectively.
National Pipe Straight Fuel (NPSF)
NPSF like NPTF creates a mechanical seal due to the deformation of the threads, however, unlike NPTF NPSF threads are not tapered.
May be listed as MNPSF or FNPSF, for male and female threads respectively.
Selecting Dope
Which form of dope you choose is often based on who you learned from, personal preference, what's on hand, what type of pipe you're working with, etc. The following are my personal guidelines.
Water
Plastic, Soft Metals, Unreactive Metals
In this situation I'll reach for PTFE tape, since it's not likely I'll require long term corrosion protection. I'm just looking to lubricate the joint, so I can tighten it to create a leak free joint.
Steel, Reactive Metals
Because I don't want the threads to rust or react negatively, in this situation I'll use paste dope. The paste dope will give me the lube I need to get the pipes properly joined, while at the same time providing corrosion protection. The paste dope will never harden or flake off, so I know the threads will be protected for a long time.
Large Diameter Pipe
If I'm working with water pipe of any material larger than ~1", I'll always use paste dope. I don't really have any particular reason for this, it's just the way I was taught.
Fuel Gas
When working with fuel gas pipes of any kind, I always use a paste dope labeled for this use. This is especially true when working with "black" pipe, since I want some corrosion protection in these joints.
Tapered VS. Straight Threads
Tapered Threads
For pipes and fittings with tapered threads, I'll simply follow the guidelines mentioned above.
Straight Threads
When working with fittings with straight threads, paste dope should always be used. Tape dope is too thick, and can actually prevent a good seal in straight thread joints. Paste dope will spread and be pushed out of the way, and will not prevent the threads from properly engaging.