Vent AV closet into attic

Yikes, this is really the first time I have ever heard of this configuration. Could this have been part of a previous central forced hot air system? It is not common to tie exhaust vents from a bathroom to any other living space. A dedicated exhaust fan vented to outdoors with a one way damper is a very important way to remove excess moisture from bathrooms and help prevent mold and mildew growth. It is not acceptable to vent a bathroom to another living space or into an attic. It is equally not right to tie this type of vent to any other air plenum. If you are feeling cold outside air through these vents, you are getting huge heat losses. There is no need to introduce outdoor air into your house intentionally unless it is done with specifically designed air exchange equipment in an extremely airtight designed house. I think you need to determine the routing of these "phantom" vents and come up with a plan to isolate the bath vents to the out of doors with a proper bathroom fan unit and block your heat form other areas from being wasted out an open plenum.

You have two big issues here.

First is that you have a serious risk of venting sewer gasses into your home. This is why drain vents open above the roof line or use one-way air admittance valves. This is likely to be unpleasant and possibly unhealthy.

Second is that a drain vent is typically too small and too moist to properly vent a bathroom fan. Your fan is probably designed to use a 4" duct, and instead it's going through a 1.5" diameter pipe. And if any lint/dust gets pushed out by your fan, it risks accumulating on the sides of the vent pipe, potentially blocking the pipe and slowing both your drainage and ventilation.

Beyond those functional problems, you'll also have a pretty visible issue that anyone inspecting your attic will see. You should do this the right way: add a properly-sized duct line and a roof vent for ventilation, and re-seal your drain vent pipe.