Plumbing – PVC toilet flange replacement problems

Background:

I wanted to replace the wax ring on my toilet but when I took it apart I found that the flange was in bad shape. The notches that hold the bolts were broken out. That, I think, led to the toilet wobbling, which led to a leaky wax ring.

The flange was screwed down and the screws were rusty and I was stripping them trying to get them out, so I had to chisel the flange apart and remove the screws with some pliers. The flange was the kind that fits inside the pipe. It looked like this after I had broken up the top of the flange and removed the screws:

That's a tile floor, slightly crumbly around the hole (it doesn't look as bad in person). You can see the little piles of rust where the screws were. Finally, I used a saw and chisel to remove the last of the flange from the inside of the pipe.

The problem:

Once I got the flange out, it looked to me like there was another layer in there. I mean it looked like the pipe contained small second pipe, into which my old flange had been inserted. I began chiseling that out as well when I stopped and wondered whether I should be doing that. It didn't seem that the inner layer was coming out as cleanly as the flange had done. At this point it looks like this:

I believe that's purple PVC cement around the inside. You can see that there appear to be two layers. On the right you can see where I started chiseling out the inner layer. On examining a bit that I chiseled out, it does appear to be an inner layer. Here is a photo of it. You can see some printing on the outside surface (by which I mean not the purple side).

The question(s):

What now? I'm up to my neck in this project and I'm not sure how to proceed. What do I need to do in order to prepare this pipe to accept a new flange? Should I stop now and just install a new flange in the pipe as-is? Do I need to remove that inner pipe? And the big question, is it time to admit defeat and call a plumber?