Smoke is normal in an electric oven, but flames are definitely not.
In order to start a fire, you either need a spark, or you need to heat something beyond its autoignition temperature (AKA kindling point). You might have had a short - or you might actually be using a gas oven with spark ignition - but I'm guessing your issue was the latter.
Cooking oil or grease being heated beyond its autoignition point is one of the most common causes of kitchen fires (grease fires). Supposedly, some oils have autoignition points as low as 550° F (or 288° C), though I'm not sure which oils those are. Olive oil would be my guess as the lowest, but pepperoni grease could very well have ignited at self-cleaning temperatures (which, as you noticed, go up to nearly 1000° F).
Fortunately for you, all modern ovens have a mechanical interlock which prevents them from being opened during a self-cleaning cycle. If you'd opened it, you would have made the problem a lot worse by (a) supplying the fire with abundant oxygen, and (b) drawing all the hot air and flames out of the oven and into your kitchen, quite possibly setting your whole home on fire. Heat wants to move to where it's cold; that's why you keep your doors and windows closed in the winter.
There are a multitude of oven cleaners available for self-cleaning ovens - you are supposed to use these before you run a self-cleaning cycle. Yes, I know it's odd, but "self-cleaning" doesn't really actually mean that it cleans itself, it just gives you a little extra help. You need to try to clear out all the grease and big chunks of food first using one of these cleaners, then run the self-cleaning cycle to deal with anything you might have missed.
I don't think it really matters exactly what happened. Unless you've left out something important, it's not working right, and you need to get it repaired. The best case is that it keeps burning your food, and the worst case is that you set your place on fire.
If you want to confirm that it's doing something wrong, you could put a thermometer in it and turn it on (no food) and confirm that it's getting way too hot. (Edit: but please be careful, as Aaronut said. If there's any sign things are going wrong, turn it off, and leave the door closed.)
Best Answer
Grease fires do happen with self cleaning ovens, it's why I stopped using that feature: I didn't feel like I could leave it to get on with the cycle safely and I couldn't sit there for hours.
Cleaning it afterwards isn't a big deal, the grey dust is ash and you can vacuum a lot of it out once the oven is clean. If any ash remains good old soapy water will get off the rest.