Turns out the problem was my hard drives after all. Bottlenecking was a major issue, despite being 7200RPM and SATA 3Gb/s which I amount to being a fairly good system.
After investigating Dxtory further, I found that using the RawCap file format allows the program to record the file over multiple hard drives, and comes with software to combine them all into a single AVI file which would then need to be rendered.
As proof, I did a Path of Exile recording test while in windowed fullscreen. 1080p was no problem at all.
First, make sure you have console enabled. Somewhere in the options for your game, there should be a setting "Enable Developer Console: Yes/No". Make sure it is set to Yes.
When you are ready to record, open the console by pressing the "~" key (to the left of the "1" key). Type "record ", and press enter. "" should be replaced with the name of your demo (If you wanted to record a demo named "Bob", you would type "record Bob" into console.) Press Enter to start recording, and the "~" again key to close console.
When you are done recording, open console and type "stop". Press Enter. The recording has now stopped. If you want to watch your demo, type "playdemo " into console. (For example, if you wanted to play the saved demo "Bob", you would type "playdemo Bob".)
If you want more options to use while watching a demo, type "demoui" into console for advanced demo playback options.
Note: Keep in mind that demos cannot be uploaded to the interwebs, or viewed without a Source Engine game open. If you want to change your demo to a video (so that it can be uploaded to the internet and played my media players), I would recommend using a recording software to record demos. (I found that recording demos does not have negative effects on the video such as screen tearing or FPS issues.)
Best Answer
Windows recorder sometimes don't recognize some games. My recomendation to record a game with free software is: