What I usually do is:
Join a server, and type "record demoname"
Later, play it back using "viewdemo <demoname>", that way you can skip around to the cool parts.
Then Frap it up from there.
Recording in real time is too much of an FPS penalty.
Also, can you see the FPS counter while in game?
Are you hitting the correct hotkey? ( the numbers turn red while it's recording)