I used to do 1/4 size, 512x384@30fps with not much problem, but then I got a new video card (X800XL) and a little more memory, and I was surprised by the terrific performance boost. Previously there had been a fair amount of stutter while I was recording, however that was only in the game, not the recording. The clips themselves were generally slightly lower fps than the game would have been if Fraps hadn't been running, if that makes sense (it's late). Also that was back in the day when Fraps wasn't as compatible as nowadays; there were some games it just plain didn't record well. The last few versions haven't had that problem at all, and I record a wide variety of games.
After my upgrade (and my computer is still barely above middle-of-the-road, I should point out), there was so little performance loss, even recording full screen at 40-60fps, that I sometimes got lost as to whether I was recording or not! I would end up with a whole bunch of 4gb files, video of everything
except what I thought I was recording.
Finally I got wise and installed a program that puts a HDD activity light on the screen, since the hard drive going crazy is the best indication that recording is in process. That fixed the problem.