Its a bit disconcerting to wake up one morning and find that overnight the entire review scene for both graphics cards and processors has changed dramatically. Together with several websites and writers in the technology field, Nvidia released their in-house frame latency testing tools to the public. Frame latency testing first took off with Tech Report’s Scott Wasson showing the world that frame latencies were to blame with jitters and micro-stutters observed in games, and that FPS averages were being used to mask the otherwise obvious issues to give graphics cards more favourable scores. Over time many sites have begun to include frame latency data using FRAPS, but Nvidia’s solution, designed with the help of PC Perspective, takes things to a different level entirely.
Very rarely do testing methods change so drastically that several heavyweights in the industry get behind new benchmarks so rapidly. For the past two years Tech Report has used FRAPS to observe frame latency in the middle of the GPU finishing up its job and handing things over to the DirectX API for processing, eventually landing on your screen. There’s a good few steps in the process from when you start moving your character to seeing something different on screen and it’s broken down awfully simply for you here.
FRAPS picks things up right at the beginning of the process, marking all the “T_present” frames and measuring them in per-second intervals to show the finally number on the overlay on the top-left corner of your screen. In the years since Scott’s first article on frame latency, he mentioned that FRAPS wasn’t perfect, but it was as close to the metal as you got. Both AMD and Nvidia can manipulate the final result through driver improvements and Microsoft can do its part with optimising DirectX, but if the “T_present” frames are delivered on time it matters not how many optimisations are made later on in the process. Many people refute this and claim that FRAPS is inaccurate, but the point still remains that its at the part of the process where neither AMD nor Nvidia can manipulate the final scores.
PC Perspective’s demonstration of what they define as stutter is shown in the video above, showing that without V-Sync there are several frames rendered on the screen but they’re not lined up properly, in some cases with as much as five different frames in different positions between screen refreshes. You’d think that with an industry as old as the PC gaming scene it would have been fixed long ago, but we’re nowhere nearer to having realistic breast movement than we are to delivering completely smooth gameplay in every single game out there. We can get close, but we’re still not quite there yet.
As a complimentary testing tool to augment FRAPS data, Nvidia helped develop its tool that has previously been used in a different form by their engineers. Called the “Frame Capture and Analysis Tools”, or FCAT, it’s a set of programs that Nvidia bundles together along with some in-house code that allow you to capture frames from a video capture card connected to your system. It records each and every frame you see on your monitor through a splitter and unlike FRAPS’ video recording, induces no performance penalty. It needs some heavy hardware to run effectively and Nvida helped most websites to setup the capture card along with a RAID0 array of SSDs to record the information. In most cases, as PC Perspective points out, the resulting data for all their benchmarks weighs in at around 600GB. This kind of in-depth testing isn’t meant to be done half-heartedly.
As you can see above, FCAT takes the various frames as they’re rendered and overlays a colour on the left for each frame to indicate its size and how long it runs for. That is how it normally looks when you have V-Sync turned off. With V-Sync on, it would be a solid colour throughout the entire screen, as V-Sync waits for the next full frame before it refreshes itself. The FCAT tool, together with the high-end hardware capture system, will now allow sites to test for stutters from both ends of the pipeline – at “T_present”, where FRAPS timestamps the frames as the GPU spits them out, to the very end of the process at “T_display” which is the final image you see on the screen.
As time goes by, the industry will drop frame averages and I expect quite a few sites that can’t get the capture tools to progress to using FRAPS for their benchmarking. After all, it doesn’t cost anything and doesn’t need beefy hardware and timestamping the frames after the GPU is done with them is the cleanest, most accurate software solution out there. As more writers adopt these testing methodologies, so AMD, Nvidia and even Intel will have to step up their game so they don’t get caught with their paints down. For years AMD appears to have used runt frames measuring only a few pixels in width to boost their scores in Crossfire and let’s hope this puts a stop to that as well.