Sony has filed patents for backwards compatibility

Two recent US patents by the company have been the subject of much speculation this week, and seem to imply that Sony is planning to include backwards compatibility in its next-gen PlayStation console using new emulation tech.

Remastering by emulation

US 10086289 B2 deals with the topic of emulating games in a computer system with their original assets, which may include textures and audio. In today’s environment, we commonly see games get new high-resolution texture packs that greatly enhance the look and feel of the title. Crysis 2 got a pretty big one, measuring at around 11GB for new high-res textures for gaming at higher than 1080p. Fallout 4 got a new high-res pack recently, though many of those assets are either copied straight from textures created by the modding community, or created by Bethesda Softworks themselves. Lots of games additionally ship with two sets of textures – one for 1080p games, where the reduced overhead of storing the smaller textures in VRAM helps improve performance, and another set for playing games at 4K or using supersampling for a 1080p display.

For games which aren’t likely to be remastered or ported to newer systems, this presents a problem for playing these titles on modern displays. Sprite-based games might fare better, but anything that uses a modern way of texturing any 2D structure on-screen runs into a problem where the original textures look ugly as sin. Sony can’t do anything to the original work if no-one’s prepared to work on updating it.

The plan, then, is to design a software shim that sits in between the game and the hypervisor that is hosting it, allowing Sony to do multiple things without altering the original work – all they have to do is repackage it. For example, if the game originally ran at 800 x 600 resolution on a CRT display, the shim will allow the game to be run at a higher resolution, but will also retrieve higher-resolution textures that will be forcefully inserted into the graphics pipeline so that the higher-res textures are shown on the display. The same method can be used, Sony claims, for remastered audio and video sequences, and to provide compatibility with new hardware interfaces.

The patent goes into detail about how the shimming is done. The remastering artists takes stock of all the audio and texture assets that a game ships with, builds a database of the names of these files, and hashes the file names with a date and time stamp. The artist then goes to the remastered assets, and does the same thing, hashing both sets of data to match each other. When drawing a new frame and issuing draw calls for rendering objects in virtual space, the software shim checks the hash of the asset and compares it to a database of higher-res assets. If there’s a match, the shim will provide the higher-res asset to the game engine, and away you go.

That database can also be anywhere. It can be stored locally on the disc, stored on the hard drive, downloaded as part of the game’s installation files, or streamed in from a remote source on a local or remote network.

The patent also presents a new form of upscaling. While digitally remastering the game, the artist working on the title will display the game on a high-resolution display, while running the game at its native resolution, blown up to match the display size. Because the original geometry can’t be changed, the artist will need to manually trace new 3D meshes, giving them a high-resolution outline of the original object. Any high-res textures or effects are then applied to this “upscaled” geometry.

This is a neat trick, and one that will help Sony sell old titles again on the PS Store without worrying about needing to port them over. The amount of time dedicated to porting over the few legacy titles that run on the PS4 isn’t worth the effort considering that Microsoft’s backwards compatibility catalogue grows by the day, so this will be Sony’s ticket to keep people in their ecosystem who want to play older games – now with free “remastering” thrown in as a bonus.

But wait, there’s more

There’s an important note that needs to be pointed out, and it’s Sony’s own words here. Look at the text below:

This technology targets games running on older systems at any resolution, including PSP and PS3 titles. Sony specifically mentions the PSP and PS3 platforms because neither of those systems offered touch interfaces. If the significance of this is lost on you, let me rephrase: Sony can’t port games from the PS Vita to other platforms without touch interfaces being present (the DS4’s touchpad doesn’t count), but PSP and PS3 games are A-OK.

Whether they announce backwards compatibility for the PS4 and PS4 Pro at Paris Games Week next month, or whether they reserve this for the PS5 next year, we’re getting faux-remasters of our favourite games on the PSP and PS3 soon.  Now we know why they decided to stop offering PS3 and older titles on the PS Plus subscription.

Video frame rate compensation through adjustment of vertical blanking

A 2014 provisional patent application, 15/911860 was granted to Sony on the basis of their method of implementing variable refresh rates through altering the vertical v-blank interval of a given display that supports the feature. On the surface, it looks similar to the methods already employed by AMD and NVIDIA in their software/hardware features called FreeSync and G-Sync respectively. To refresh your mind, NAGlings, when a game engine spits out frames that are completed by the GPU and sent to a display, the display can support a function where it analyses the frame rate of the output frame and will adjust the refresh rate to match the current frame rate.

Sony’s patent, US 2018/0277054 A1 is a little different, and reaches back into history to perform its trick. Instead of adjusting the refresh rate of the monitor, the idea is to instead adjust the v-blank interval. A v-blank (VBI) is the action of clearing a display in preparation to draw an entire new frame. This was required for CRT displays, because you wanted to allow all residual light to leave the tube before moving on to the next image, and because the beam could not change direction arbitrarily. CRT technology relies on a laser that scans very fast and hits the edge of the tube, and the laser scans out each line of the display and each pixel individually in a uniform fashion. Modern displays have no need for a VBI, but a faux VBI is available on newer displays that will strobe the backlight in between successive frames to create the illusion of smooth motion, reducing any blur.

Sony’s patent aims to smooth out the frame delivery of a game by using a variable VBI to adjust the effective frame rate. As the diagram above shows, the GPU renders source frames and puts them into a buffer in the VRAM and system RAM, one called the front buffer and one called the back buffer respectively. This is what double-buffering is, queuing up two frames at a time to allow the GPU more time to work on the second frame to minimise any hiccups and smooth out the frame delivery. When the frames are outputted at 118, it yields an effective refresh rate that affects the VBI at 120. Depending on the refresh rate of the monitor and the frame rate of the game, the VBI is altered and increases the amount of time (102) between a completed frame moving from the back buffer to the front buffer (between 112 and 110).

This feature almost certainly ties in with US 10086289 B2, “Remastering Though Emulation”because games made for older platforms were designed for CRT displays that performed v-blanking on their own at a constant pace. Any variability in the software environment, like the GPU or CPU automatically overclocking itself would be bad for the game experience, because it would then have frame rates that would vary wildly. But wait, there’s more here too!

Part of the patent also applies to using the VBI to adjust the frame rate of games that are streamed to another device. In the above scenario, the VBI is altered on the fly to produce a smooth, consistent frame delivery, but it’s not doing this for a target display. Instead, it’s limiting the frame rate by slowing down the switch from the back to the front buffers (302), and allowing for a consistent frame capture rate target (say, 30fps or 60fps). The captured frames are sent to the hardware encoder, which will vary the compression rate according to the VBI.

This means that where the frame rate is low, the compression rate can be lower to boost the image quality, and vice versa for games which have high frame rates, incurring a higher compression ratio. This is technology meant for emulating older, pre-PS4 titles via Sony’s PlayStation Now service. Previously, PS Now would compress all games with the same compression ratio, which meant crushing colours and making everything look a lot more washed out. With this tech, they could improve the visual fidelity of the game streamed over the internet, and vary this on a per-game basis. Given that they already allow for some games to have an option to use a targeted or unrestricted frame rate, it would be interesting to see if PS Now allows users to select the visual quality they’d prefer.

Black Adder Prince George
On top of everything else, Fallout 76 now has a minor class war going on