What Is Shader Cache

Return to site

What Is Shader Cache

Clear Shader Cache
What Is Directx Shader Cache
What Is Unity Shader Cache
What Is Shader Cache
What Is Shader Cache Cemu
DDC stands for Derived Data Cache.
Many Unreal Engine Assets require additional 'derived data' before they can be used. (A simple example would be a Material that has a shader). Before the Material can be rendered, the shader must be compiled for the platform the editor is running on. 

Because the derived data is large, and at times may need regenerating, it is not checked into source control. Instead it's kept in the Derived Data Cache (DDC). 
Where the DDC Is StoredDepending on how your project and system are configured, there can be several DDC caches in a hierarchy that ranges from fast to slow. When assessing derived data, your system will do the following to determine how quickly it can access it: 
If there is a directory there called 'NVCache', delete it. Warning, turning off the NVIDIA Shader Cache may put a little extra stress on your CPU, but I haven't noticed a significant change. If anything, my game seems to run a little smoother overall, even at the start of a match. In this Cemu 1.9.0 Guide I attempt to show you exactly what a Shader Cache does in relation to Cemu Emulator, and how you can easily build one yourself.
When a piece of derived data is needed, the fastest cache is checked first, then the next fastest, and so on, until the data is found. 
When the data is found, it is copied into the fastest local cache so that it is quicker to access next time. 
If the data is not found, then it is first generated, then asynchronously copied into the caches so it is available for you (and potentially your team) in the future. 
Content stored in the DDC is disposable, and can be regenerated at any time using the data stored in the .uasset file. Storing these derived formats externally makes it possible to easily add or change the formats used by the engine without needing to modify the source Asset file. 
Types of DDCsTypically, you will have at least the following two caches: 

Because the derived data is large, and at times may need regenerating, it is not checked into source control. Instead it's kept in the Derived Data Cache (DDC). 
Where the DDC Is StoredDepending on how your project and system are configured, there can be several DDC caches in a hierarchy that ranges from fast to slow. When assessing derived data, your system will do the following to determine how quickly it can access it: 
If there is a directory there called 'NVCache', delete it. Warning, turning off the NVIDIA Shader Cache may put a little extra stress on your CPU, but I haven't noticed a significant change. If anything, my game seems to run a little smoother overall, even at the start of a match. In this Cemu 1.9.0 Guide I attempt to show you exactly what a Shader Cache does in relation to Cemu Emulator, and how you can easily build one yourself.
When a piece of derived data is needed, the fastest cache is checked first, then the next fastest, and so on, until the data is found. 
When the data is found, it is copied into the fastest local cache so that it is quicker to access next time. 
If the data is not found, then it is first generated, then asynchronously copied into the caches so it is available for you (and potentially your team) in the future. 
Content stored in the DDC is disposable, and can be regenerated at any time using the data stored in the .uasset file. Storing these derived formats externally makes it possible to easily add or change the formats used by the engine without needing to modify the source Asset file. 
Types of DDCsTypically, you will have at least the following two caches: 
Boot DDC: Startup files that are loaded into memory to increase boot times. 
For Example: ProjectDir/DerivedDataCache/Boot.ddc
Local DDC: Derived data for your projects. 
For Example: EngineDir/DerivedDataCache
However, your project may also be set up to use a DDC Pak or Shared DDC.
DDC PakIf you download Unreal Engine from the Epic Games Store, the engine will come with a DDC Pak (.ddp). The DDC Pak contains derived data for all engine content, so you can start working without compiling shaders and so on. Similarly, some samples are shipped with a DDC Pak for the same reason. 
Engine DDC Pak 
For Example: EngineDir/DerivedDataCache/Compressed.ddp
Project DDC Pak 
For Example: ProjectDir/DerivedDataCache/Compressed.ddp
Shared DDCOne of our strongest recommendations for teams in the same location is that they set up a shared DDC. This is a network drive that all team members and build machines can read/write to. This then writes off the cost of creating any required DDC data across the entire team. 30 in one electronic project lab manual. For example, when an artist edits a shader, the DDC data will be written straight to the DDC share. 
Shared DDC: Will be either a network or mapped drive. 
An example of a shared drive would be: epicgames.netrootDDC-Global-Fortnite
Using a Shared DDCA studios should use a shared DDC that all users in a particular location can access. This way, only one person needs to build the derived Asset format(s) and they will be automatically available to all other users. There will occasionally be stalls when Assets need to be processed, but the results are stored and shared. Even on a fairly small team, sharing Asset processing work in this way will eliminate most processing time. 
Clear Shader CacheIt is not advised to copy an entire DDC across the internet, or back up your DDC, or restore a DDC from a backup. While not necessarily harmful, this would be a waste of time—it takes longer to transfer the amount of data stored in the DDC than it does to generate it from scratch locally. If you have a large project and want to distribute pre-built DDC data, you should generate a DDC Pak.
Setting Up Shared DDCAs you may see in BaseEngine.ini, the editor is already set up to use a Shared DDC that can be enabled one of three ways. 
Here are the options from most to least recommended. 
Add an override (as pictured below) in the DefaultEngine.ini for your project that sets the path to a valid location for your team. 
[DerivedDataBackendGraph] Shared=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=10, FoldersToClean=10, MaxFileChecksPerSec=1, ConsiderSlowAt=70, PromptIfMissing=false, Path=YourCompanyServerDDC, EnvPathOverride=UE-SharedDataCachePath, EditorOverrideSetting=SharedDerivedDataCache) 
Set an environment variable (as pictured below) of UE-SharedDataCachePath (On Mac/Linux UE_SharedDataCachePath) to the folder to use. 
In the editor, set the SharedDerivedDataCache variable. 
Click image for full size.
You don't have to do anything special to YourCompanyServerDDC; as long as it is writable, the engine will create the structure it needs. 
Disabling a Shared DDCDevelopers working remotely with a project that has a shared DDC configured may experience poor performance due to the fact that it takes longer to access DDC data than it does to generate it. To temporarily disable a shared DCC, use one of the methods below: 
Pass -ddc=noshared on the command line. 
Set the environment variable to a local drive hard drive: 
UE-SharedDataCachePath=None
On Mac: UE_SharedDataCachePath=None
Building Derived DataThe user who imports the Asset is the one who builds the derived data since they will most likely be using and testing that Asset in the Engine. However, there may be times when a new Asset needs to be processed. This happens automatically on an as-needed basis and should not result in much of an impact when running on fast hardware, though there may be occasional stalls. 
You can fill your DDCs at any time by running the following command: 
Epic Games does this on a nightly basis to ensure that the DDC is always primed, but it is not necessary as the general automatic caching feature should suffice.
Mounting Your DDC FolderFor the DDC to be used over your internal network, the folder that contains the DDC must be mounted to a network drive. If you are unfamiliar with how to do this, see Map A Network Drive on Windows.
Distributing with a DDCCooking is the preferred method for packaging games since cooked builds do not need or use a DDC. However, DDCs can be packaged for distribution if the need arises. 
Cooking is the preferred method for packaging games since cooked builds do not need or use a DDC. However, DDCs can be packaged for distribution if the need arises. 
To package a DDC:
Run UE4Editor.exe from the UE4/Engine/Binaries/Win64 directory, passing the arguments shown below: 
UE4Editor.exe ProjectName -run=DerivedDataCache -fill -DDC=CreatePak
This creates a DDC.ddp file in the UE4ProjectNameDerivedDataCache directory. 
The engine automatically detects and uses the .ddp file. Rosetta for mac free.
DDC SettingsWhere are DDC settings kept?DDC settings are kept in DefaultEngine.ini under the [DerivedDataBackendGraph] section. You can view the defaults in BaseEngine.ini.
Common QuestionsQ: Can I have multiple DDC settings?Yes! Just create a new [YourDDCSettings] entry in DefaultEngine.ini and then run the editor with -ddc=YourDDCSettings.
At Epic, we do this for three reasons: 
For teams in offices that want to use their own DDC instead of the global one. 
When creating DDC Paks we use one of several options that control what goes into the .pak file, such as [CreateInstalledEnginePak].
For people working from home who don't want to use a Shared DDC due to slower internet or not having VPN enabled. 
You'll see that BaseEngine.ini contains several DDC entries by default, including NoShared, which is used for #3 above (-ddc=NoShared). 
Q: I am short on disk space. Can I move the Local DDC elsewhere?Yes! Just set an environment variable of UE-LocalDataCachePath to a path of your choice. For example: UE-LocalDataCachePath=d:DDC.
At the command prompt, type 'setx UE-LocalDataCachePath d:DDC'.
Restart Unreal Engine, along with any application you launch it from, such as Epic Launcher, UGS, or Visual Studio. 
This can be used when you are short of space on your primary drive, or when you are working on multiple branches and wish to ensure the data isn't duplicatednote
Q: I have a slow network connection. Can I turn off the Shared DDC?Yes! There are two options, depending on whether you want this to be temporary or permanent: 
What Is Directx Shader CacheIf you set the environment variable for a DDC to None it will disable it. In this case, you would set UE-SharedDataCachePath=None.
Launch the editor with -ddc=noshared.
Q: Can I change the Shared DDC path?Yes! You can change it by setting the UE-SharedDataCachePath mentioned above. 
However, you should never set UE-SharedDataCachePath to a local path! Doing this means you would have both Local and Shared caches on your machine—twice the disk space for zero gain! 
Q: How can I diagnose DDC issues?If you think your editor is not correctly reading DDC data, then search your logfile for LogDerivedDataCache.
Here you can see: 
There's no project or Engine Pak cache, which is what would be expected as I compiled this build myself from Perforce. 
My local data cache path is mapped to d:DDC and is writable. 
I am using the Epic shared cache, which is writable. 
You can also run with -logcmds='LogDerivedDataCache Verbose' to turn on verbose logging. 
Q: What is the S3 DDC?The Fortnite DDC is extremely large due to the amount of content we have. When Covid19 struck, we quickly realized that it was very inefficient for developers to pull all this data from a network share over the limited bandwidth of a VPN to the office, so instead we distribute the most commonly used data via S3. 
Setting up the engine to use the S3DDC is extremely easy, but you will need to set up automation that generates the DDC content and uploads it to S3. 
Q: What is IDDCUtilsModuleInterface?This interface allows you to use ? Name arguments in your DDC settings that can then be resolved dynamically at runtime based on platform or domain. 
Unfortunately, we don't ship an example of this, but implementing it is straightforward: 
What Is Unity Shader CacheClick image for full size.
Q: How do I create a DDC Pak?Creating a project-specific DDC: 
UE4Editor.exe ProjectName -run=DerivedDataCache -fill -DDC=CreatePak
This commandlet will create a DDC for all content in the project. Alternatively, you can simply supply -DDC=CreatePak while running automation or even a user session to generate a more targeted set of content. 
What Is Shader CacheCreating an Engine DDC: 
What Is Shader Cache CemuUE4Editor.exe -run=DerivedDataCache -fill -DDC=CreatePak