Killing Floor: Custom Weapons, Memory Limits and Asset Refs

In January 2013 the Tripwire forum had a long thread about a problem every heavily modded Killing Floor server ran into: clients crash with "out of virtual memory", and the more custom weapons and zeds the server runs, the more often it happens. The participants included Dave_Scream, poosh (author of ScrnBalance), FluX and Guardiannknight, and the thread ended with a working recipe. This article walks through it.

Why memory runs out

• Killing Floor is a 32-bit Unreal Engine 2.5 game. Without the Large Address Aware flag the process gets only 2 GB of address space, however much RAM the PC has. One player in the thread had 16 GB of RAM and Windows 7 and still crashed every few seconds on some maps.
• Custom content in the original ServerPerks setup is loaded statically. A line such as TraderInventory=3:SVD-cMut.SVDcPickup makes ServerPerks load the pickup class. The pickup links the weapon class, and the weapon links meshes, skins, scope materials, sounds, the 3rd-person attachment, fire modes and projectiles. All of that ends up in memory just to show one row in the trader menu.
• On a server with 100+ weapons, most of them are never bought in a given game. In a 6-player game only about 18 weapons are really in use per wave (2-4 per player plus the knife, 9mm and grenades).

poosh's stress test: spawning 33,756 pickup objects on a big map dropped the game below 1 FPS, but it did not crash or hit the memory limit. Actor instances weigh next to nothing. The cost comes from the multimedia (meshes, textures, sounds) that static links pull in.

Two other observations from the thread:

• obj linkers in the console showed packages from other maps sitting in memory with Lazy=0 on a tiny map (for example MountainPass_T.utx, HillbillyHorror_anim.ukx). Maps reference each other's assets, so loading one map can drag in parts of others.
• Many custom weapons ship uncompressed textures, which multiplies their memory footprint.

The idea: Ref strings instead of static links

Stock KF weapons already work this way. Instead of Mesh=SkeletalMesh'...', the class holds a plain string, MeshRef="Package.Object". A string links nothing, so nothing is loaded until the code calls DynamicLoadObject(). When the weapon is destroyed, the defaults are set back to none and the garbage collector can free the resources.

Static links (old style):

class SVDc extends KFWeapon;

defaultproperties
{
    HudImage=Texture'svd-c_T.SVD-cUnselected'
    SelectedHudImage=Texture'svd-c_T.SVD-cSelected'
    SelectSound=Sound'Svd-c_SN.svd-c_draw'
    Mesh=SkeletalMesh'svd-c_A.svdmesh'
    Skins(0)=Shader'svd-c_T.Svd_D_Sh'
    Skins(1)=Texture'KF_Weapons2_Trip_T.hands.BritishPara_Hands_1st_P'
    ZoomMat=FinalBlend'svd-c_T.SVDScopeFB'
}

The same weapon with refs:

defaultproperties
{
    HudImageRef="svd-c_T.SVD-cUnselected"
    SelectedHudImageRef="svd-c_T.SVD-cSelected"
    SelectSoundRef="Svd-c_SN.svd-c_draw"
    MeshRef="svd-c_A.SVDmesh"
    SkinRefs(0)="svd-c_T.Svd_D_Sh"
    SkinRefs(1)="KF_Weapons2_Trip_T.hands.BritishPara_Hands_1st_P"
    ZoomMatRef="svd-c_T.SVDScopeFB"
}

KFWeapon.PreloadAssets() turns these strings into real objects: it calls DynamicLoadObject() for the mesh, HUD icons, select sound and every entry of SkinRefs, then copies them onto the spawned weapon. If a ref is wrong, the object stays none. The weapon still fires, but it is invisible and silent.

The catch: who calls PreloadAssets

In stock KF, KFGameType.WeaponSpawned() calls KFPlayerController.ClientWeaponSpawned(), and that function is a hardcoded switch over every stock weapon (AA12, AK47, Benelli, Chainsaw, Crossbow and so on). A custom weapon never reaches PreloadAssets(), so simply replacing static links with refs leaves it without a model or sounds.

The fix has to live in the PlayerController. If every weapon author overrode it, weapons would conflict with each other, so the thread settled on doing it at the ServerPerks / ScrnBalance level:

• ScrnBalance already calls PreloadAssets()/UnloadAssets() for any weapon whose default mesh is empty.
• With customised ServerPerks, override ClientWeaponSpawned() and ClientWeaponDestroyed() in KFPCServ.
• With stock ServerPerks you either wait for the feature or run ScrnBalance next to it.

poosh's universal ClientWeaponSpawned(), including the 2013-01-22 fix for the 3rd-person attachment:

simulated function ClientWeaponSpawned(class<Weapon> WClass, Inventory Inv)
{
    local class<KFWeapon> W;
    local class<KFWeaponAttachment> Att;

    W = class<KFWeapon>(WClass);
    // preload only weapons that have no static assets
    if ( W != none ) {
        if ( W.default.Mesh == none )
            W.static.PreloadAssets(Inv);
        Att = class<KFWeaponAttachment>(W.default.AttachmentClass);
        if ( Att != none && Att.default.Mesh == none ) {
            if ( Inv != none )
                Att.static.PreloadAssets(KFWeaponAttachment(Inv.ThirdPersonActor));
            else
                Att.static.PreloadAssets();
        }
        PreloadFireModeAssets(W.default.FireModeClass[0]);
        PreloadFireModeAssets(W.default.FireModeClass[1]);
    }
}

simulated function ClientWeaponDestroyed(class<Weapon> WClass)
{
    local class<KFWeapon> W;
    local class<KFWeaponAttachment> Att;

    W = class<KFWeapon>(WClass);
    // a weapon with a static default mesh keeps its assets
    if ( W != none && W.default.MeshRef != "" && W.static.UnloadAssets() ) {
        Att = class<KFWeaponAttachment>(W.default.AttachmentClass);
        if ( Att != none && Att.default.Mesh == none )
            Att.static.PreloadAssets();
        UnloadFireModeAssets(W.default.FireModeClass[0]);
        UnloadFireModeAssets(W.default.FireModeClass[1]);
    }
}

PreloadFireModeAssets() / UnloadFireModeAssets() call the matching static function on KFFire, KFMeleeFire or KFShotgunFire, then on the projectile class.

Without the 2013-01-22 fix, a client could crash when switching to 3rd-person view (F4) or dying while holding a custom weapon.

Converting a weapon, step by step

1. Weapon class. Move HudImage, SelectedHudImage, SelectSound, Mesh and Skins to HudImageRef, SelectedHudImageRef, SelectSoundRef, MeshRef and SkinRefs. Use the Ref names: one of the first examples in the thread forgot to rename HudImage.
2. Extra assets such as a scope: add your own ref variables and extend the static functions.

var string ZoomMatRef;
var string ScriptedTextureFallbackRef;

static function PreloadAssets(Inventory Inv, optional bool bSkipRefCount)
{
    super.PreloadAssets(Inv, bSkipRefCount);
    default.ZoomMat = FinalBlend(DynamicLoadObject(default.ZoomMatRef, class'FinalBlend', true));
    default.ScriptedTextureFallback = texture(DynamicLoadObject(default.ScriptedTextureFallbackRef, class'texture', true));
    if ( SVDc(Inv) != none ) {
        SVDc(Inv).ZoomMat = default.ZoomMat;
        SVDc(Inv).ScriptedTextureFallback = default.ScriptedTextureFallback;
    }
}

static function bool UnloadAssets()
{
    if ( super.UnloadAssets() ) {
        default.ZoomMat = none;
        default.ScriptedTextureFallback = none;
    }
    return true;
}

3. Attachment. Mesh and AmbientSound become refs.
4. Fire modes. FireSound, StereoFireSound, NoAmmoSound, and MeleeHitSounds for melee weapons.
5. Projectiles. Projectile itself has no PreloadAssets(). Either derive from a stock class that does (CrossbuzzsawBlade, CrossbowArrow, M99Bullet, LAWProj, M79GrenadeProjectile, MP7MHealinglProjectile, PipeBombProjectile) or call the projectile's preload from your fire mode. ShotgunBullet has no such functions either. Among stock shotgun projectiles only the nail has a mesh or sounds, so most custom pellets need nothing.

class MyShotgunBullet extends ShotgunBullet;
var string StaticMeshRef;

static function PreloadAssets()
{
    UpdateDefaultStaticMesh(StaticMesh(DynamicLoadObject(default.StaticMeshRef, class'StaticMesh', true)));
}

static function bool UnloadAssets()
{
    UpdateDefaultStaticMesh(none);
    return true;
}

Pitfalls reported in the thread

Symptom	Cause	Fix
The weapon is invisible and silent	No code calls PreloadAssets() (stock ServerPerks), or a ref is misspelled	Add the PlayerController hook and check every ref string
Other players see no 3rd-person model	The package with the attachment mesh is no longer linked from anything, so it is not sent to clients	Keep at least one static link to every package, for example leave the attachment Mesh static. Calling AddToPackageMap() from the pickup was suggested but not tested
1st-person mesh missing when a player spawns with the weapon, fixed after drop and pick-up; log shows Playanim: Sequence 'Idle' not found for mesh	Assets were not preloaded on the initial spawn path	Make sure the initial inventory goes through the same preload hook
Crash on F4 or on death while holding a custom weapon	Old ClientWeaponSpawned() without the attachment fix	Use the fixed version above
A gun built by copying a stock weapon's code fires and deals damage, is visible in third person, but has no first-person mesh	According to Dr_Killjoy, MeshRef is only supported for stock actors	Use the old Mesh variable ("[HELP] Не отображается меш оружия в игре." (Weapon mesh not shown in game) thread)

ServerPerks adds the weapon's .u file to ServerPackages automatically, and clients download every package that .u links. Once you replace a static link with a string, the linked .ukx/.utx/.uax may stop being sent. Pickups usually keep sounds and textures linked (PickupSound, TraderInfoTexture), which is why the attachment mesh is the most common thing to go missing.

#exec OBJ LOAD FILE=... does not solve this. It only tells ucc which files to load while compiling and is not stored in the compiled .u.

What about zeds and skins

Zeds were left alone on purpose. Zeds in the squad lists will spawn sooner or later anyway, so dynamic loading would only add hitches between waves. If you do not want some custom zeds, do not load their mutator. Zeds that only exist for !rtd or admin summons still cost memory. The thread raised that question but did not solve it.

Character skins (PlayerRecordClass) were also discussed without a ready solution.

Results and limits

• Guardiannknight saw no visible saving on the server, but noticeably fewer client crashes after converting almost all weapons.
• poosh ran Doom 3 monsters, Brute, Shiver, Jason and a big custom weapon list with no memory issues after switching to refs.
• Some crashes are not memory at all. Specific maps (some of Murphy's maps, HarbourV3, Pandora Box) were reported to crash dedicated servers after a map restart. Test with official maps and without custom content before blaming memory.
• "More doesn't mean better": dropping near-duplicates (AK47 / Golden AK47 / MKb42) saves memory and rarely changes gameplay.

Wormbo from the Epic forums warned against a generic mutator that nulls defaults for arbitrary classes. In pre-UE3 engines a garbage-collector run is synchronous and can take seconds, defaults are not reset on map change, and the server may still need asset metadata (sound duration, texture size, bones). The consensus was to convert the weapons themselves.