Decompiling Black Mesa Maps

    Logging in with you Steam ID has now been restored ♥

    • Decompiling Black Mesa Maps


      I had some significant reservations about posting on this topic publicly, but I've concluded that I will not be the only person ever to realize it could be done and more to the point I'm offering a workaround for something (namely, providing up-to-date sources for all all areas of the game) that the developers have stated they themselves plan to do but are understandably too busy to bother with at the moment. That said, please be polite in your decompiling adventures.


      Getting a Decompiled Map
      • The actual maps which Black Mesa runs need to be extracted from the vpk files like any other game asset. This can be done by dragging bms_maps_dir.vpk over the vpk.exe program stored in the game's 'bin' folder.
      • You will need ata4's bspsrc utility. The current best (only?) place to get it is from the releases section of the author's github page.
      • Unzip and run bspsrc.bat.
      • Leave the advanced compilation options as defaults.
      • Add as many maps as you want and decompile. The program will throw a number of "invalid vector" errors.
      • This will produce decompiled VMFs with some bad data in them that will cause Hammer to crash if opened directly.
      Making A Decompiled Map Functional
      • In order to make the VMFs you created up above work, you will need to open them in a text editor like Notepad and remove a block of bad data.
      • The data you are looking for are a bunch of spurious lighting origins lumped together near the bottom of the file. They take the form of blocks like this:
        entity { "id" "<SOME NUMBER>" "classname" "info_lighting" "targetname" "sprp_lighting_<SOME NUMBER>" "origin" "<THREE NUMBERS>" }
        The coordinates in the origin are occasionally something absurd like "0 0 103161719131765890000000000"- it's these Very Large Numbers that break Hammer when the map is loaded. However, all of these lighting origins will cause the props they are connected to to light improperly, and will likely be in a place that will cause map leaks, so there's nothing to be gained by only deleting the weird ones.

      • There are typically several hundred of these entites (Gasworks has nearly two thousand!) but since they are all in a row and the decompiler never includes any "legitimate" lighting origins you can just find the first one, find the last one, select everything in between, and delete.
      • The map can now be opened in Hammer, but I find that the reconstructed optimization (areaportals, occluders, and hint brushes) almost universally have an extremely negative effect on both compile time and in-game performance, and may cause compile errors. So, I typically use the 'auto' visgroups to hide everything except for those three categories, select the entire map area, and delete all of it. This makes a map that is barely playable and will take a long time to compile, but it's better than nothing and you can fill in at least basic optimization yourself.
      • Very rarely other elements may be decompiled in a way that causes errors; it's typically possible to just delete them and make new ones that look the same.
      • If you're looking to update one of the maps like the outdoor ones that only have had new static props added since the official VMF release, you are probably better off just using hammer's "entity report" or visgroup functionality to delete every prop_static from the official map, select every prop_static from a decompiled map, and copy them over. This will let you use the new props with the original optimization.
      Pre-Disaster Planning Thread: Thinking of Restoring a Part of Black Mesa? Come Here First!

      Office Complex PD: I'm the Slowest Pre-Disaster Mapper!

      Pre-Disaster Questionable Ethics: It's here!
    • The current version of bspsrc can't properly read the static prop structure in Black Mesa's BSP format. The version of the static prop structure in Black Mesa's BSP format is identified as V11, which isn't supported by bspsrc, and as a result, it falls back to reading the static prop structure as a V4 structure. This works for reading some of the static prop data, but fails from the point where the two structure versions are different and it will interpret the rest of the data incorrectly. That's what's causing the erroneous data like the spurious lighting origins and invalid numbers that cause Hammer to crash.

      I've made some modifications to the source code of bspsrc to add support for Black Mesa's V11 static prop structure, and I think it's working perfectly now. Decompiled maps can be opened in Hammer without crashing it, and the lighting origins should be correct and other settings should be present in the static prop data. But I need to test it some more. There's one piece of data in the static prop structure that I've ignored, and that's the DiffuseModulation struct member. This corresponds to the "rendercolor" and "renderamt" settings in the .VMF. Hammer will apply the color to the static prop in the 3D view, but you can't edit the settings in the prop's properties, and it doesn't appear to be rendered in-game.

      If anyone wants to try/test my version of bspsrc, it's available for download below along with the modified source code.

      EDIT 2017-07-22:

      Uploaded new version. Changes:
      • Fixed buggy handling of V10 structs. Cleaned up V11 struct code so that it now extends V10.
      Other notes:

      BM's V11 static prop struct also supports prop lightmaps settings, but they don't appear to be handled by BM's vbsp.exe, i.e., the flag that enables it and the Lightmap Resolution X and Y don't get written to the .BSP.

      EDIT 2017-12-31:

      New version. Changes:
      • Added support for the newer V11 prop_static structure format found in the Black Mesa December 2017 Update while maintaining support for the older V11 structure found in surface-tension-update(cu3) and halloween-update(cu4), as well as the V10 structure found in the crossfire-update(cu2) (and likely in older releases). The new structure contains an additional (32-bit) integer (m_FlagsEx) which use is currently unknown, and this data will therefore not be processed by BSPSource.
      • Changed fallback mechanism for unknown static prop lumps so that it falls back to v4 only if the static prop lump version is < 10. If it's >= 10 it will fall back to v10.
      • Added code to sort the list in the BSP format combobox alphabetically for easier selection.
      • Updated file pattern for Black Mesa in the app database to include 'i' maps, eg., bm_c3a2i, and added some new entities to improve auto detection of current and future Black Mesa map files.
      • Added support for displacement surface flags (No Physics Collision, No Hull Collision, No Ray Collision) based on a patch by jason-e.


      Download
      bspsrc_1.3.23_bmfix_r4.zip | Source code (zip)

      Previous version:
      bspsrc_1.3.23_bmfix_r3.zip | Source code (zip)

      The post was edited 2 times, last by Flavrans: Uploaded new version (v1.3.23-bmfix-r4) ().

    • Ummm for some reason bspsrc_1.3.23 won't work, even the bmfix version. As soon as I hit the "decompile" button, the program crashes and leaves an error log. Anyone else getting this problem? :(
      Oddly enough, it worked just fine for me over the summer. Maybe I'm forgetting to do a step...
    • Super Ninja Fat wrote:

      Ummm for some reason bspsrc_1.3.23 won't work, even the bmfix version. As soon as I hit the "decompile" button, the program crashes and leaves an error log. Anyone else getting this problem? :(
      Oddly enough, it worked just fine for me over the summer. Maybe I'm forgetting to do a step...
      Does it crash with any .bsp, or just one specifically? Which version of the Java runtime is installed on your system? (Click the Windows Start Menu button and type "About Java" and then click it/press Enter to see the installed version number.) Perhaps an update broke it. The version I'm using is 8u151, but the latest is 9.0.1. If you are on 9.0.1, then you could try downgrading to 8u151 and see if that works (available here).

      EDIT: I tried running bspsrc 1.3.23 (orig. and bmfix) under JRE-9.0.1, and didn't experience any crashes (decompiling some stock BM maps). So, that's probably not it.

      The post was edited 1 time, last by Flavrans ().

    • Flavrans wrote:

      Super Ninja Fat wrote:

      Ummm for some reason bspsrc_1.3.23 won't work, even the bmfix version. As soon as I hit the "decompile" button, the program crashes and leaves an error log. Anyone else getting this problem? :(
      Oddly enough, it worked just fine for me over the summer. Maybe I'm forgetting to do a step...
      Does it crash with any .bsp, or just one specifically? Which version of the Java runtime is installed on your system? (Click the Windows Start Menu button and type "About Java" and then click it/press Enter to see the installed version number.) Perhaps an update broke it. The version I'm using is 8u151, but the latest is 9.0.1. If you are on 9.0.1, then you could try downgrading to 8u151 and see if that works (available here).
      EDIT: I tried running bspsrc 1.3.23 (orig. and bmfix) under JRE-9.0.1, and didn't experience any crashes (decompiling some stock BM maps). So, that's probably not it.
      Any .bsp file. I've installed the latest version of java. For some reason, About Java isn't working right now (I'll check it again later) but I think it's 9.0.1. So yeah I'll just try downgrading. Thanks!
    • Flavrans wrote:

      FYI, the December Update (aka. the Xengine Update) broke BSPSource again, including my version which included a fix to handle Black Mesa's static prop lump.

      The format of the static prop lump has been changed again, thus causing the same problem as before. I'm looking into it and I'll see if I can patch BSPSource again to handle the new format if/when I find some time.
      Still works for me. I managed to decompile bm_c2a4h_d from the public beta.
    • I've built a new version of bspsrc-bmfix which adds support for the new V11 prop_static structure. See my first post in this thread for download links and changelog.

      So, the static prop lump found in the December update map files is still identifying itself as version 11, like in cu3 and cu4, but has grown by four bytes (from 76 to 80 bytes) compared to the earlier version of the structure. The four additional bytes are used by a new struct member called m_FlagsEx (a 32-bit integer), which I have no idea what is being used for. In the few maps I've checked it's just set to 0 and there's nothing in the .fgd's that might shed some light on what it's supposed to be used for. Perhaps it's something to do with the upcoming improved prop lighting feature (as mentioned in the December update announcement).

      I've just made sure that BSPSource is able to detect the format and able to read the rest of the relevant data from the structure properly.

      Revvylo wrote:

      Still works for me. I managed to decompile bm_c2a4h_d from the public beta.
      Yes, the decompilation would appear to finish successfully, but you would see the following messages in the errors & warnings box:

      Source Code

      1. [warning] BspFileReader: Static prop struct size mismatch: expected 80, got 76 (using DStaticPropV11)
      2. [warning] BspFileReader: Falling back to static prop v4
      This means that static props have been read incorrectly, and will have some properties that will likely be wrong. Most if not all static props will have an invalid lighting origin defined, even if they are not supposed to have a lighting origin, and those that are supposed to, will have lighting origin coordinates filled with invalid numbers that might crash Hammer upon opening the .vmf. This has now been fixed in my new build of bspsrc-bmfix.
    • TJHL wrote:

      This doesn't work for me. When i launch the .bat, an error pops up saying "Windows cannot find 'javaw' ".
      You need to download and install the Java SE Runtime Environment:

      oracle.com/technetwork/java/ja…10-downloads-4417026.html

      The modified version of bspsrc I posted in this thread was built in a Java 8 environment. If there's a problem with running bspsrc under JRE 10, uninstall the JRE 10 and try the JRE 8 (8u171) instead:

      oracle.com/technetwork/java/ja…e8-downloads-2133155.html