There's a neat writeup I stumbled across recently titled "Reproducible codesigning on Apple Silicon" from Keith Smiley about some gotchas when it comes to compiling a binary in a way that's repeatable and always generates the exact same byte output (which would then checksum to the exact same hash) - even if compiled on a different Mac.
In applying the suggestions I found in the blog post, I found a few other corner cases that I just wanted to get documented more explicitly somewhere.
Footnote 2 from that blog post is important:
As long as you're using the same version of clang
This can be challenging when people are trying to collaborate and have different hardware, because different harware may carry different minimum and maximum OS versions - each of which control which versions Xcode or the Command Line Tools are available for downloading and running.
You can partially mitigate this by pinning your build instructions to a specific standalone pre-compiled Clang version from the LLVM Project releases.
For example, Xcode 14 released with Clang 14 and supports a minimum macOS version of 12.5, its latest version as of writing this supports only macOS 13 or later. The LLVM Project's release of Clang 14, however, as built will technically run all the way back to at least macOS 11.6 without having to recompile it from source:
cmd LC_BUILD_VERSION
cmdsize 32
platform 1
minos 11.6
While this can help greatly with reproducibility, it can make compiling a little more challenging in that the copy of Clang shipped with Xcode is already well-informed about where headers, frameworks, and SDKs are, so you'll have to provide it a little bit more information.
Let's look at this very simple program main.c:
#include <stdio.h>
int main()
{
printf("Hello, world!\n");
return 0;
}With Xcode / CLI tool compilation, you can simply do:
clang -o output main.c
But if you try that with the Clang distributed in the LLVM Project releases, you'll probably see:
main.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
^~~~~~~~~
stdio.h is an extremely common header - but without some help, this version of Clang won't automatically find the one you want to use. You can help it find the common paths on macOS simply by setting the SDKROOT environment variable to a place where you've got an SDK installed:
SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.sdk clang -o output main.c
In addition to which Clang you're relying on having impact on what hardware and OSes you can build it on, you have to consider that each new version of Clang contains new tricks and changes about how it compiles code. Some things it's better to be explicit about when using it so that users trying to reproduce your results don't accidentally have conflicting settings.
If you don't specify a target triplet argument (like: -target x86_64-apple-macosx10.11.0), Clang will analyze your machine and try to build one, but this will incorporate things such as the current major OS you're on. If that OS is newer than the OSes your project supports, it may cause Clang to focus on optimizations compatible with "Intel processor instructions found on Intel Macs that can run macOS 13 or later" instead of "Intel processor instructions found on Intel Macs that can run macOS 10.11 or later". Using the target triplet mentioned earlier would disable SSE 4.1, for example.
And while optimization level settings like -O1 vs. -O0 have some impact on your project and will definitely change the code that gets generated (maybe pick one explicitly, just so people avoid accidentally setting a different one), the bigger impact still lies in using the same version of Clang. What optimizations are available - and what even the "no optimization" setting means - varies from release to release, all of which will change what instructions get written in the end.
Maybe you've done the work and tried to stick to APIs that have been broadly available and unchanged for your project across all the OS versions you support.
But even though any one of the SDKs within that OS version range would work when it comes to compiling and eventually linking your final project, the one that you actually use during compiling has side effects which are recorded in your binary as metadata in a few different places.
One of the places is in the link commands LC_VERSION_MIN_MACOSX and its modern replacement LC_BUILD_VERSION (macOS 11+)
If you rely on the Xcode or Command Line Tools provided Clang, you have some basic controls over the metadata that gets recorded in these, but even if you set your minimum macOS version and target triplet to the oldest OS versions you support, notice the following output:
% clang -o main.10.15 -target x86_64-apple-macosx10.15.0 -mmacos-version-min=10.15.0 main.c
% vtool -show main.10.15
main.10.15:
Load command 10
cmd LC_BUILD_VERSION
cmdsize 32
platform MACOS
minos 10.15
sdk 13.1
ntools 1
tool LD
version 820.1
Load command 11
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
% clang -o main.11 -target x86_64-apple-macosx11.0.0 -mmacos-version-min=11.0.0 main.c
% vtool -show main.11
main.11:
Load command 10
cmd LC_BUILD_VERSION
cmdsize 32
platform MACOS
minos 11.0
sdk 13.1
ntools 1
tool LD
version 820.1
Load command 11
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
The specific version of ld that was used to link the binary was recorded in the command, as well as the SDK version at the time of linking. Even though every API in the binary you built may support and run on an older OS, the SDK you used at the time has made its mark on the contents of your binary.
The same goes for the versions of the frameworks and libraries that get linked to your final binary.
SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.0.sdk clang -o main.sdk.12 -target x86_64-apple-macosx12.0.0 -mmacos-version-min=12.0.0 main.c
$ otool -l main.sdk.12 | grep -B2 -A3 libSystem
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Wed Dec 31 16:00:02 1969
current version 1311.0.0
compatibility version 1.0.0
SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.sdk clang -o main.sdk.13 -target x86_64-apple-macosx12.0.0 -mmacos-version-min=12.0.0 main.c
$ otool -l main.sdk.13 | grep -B2 -A3 libSystem
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Wed Dec 31 16:00:02 1969
current version 1319.0.0
compatibility version 1.0.0
Even though the "versions of this library I'm compatible with" field is set very low to 1.0.0 - compatible with just about any OS version out there - and even though the binary has the minimum compatible OS version recorded elsewhere in LC_VERSION_MIN_MACOSX or LC_BUILD_VERSION, the binary will still have a recording of the specific library version that it saw in the SDK at the time you linked it into the final executable. Which means the hash of the binary will potentially be different.
It's mentioned at the end of the original blog post as an Update footnote, but the LC_UUID load command is something that's added by default to binaries generated by Clang. And not only does it complicate ad-hoc codesigning, as mentioned - but yes - once again, it can change the contents/hash of your binary.
The Clang shipped with Xcode and the Command Line Tools does a good job in generating a deterministic UUID in that it fills the binary initially with a zero byte filled UUID, hashes the binary with that in place, then derives the true UUID and overwrites the placeholder. You also have the option of passing in the -no_uuid flag and saying "please don't add this load command", which can have some side effects if you need to debug the process but can definitely result in an even more predictable binary.
Unfortunately the open source LLVM Project doesn't seem to do either of those things. The UUID it generates seems to still be semi-random, and the -no_uuid flag is not actually currently supported.
So let's take everything I've said here and see if we can get you the reader a working reproducble Universal 2 ad-hoc codesigned binary example that has the exact same hash as mine - without me giving you the binary directly, regardless of you building it on an Intel Mac or a modern Apple silicon Mac.
Here's the code we'll be using:
myhash.m:
#import <Foundation/Foundation.h>
#import <CommonCrypto/CommonDigest.h>
int main(int argc, const char * argv[]) {
@autoreleasepool {
NSString *myPath = NSProcessInfo.processInfo.arguments[0];
NSURL *myURL = [NSURL fileURLWithPath:myPath];
NSData *myData = [NSData dataWithContentsOfURL:myURL];
unsigned char myDigest[CC_SHA1_DIGEST_LENGTH];
if (CC_SHA1([myData bytes], [myData length], myDigest)) {
NSMutableString *myHex = [NSMutableString string];
for (int i=0; i<CC_SHA1_DIGEST_LENGTH; i++) {
[myHex appendFormat:@"%02x", myDigest[i]];
}
printf("My SHA1 hash is: %s\n", [myHex UTF8String]);
}
}
return 0;
}It's a simple program that, when you run it, tries to figure out where its own file is and then prints you the SHA1 sum of the bytes of that file (which should match the output of /usr/bin/shasum /path/to/compiled/binary).
For our compiling toolchain, let's use LLVM Project Clang version 14.0 - clang+llvm-14.0.0-x86_64-apple-darwin.tar.xz (which can build both x86_64 and arm64)
% shasum clang+llvm-14.0.0-x86_64-apple-darwin.tar.xz
7327a95e683d263721edc47f8f43fb42e1f32026 clang+llvm-14.0.0-x86_64-apple-darwin.tar.xz
So I've created a directory named "myhash" in my home folder, put myhash.m in there, and expanded the .tar.xz download as a folder named "llvm-14", so that I have /Users/frogor/myhash/llvm-14/bin/clang, etc.
For the goal of the project, I want to do minimum OS version macOS 11 and use the macOS 11.1 (20C63) SDK. Obtaining a specific SDK version requires downloading a specific Xcode release. It can be challenging to know which version is paired with which Xcode release, but if you check through the release notes you can find 11.1 shipped with the Xcode 12.3 release. You can download the releases from Apple's Developer site additional downloads. Even if you're not paying for a Developer account, you can use a free Apple ID to sign into the site and obtain older Xcode releases.
The first machine I'm going to build this on is running macOS 13.2.1, which Xcode 12.3 won't run on. That's ok, we just need the SDK inside it. So after downloading and expanding the .xip, right/control-click on it, look inside and pull out the SDK at Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk (which MacOSX11.1.sdk inside there is a symlink to), and drop it right into the root of the myhash folder.
Setting up environment variables:
export PATH="$HOME/myhash/llvm-14/bin:$PATH"
export SDKROOT="$HOME/myhash/MacOSX.sdk"
Let's emit LLVM's IR - make sure we're all starting on the same step:
clang -S -emit-llvm -target x86_64-apple-macos11.0.0 -O0 -o myhash.ll myhash.m
A few choice lines from the .ll file:
attributes #4 = { "frame-pointer"="all" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="penryn" "target-features"="+cx16,+cx8,+fxsr,+mmx,+sahf,+sse,+sse2,+sse3,+sse4.1,+ssse3,+x87" "tune-cpu"="generic" }
...
!0 = !{i32 2, !"SDK Version", [2 x i32] [i32 11, i32 1]}
...
!10 = !{!"clang version 14.0.0 (https://github.com/tru/llvm-release-build fc075d7c96fe7c992dde351695a5d25fe084794a)"}
Penryn CPU target, SDK version 11.1, Clang 14.0.0 release build fc075d7c96fe7c992dde351695a5d25fe084794a. Looking good.
% shasum myhash.ll
d15b9f26b607f918d41dbf5f1bbc0ae49a4b3f1a myhash.ll
Let's move from IR to assembly:
llc -o myhash.s -O0 myhash.ll
Spot check on some of the lines:
.build_version macos, 11, 0 sdk_version 11, 1
SDK 11.1 - still looking good.
% shasum myhash.s
c8a531c1b78ad9fb61becf332191c2dff11e8654 myhash.s
Let's convert the code, before linking, to a standalone object:
clang -target x86_64-apple-macos11.0.0 -O0 -c -o myhash.o myhash.s
No LC_UUID yet, but let's check the other commands:
% vtool -show myhash.o
myhash.o:
Load command 1
cmd LC_BUILD_VERSION
cmdsize 24
platform MACOS
minos 11.0
sdk 11.1
ntools 0
Looking good so far.
% shasum myhash.o
b5194fee04c2bd03b3e8b9e30a27b1a82e5d464d myhash.o
For this last step - because of the LC_UUID issues mentioned above - we're going to use the ld from the Xcode I have on this Mac already (Xcode 14.2):
% ld -v
@(#)PROGRAM:ld PROJECT:ld64-820.1
This is the riskiest part for change and variation here, but necessary for now until the LLVM Project fixes some outstanding issues.
ld doesn't understand -target, so we're going to have to split the triplet out with:
-arch x86_64 -macos_version_min 11.0 -platform_version macos 11.0 11.1
It also doesn't understand SDKROOT, so we're going to have to inform the -I include, -L library, and -F framework paths. And even though LC_UUID would be deterministic here, we're also going to use -no_uuid, and I'll explain why shortly.
ld -arch x86_64 -macos_version_min 11.0 -platform_version macos 11.0 11.1 -no_uuid -L "$SDKROOT/usr/lib" -lSystem -F "$SDKROOT/System/Library/Frameworks" -framework Foundation -o myhash myhash.o
If you run otool -L on the resulting myhash, you should see matching version numbers across the board assuming you used SDK 11.1:
% otool -L myhash
myhash:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
Now let's check some of those load commands:
% vtool -show myhash
myhash:
Load command 9
cmd LC_BUILD_VERSION
cmdsize 32
platform MACOS
minos 11.0
sdk 11.1
ntools 1
tool LD
version 820.1
Load command 10
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
While minos looks right (11.0), and sdk is correct (11.1) - notice that LD has snuck in its own version into the load command metadata.
This is unfortunate. This means that the version "820.1" is now included as part of the calculation, which would have influenced LC_UUID - which is why we turned off generating one.
"How about using ld from the LLVM Project download?" That would be lld.l64 in this case.
If we tried to use the arguments as-is:
ld64.lld -arch x86_64 -macos_version_min 11.0 -platform_version macos 11.0 11.1 -no_uuid -L "$SDKROOT/usr/lib" -lSystem -F "$SDKROOT/System/Library/Frameworks" -framework Foundation -o myhash myhash.o
We get a lot of warnings and errors, like:
ld64.lld: warning: Option `-no_uuid' is not yet implemented. Stay tuned...
ld64.lld: warning: /usr/lib/system/libsystem_kernel.dylib has version 13.2.0, which is newer than target minimum of 11.0
ld64.lld: warning: /usr/lib/system/libsystem_platform.dylib has version 13.2.0, which is newer than target minimum of 11.0
ld64.lld: warning: /usr/lib/system/libsystem_pthread.dylib has version 13.2.0, which is newer than target minimum of 11.0
...
ld64.lld: error: unable to locate re-export with install name /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
The errors are a bit of a known quirk here with how Foundation links to CoreFoundation and are something we can fix by just providing another path for linking to find CoreFoundation directly:
ld64.lld -arch x86_64 -macos_version_min 11.0 -platform_version macos 11.0 11.1 -L "$SDKROOT/usr/lib" -L "$SDKROOT/System/Library/Frameworks/CoreFoundation.framework" -lSystem -F "$SDKROOT/System/Library/Frameworks" -framework Foundation -o myhash.o
So - this looks better:
% vtool -show myhash
myhash:
Load command 10
cmd LC_BUILD_VERSION
cmdsize 32
platform MACOS
minos 11.0
sdk 11.1
ntools 1
tool LD
version 14.0
It now has the tool ld version of 14.0, from the toolset we downloaded - but now it also has a LC_UUID:
% otool -l myhash | grep LC_UUID -A2
cmd LC_UUID
cmdsize 24
uuid 4C4C4418-5555-3144-A1D8-C96F1C7DB394
But if I take these exact same steps over to another machine - this time an M1 Mac Mini running macOS 12.6.3 - and repeat them all via Rosetta 2, I get a hash match at every step of the way until the final binary is generated, because this is the output I receive there:
% otool -l myhash | grep LC_UUID -A2
cmd LC_UUID
cmdsize 24
uuid 4C4C445C-5555-3144-A1A5-7D16148136E8
And if we open up those two binaries in Hex Fiend and compare them, literally the only difference, byte for byte, between these two binaries are the bytes in the UUID themselves.
So what do we do?
We have three options:
- Figure out how to strip out just the
LC_UUIDload command (I don't know how to do this one off the top of my head) - Fix the
LC_UUIDbytes so that they're deterministic, not semi-random, when using the LLVM Project (pretty easy - the UUID bytes are right there in-order within the executable - but would have to build a tool for it) - Figure out how to fix the
LC_BUILD_VERSIONso it either doesn't mention LD version or it's a version we control (we can do this one really easy!)
The vtool which I've been using during this all is a part of Xcode since version 11 or so, and in addition to showing information - it can also rewrite it. And it does it in a pretty predictable way.
So let's go back to the step where we use the ld that comes with Xcode / Command Line Tools:
ld -arch x86_64 -macos_version_min 11.0 -platform_version macos 11.0 11.1 -no_uuid -L "$SDKROOT/usr/lib" -lSystem -F "$SDKROOT/System/Library/Frameworks" -framework Foundation -no_uuid -o myhash myhash.o
On both Macs, running myhash we get:
% ./myhash
My SHA1 hash is: e9886884d5a1468b8f34f6f53a031b21d3bb0be2
Great start, but remember it still has the too-specific LD version encoded in it.
Now let's run vtool and have it rewrite the information and output a new myhash-mod:
% vtool -set-build-version macos 11.0 11.1 -tool ld 14.0 -output myhash.x86_64 myhash
% vtool -show myhash.x86_64
myhash.x86_64:
Load command 9
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
Load command 17
cmd LC_BUILD_VERSION
cmdsize 32
platform MACOS
minos 11.0
sdk 11.1
ntools 1
tool LD
version 14.0
You'll notice it's no longer load command 9 for the LC_BUILD_VERSION - that's been removed and now it's a new load command 17 which has been added to the end.
Let's run the binary on the iMac Pro and see what it outputs:
% ./myhash.x86_64
My SHA1 hash is: 15fb1ff3731c29c729e14fd6a03a7a8a63359def
And on the Mac Mini?
% ./myhash.x86_64
My SHA1 hash is: 15fb1ff3731c29c729e14fd6a03a7a8a63359def
We now have a completely generic x86_64 binary that we've built from source on two completely different OSes and hardware returning the same output.
And now let's go one step further and ad-hoc codesign what is, so far, an x86_64-only binary:
codesign -s - --identifier org.frogor.myhash myhash.x86_64
Remember - Keith's original blog post mentioned we need to pick an identifier to use here, otherwise aspects of the codesigning would be based on filename and other potentially variable details.
So if we do this, do we get the same result now on both Macs?
% ./myhash.x86_64
My SHA1 hash is: 016d590e2bd8a30f197fc2e0832d75e9b317ef5f
% codesign --display -vvv myhash.x86_64
Executable=/Users/frogor/myhash/myhash.x86_64
Identifier=org.frogor.myhash
Format=Mach-O thin (x86_64)
CodeDirectory v=20400 size=586 flags=0x2(adhoc) hashes=13+2 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=e5a9f3a9aa5dc7ed6599b35db0fb7af69b0764d4
CandidateCDHashFull sha256=e5a9f3a9aa5dc7ed6599b35db0fb7af69b0764d4f3a003220b89913da73e024f
Hash choices=sha256
CMSDigest=e5a9f3a9aa5dc7ed6599b35db0fb7af69b0764d4f3a003220b89913da73e024f
CMSDigestType=2
Launch Constraints:
None
CDHash=e5a9f3a9aa5dc7ed6599b35db0fb7af69b0764d4
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12
100% identical on both machines, in all ways.
So let's make it universal 2 now!
We mostly repeat the same steps as before, just this time making a few changes to the commands, replacing x86_64 with arm64:
clang -S -emit-llvm -target arm64-apple-macos11.0.0 -O0 -o myhash.arm64.ll myhash.m
SHA1: ad50e74207dfa64c1ab3afe084789b2c1f818a09
llc -o myhash.arm64.s -O0 myhash.arm64.ll
SHA1: ab45b353a590549107117f008f485887839ecfa3
clang -target arm64-apple-macos11.0.0 -O0 -c -o myhash.arm64.o myhash.arm64.s
SHA1: d68c1dfab3d55b09a96cecefa3dc626a0afae4be
Now this is where for arm64 we have to remember that when we link the binary, we have to tell it to not attempt to automatically ad-hoc codesign here for the arm64 build (which it normally would by default):
ld -arch arm64 -macos_version_min 11.0 -platform_version macos 11.0 11.1 -L "$SDKROOT/usr/lib" -lSystem -F "$SDKROOT/System/Library/Frameworks" -framework Foundation -no_uuid -no_adhoc_codesign -o myhash-unmod.arm64 myhash.arm64.o
At this point, we can't run myhash-unmod.arm64 on the iMac Pro because it can't run arm64 binaries. And we can't run it on the M1 Mac Mini, because Apple silicon Macs have a platform restriction that all arm64 binaries must be ad hoc codesigned at a minimum.
But that's ok, because we have to fix the binary anyways. So let's do that and then ad-hoc codesign it:
vtool -set-build-version macos 11.0 11.1 -tool ld 14.0 -output myhash.arm64 myhash-unmod.arm64
codesign -s - --identifier org.frogor.myhash myhash.arm64
Now that we've done that, we can run the binary on the M1 at least:
% ./myhash.arm64
My SHA1 hash is: aa2330137ef77882b751aaf9e986e08cd2e1d87b
% shasum myhash.arm64
aa2330137ef77882b751aaf9e986e08cd2e1d87b myhash.arm64
And even though we can't execute the binary on the iMac Pro - let's see what we get for SHA1 hash at least:
% shasum myhash.arm64
aa2330137ef77882b751aaf9e986e08cd2e1d87b myhash.arm64
Now, to make the universal 2 binary - we need to lipo the two architectures together:
lipo -create myhash.x86_64 myhash.arm64 -output myhash
In the end, here are your final results which match on both the iMac Pro and the M1 Mac Mini - each of which were running different versions of macOS and independently built these binaries from their own file copies:
% file myhash
myhash: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
myhash (for architecture x86_64): Mach-O 64-bit executable x86_64
myhash (for architecture arm64): Mach-O 64-bit executable arm64
% shasum myhash
112fb78f796f0deb2cb21562d537c350250209f7 myhash
% ./myhash
My SHA1 hash is: 112fb78f796f0deb2cb21562d537c350250209f7
Hopefully if you followed through it all this far, you've reached the same results as I have!
... if you haven't ...
Even though both Macs were running different versions of macOS and one had Xcode 14.2 installed and the other had only the Command Line Tools installed - they both relied on ld from those releases because that ld was able to suppress the LC_UUID command and the LLVM Project's ld64.lld was not - and ld64.lld was generating different UUIDs on different platforms, even though the entire binary was completely identical.
As such, if you received different results than I did anywhere in your hashes above during the linking steps - it may be because you were using a different ld from Apple than version 820.1 which resulted in a different linker ouput.
It would be much better to rely on the LLVM Project from a reproducible portability standpoint, but in order to do that you'd probably need to develop a tool that would overwrite the LC_UUID output to the same value (or remove the command altogether). I know one of the bugs mentioned that got fixed had to do with file names - possibly I got different UUIDs because the user account names (and thus the home folder names) were different on each Mac. Who knows, I haven't really dug into it yet.
I hope you found this interesting, even if a bit wordy. I just needed to get this written out somewhere so that my brain would simmer down a bit.
