2021-08-22
When your Go modules become too large.
What can you do when your Go module becomes too large for the Go tools to handle?!
When you see this?
module source tree too large (max size is 524288000 bytes)
Go modules are a convenient to share code. You define a module with its version in go.mod and to Go tools automatically take care of downloading the right version, checking its integrity and compiling it.
CGo enables integrating non-Go code by either allowing you to compile C code or just link to external libraries and call functions exported there through a convenient interface.
Let’s say you have an externally compiled library that you want to distribute with your Go module but still use the usual Go tools. Adding a compilation step can’t happen, since that would require invoking, and perhaps installing, external tools, which the Go tools doesn’t support.
The solution is to commit your library as part of the Go module codebase, which gets downloaded together with the rest of the source code.
Now, let’s say you want to support multiple platforms. Fortunately, Go has built in capabilities to include source files based on the platform compiling, so we can:
This works reasonably well, with the caveat that the Go module the tools need to download becomes larger as a result.
What happens when the library become too large? When it’s a static library that didn’t optimize for size and relied on consumers just taking what they need, which the Go tools do? This is what you see if your Go module is too large:
module source tree too large (max size is 524288000 bytes)
Concretely, if you want to support Linux, macOS, Windows on x86_64, ARMv7, ARMv8 and s390x, where applicable, that becomes a big set of libraries that can exceed 500mb.
Why is that important? Because the maximum module size Go supports is 500mb. This is intentional to prevent denial-of-service attacks, as mentioned in https://golang.org/ref/mod#zip-path-size-constraints.