Python Development with Bazel on an M1 Mac

Apple’s started its transition to Apple Silicon for computers this year. This means a complete architecture change for macOS. They released a new MacBook Air, a MacBook Pro, and a Mac mini that are all based on their new M1 chip.

As I’ve been using a 2013-late 15" MacBook Pro (8GB memory, 256GB storage) since January 2014 and it was getting a bit sluggish for photo editing and development. As for everyone else, 2020 made me stay home more than I expected, so I’ve decided to upgrade to the new Mac mini. I’ve also accepted to have some of the early-adopter problems…

I’ve a monorepo for all my projects and I use Bazel for building them. As I started to setup my development environment on my Mac and I noticed that some of my Python scripts are broken, the ones relying on native libraries (C extensions). rules_python helps creating Python libraries and executables with Bazel. However, as of November 2020, it uses the system Python and it doesn’t manage a Python interpreter version itself. This makes these Python builds non-hermetic¹ and can lead to issues.

The Problem

The non-hermetic Python interpreter creates a problem on Macs with M1 chip: macOS ships Python interpreters compiled to CPU architecture (arm64), but Bazel is running with Rosetta 2 currently and uses the x86_64 architecture. This makes pip to compile native packages like lxml to the architecture dictated by Rosetta 2, leading to mismatch between the interpreter and the package to be loaded at runtime:

Traceback (most recent call last):
  File "/private/var/tmp/_bazel_user/.../path/to/module.py", line 5, in <module>
    from lxml import etree
ImportError: dlopen(/private/var/tmp/_bazel_user/.../pip/pypi__lxml/lxml/etree.cpython-39-darwin.so, 2): no suitable image found.  Did find:
    /private/var/tmp/_bazel_user/.../pip/pypi__lxml/lxml/etree.cpython-39-darwin.so: mach-o, but wrong architecture
    /private/var/tmp/_bazel_user/.../pip/pypi__lxml/lxml/etree.cpython-39-darwin.so: mach-o, but wrong architecture

A Solution

Requirement: having Homebrew installed under /usr/local for Intel architecture. (It’s recommended to install Homebrew under /opt/homebrew on macOS ARM architecture.)

Install a Python interpreter compiled to x86_64 architecture:

$ arch -x86_64 /usr/local/bin/brew install python@3.9

Specify this Python as a toolchain in Bazel e.g. in the toolchains/python/BUILD.bazel file:

load("@rules_python//python:defs.bzl", "py_runtime", "py_runtime_pair")

py_runtime(
    name = "py3.9_x86_64",
    interpreter_path = "/usr/local/bin/python3.9",
    python_version = "PY3",
)

py_runtime_pair(
    name = "python_x86_64",
    py3_runtime = ":py3.9_x86_64",
)

toolchain(
    name = "python_toolchain_x86_64",
    target_compatible_with = [
        "@platforms//os:macos",
        "@platforms//cpu:x86_64",
    ],
    toolchain = ":python_x86_64",
    toolchain_type = "@rules_python//python:toolchain_type",
)

Register the toolchain in your WORKSPACE file, so rules_python can pick it up:

http_archive(
    name = "rules_python",
    sha256 = "b6d46438523a3ec0f3cead544190ee13223a52f6a6765a29eae7b7cc24cc83a0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.1.0/rules_python-0.1.0.tar.gz",
)

load("@rules_python//python:pip.bzl", "pip_install")

register_toolchains("//toolchains/python:python_toolchain_x86_64")

pip_install(
    requirements = "//:requirements.txt",
)

Now your native dependencies should match the Python interpreter architecture.