Taming Python 3 Migrations using Pants Build 2

See how Pants can help you manage smooth incremental changes in your codebase, such as when migrating to Python 3.

------
Transcript:

One of the things that people recommend, after finishing a big Python 3 migration, is to split up the migration incrementally. Meaning that rather than doing it all in one big pull request, you split it up into a couple of smaller, incremental steps. This is exactly the type of complex change that Pants aims to empower you to do — regardless of the size of your codebase.

Pants has a couple of features specifically to help you with Python 3 migrations. The main way that Pants helps is to allow you to precisely track the status of your migration at the individual file level. So you can have some files that are using only Python 2, some that are using only Python 3, and some files that work with both interpreters.

So we have here, for example, a fairly simple project where we have some files that work with only Python 2, like this that uses the old print statement. We have some that work with Python 3 only. Such as new new/app.py using an fstring. And we have a common util folder that works with both Python 2 and with Python 3. Pants allows us to express this reality exactly through a mechanism called BUILD files. BUILD files are how you express metadata in Pants. They are normally auto-generated files and normally really small, only about one or two lines, thanks to sensible defaults.

BUILD files have an option called interpreter constraints that allows us to say that the code in this folder uses, for example, Python 2.7. We can also show that our common folder, meanwhile, works with both Python two and Python 3. Pants then understands how those specific interpreter constraints fit in with the rest of your code and your project.
So, for example, our legacy app.py file: even though its code only works with Python 2.7. It depends on the file mathutil.py, which works with both Python 2.7 and Python 3. Pants knows how to merge those constraints together and realizes that we need to use Python 2.7 when we're doing things like running our tests or running our linters on legacyapp.py.

Pants understands these dependencies for you automatically. Through a thing called dependency inference, Pants will read your import statements for you and map those imports back to the rest of your project.
With that really fine-grained understanding of what constraints each specific file should use, Pants can then choose the right interpreter for you when you're doing things like running your linters and type checkers and tests. With some tools like black and isort, it doesn't really matter what interpreter you use when you run the tool. Whereas other tools like flake8 and mypy, you have to run with the exact right interpreter for it to work properly. Pants automates that all for you.

So for example, we can run this command `pants lint` and Pants will split up the flake8 run into three different partitions here. That it runs the Python 2 code separately. Then the Python 3 only code, for example. What this means is that we can have some Python 2 only code in our project at the same time that we have Python 3 only code and some code that works with both — without needing to have a hacky bash script that tries to manually partition for us. This also works with test that we can have some tests that work with only in Python 2, some that work with only Python 3. We can even tell Pants to run the same test with both Python 2 and Python 3. Like we do with our common util that we want to make sure works with both interpreters. We can run this command `pants test` and Pants will automatically make sure that each test used the right interpreter.

A final way that Pants helps us with incremental Python 3 migrations is to help prioritize where next to go in your project. Pants can generate a CSV for you with a list of each of your files. And those files' specific interpreter constraints, along with their transitive constraints, when you merge in all of the dependencies of that file. Pants will also calculate for you the number of dependencies — meaning all of the code that file imports and depends on — and the number of dependees — meaning how many other files in your project depend on that specific file. Both of these things are really useful to help with prioritization. The number of dependencies is a good proxy for how hard it is to migrate something. If you depend on a lot of code, it's probably harder to migrate because you need to first migrate all of the dependencies before you migrate that file itself.

Number of dependees is a really useful proxy for how impactful it is to port a file. If you have a lot of dependees, migrating that file means that you'll unblock a lot of the rest of your project.

So all three of these features can be combined to empower you to have incremental Python 3 migrations.