pip install pytorch
Environment broken
Spend 4 hours fixing python environment
pip install Pillow
Something something incorrect cpu architecture for your Macbook
Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python
pip install … oh time to go home!
If you're still doing ML locally in 2024 and also use an ARM macbook, you're asking for trouble.
Funnily, the only real competitor for Nvidias' GPUs are Macbooks with 128GB of RAM.
And they don't compete in performance.
I see your contemporary hardware choices and raise you my P900 ThinkStation with 256GB of RAM and 48 Xeon cores. Eventually it might even acquire modern graphics hardware.
Can you expand on this a bit? My recent experiences with MLX have been really positive, so I'm curious what footguns you're alluding to here.
(I don't do most of my work locally, but for smaller models its pretty convenient to work on my mbp).
MPS implementations generally lag behind CUDA kernels, especially for new and cutting edge stuff. Sure, if you're only running CPU inference or only want to use the GPU for simple or well established models, then things have gotten to the point where you can almost get the plug and play experience on Apple silicon. But if you're doing research level stuff and training your own models, the hassle is just not worth it once you see how convenient ML has become in the cloud. Especially since you don't really want to store large training datasets locally anyways.
Nahh im l33t - intel macbook and no troubles.
can't read? parent clearly says "ARM macbook".
For real
What can I say, I enjoy pain!?
I wish my company would understand this and let us use something else. Luckily, they don't really seem to care that I use my Linux based gaming machine most of the time
Maybe pip should not work by default (but python -m venv then pip install should)
Legends say there were times when you'd have a program.c file and just run cc program.c, and then could just execute the compiled result. Funny that programmer's job is highly automatable, yet we invent ourselves tons of intermediate layers which we absolutely have to deal with manually.
And then you'd have to deal with wrong glibc versions or mysterious segfaults or undefined behavior or the the code assuming the wrong arch or ...
python solves none of those issues. It just adds a myriad of ways those problems can get to you.
All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.
In theory you're right, CPython is written in C and it could segfault or display undefined behavior. In practice, you're quite wrong.
It's not really much of a counterargument to say that Python is good enough that you don't have to care what's under the hood, except when it breaks because C sucks so badly.
I was specifically talking about python packages using C. You type "pip install" and god knows what's going to happen. It might pull a precompiled wheel, it might just compile and link some C or Fortran code, it might need external dependecies. It might install flawlessly and crash as soon as you try to run it. All bets are off.
I never experienced CPython itself segfault, it's always due to some package.
I agree simplicity is king. But you're comparing making a script using dependencies and tooling for those dependencies and a C program with no dependencies. You can download a simple python script and run it directly if it has no dependencies besides stdlib (which is way larger in python). That's why I love using bottle.py by example.
Agree. But even with dependencies running "make" seems to be way simpler than having to install particular version of tools for a project, making venv and then picking versions of dependencies.
The point is the same - we had it simpler and now, with all capabilities for automation, we have it more complex.
Frankly, I suspect most of the efforts now are spent fighting non-essential complexities, like compatibilities, instead of solving the problem at hand. That means we create problems for ourselves faster than removing them.
You can do the same in python if you're not new to programming.
I actually did a small C project a couple of years ago, the spartan simplicity there can have its own pain too, like having to maintain a Makefile. LOL. It’s swings and roundabouts!
Some Linux distros are moving that way, particularly for the included Python/pip version. My Arch Linux already does so some years, and I do not set it up myself - so I think it is default.
Can recommend using conda, more specifically mambaforge/micromamba (no licensing issues when used at work).
This works way better than pip, as it does more checks/dependency checking, so it does not break as easily as pip, though this makes it definitely way slower when installing something. It also supports updating your environment to the newest versions of all packages.
It's no silver bullet and mixing it with pip leads to even more breakages, but there is pixi [0] which aims to support interop between pypi and conda packages
[0] https://prefix.dev/
I had a bad experience with Conda:
- If they're so good at dependency management, why is Conda installed through a magical shell script?
- It's slow as molasses.
- Choosing between Anaconda/Miniconda...
When forced to use Python, I prefer Poetry, or just pip with freezing the dependencies.
The Python people probably can't even imagine how great dependency management is in all the other languages...
Mamba/micromamba solves the slowness problem of conda
To add. Conda has parallelized downloads and is faster. Not as fast as mamba, but faster than previously. pr merged sep 2022 -> https://github.com/conda/conda/pull/11841
I absolutely hate conda. I had to support a bunch of researchers who all used it and it was a nightmare.
rye and uv, while "experimental", are orders of magnitude better than poetry and pip IMHO.
Yeah, I agree, maybe I should have also mentioned the bad things about it, but after trying many different tools that's the one that I stuck with, as creating/destroying environments is a breeze once you got it working and the only time my environment broke was when I used pip in that environment.
Yep, I wish I could use another language at work.
I went straight with mamba/micromamba as anaconda isn't open source.
Whenever anyone recommends conda I automatically assume they don't know much about python.
Yes I started with conda I think and ended up switching to venv and I can’t even remember why, it’s a painful blur now. It was almost certainly user error too somewhere along the way (probably one of the earlier steps), but recovering from it had me seriously considering buying a Linux laptop.
This happened about a week ago
I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc.
Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist?
Are python engineers just bad, or?
(Background: I never used python except for one time when I took a coursera ML course and was immediately assaulted with conda/miniconda/venv/pip/etc etc and immediately came away with a terrible impression of the ecosystem.)
Two problems intersect:
- You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility
This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above.
There is no good solution to the above problem.
This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained.
The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library.
Most of the tooling follows approach 1 or 4
Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv.
I threw it online at https://github.com/fragmede/python-wool/
You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions.
Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE
Just replace python with mac or apple in your comment and I think you will understand.
Unironically yes, it really is that bad. A moderately bad language that happened to have some popular data science and math libraries from the beginning.
I can only imagine it seemed like an oasis to R which is bottom tier.
So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses…
things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.
It's not bad. It works really well. There's always room for improvement. That's technology for you. Python probably does attract more than it's fair share of bad engineers, though.
I think it is worth separating the Python ML ecosystem from the rest. While traditional Python environment management has many sore points, it is usually not terrible (though there are many gotchas still-to-this-day-problems that should have been corrected long ago).
The ML system is a whole another stack of problems. The elephant in the room is Nvidia who is not known for playing well with others. Aside from that, the state of the art in ML is churning rapidly as new improvements are identified.
Do people doing ML/DS not use conda anymore?
A lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction (https://github.com/google/XNNPACK/issues/6389). Except the thing was, that wouldn't happen when building outside of of the Conda environment. Very confusing. Standard virtualenvs are boring but nearly always work as expected in comparison.
I'm an Applied Scientist vs. ML Engineer, if that matters.
It's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler...
Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.
I do all the time, and always have (in fact my first job was bare metal OS install automation), this was Rocky 9.4. New codebase, new compiler weird errors. I did actually reinstall and switch over to Ubuntu 24.04 after that issue lol.
If they are they should stop.
It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community.
Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.
Fully agree with this. Understand the basic tools that currently exist and you'll be fine. Conda constantly fucks things up in weird hard to debug ways...
I can't point at a single reason, but I got sick of it.
The interminable solves were awful. Mamba made it better, but can still be slow.
Plenty of more esoteric packages are on PyPI but not Conda. (Yes, you can install pip packages in a conda env file.)
Many packages have a default version and a conda-forge version; it's not always clear which you should use.
In Github CI, it takes extra time to install.
Upon installation it (by default) wants to mess around with your .bashrc and start every shell in a "base" environment.
It's operated by another company instead of the Python Software Foundation.
idk, none of these are deal-breakers, but I switched to venv and have not considered going back.
You could learn how to use Python. Just spend one of those 4 hours actually learning. Imagine just getting into a car and pressing controls until something happened. This wouldn't be allowed to happen in any other industry.
Could you be a bit more specific about what you mean by "You could learn how to use python"? What resources would you recommend to learn how to work around problems the OP has? What basic procedures/resources can you recommend to "learn python"? I work as a software developer alongside my studies and often face the same problems as OP that I would like to avoid. Very grateful for any tips!
Basically just use virtual environments via the venv module. The only thing you really need to know is that Python doesn't support having multiple versions of a package installed in the same environment. That means you need to get very familiar with creating (and destroying) environments. You don't need to know any of this if you just use tools that happen to be written in Python. But if you plan to write Python code then you do. It should be in Python books really, but they tend to skip over the boring stuff.
How do you learn anything in the space of software engineering? In general, there are many different problems and even more solutions with different tradeoffs.
To avoid spending hours on fixing broken environments after a single "pip install", I would make it easy to rollback to a known state. For example, recreate virtualenv from a lock requirements file stored in git: `pipenv sync` (or corresponding command using your preferred tool).
Oh wow, I’ve been a Python engineer for over a decade and getting dependencies right for machine learning has very little to do with Python and everything to do with c++/cuda
I've done it. Isn't it just following instructions? What part of that means destroying every mention of Python on the system?
I've been programming with python for decades and the problem they are describing says more about the disastrous state of python's package management and the insane backwards compatibility stance python devs have.
Half of the problems I've helped some people solve stem from python devs insisting on shuffling std libraries around between minor versions.
Some libraries have a compatibility grid with different python minor versions, because how often they break things.
I thought most ML engineers use their laptops as dumb terminals and just remote into a Linux GPU server.
Yeah, the workday there looks pretty similar though, except that installing pytorch and pillow is usually no problem. Today it was flash-attn I spent the afternoon on.
Isn't this what containers are for. Someone somewhere gets it configured right and then you download and run pre setup container and add your job data ? Or am I looking at the problem wrong?
But then how do you test out the latest model that came out from who knows where and has the weirdest dependencies and a super obscure command to install?
Just email all your data to the author and ask them to run it for you.
Spoiler: my main role isn’t ML engineer :) and that doesn’t sound like a bad idea at all
Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.
Here's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/
Since then the whole python ecosystem has gotten worse.
We are building towers on quicksand.
It's not about python, it's about people who don't care about dependencies.
Dependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.
Do you know what other ecosystem comes closest to the existing in Python? I've heard good things about Julia.
13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends).
There is also a lot of development in Elixir ecosystem around the subject [1].
[1](https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instruc...)
As an amateur game engine developer, I morosely reflect my hobby seems to actually consist of endlessly chasing things that were broken by environment updates (OS, libraries, compiler, etc.) That is, most of the time I sit down to code I actually spend nuking and reinstalling things that (I thought) were previously working.
Your comment makes me feel a little better that this is not merely some personal failing of focus, but happens in a professional setting too.
Oh god yes I remember trying to support old Android games several OS releases later… Impossible, I gave up!
It’s why I still use react, their backcompat is amazing
Happens in AAA too but we tend to have teams that shield everyone from that before they get to work. I ran a team like that for a couple years.
For hobby stuff at home though I don't tend to hit those types of issues because my projects are pretty frozen dependency-wise. Do you really have OS updates break stuff for you often? I'm not sure I recall that happening on a home project in quite a while.
...and people criticise node's node_modules. At least you don't spend hours doing this
Not nearly as hard of a problem. Python does work just fine when it’s pure Python. The trouble comes with all the C/Cuda dependencies in machine learning
But you do because your local node_modules and upstream are out of sync and CI is broken. Happens at least once a month just before a release of course. I'd rather have my code failing locally than trying to debug what's out of sync on upstream.
I do this... but air-gapped :(
that sounds very painful.
Oof. At our company only CI/CD agents (and laptops) are allowed to access the internet, and that's bad enough.
Just use conda.
Then you get one of my favorites: NVIDIA-<something> has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Iirc I originally used conda because I couldn’t get faiss to work in venv, lol. That was a while ago though
Why do ML, especially nowadays on a Mac, when you can do it on an Ubuntu based machine.
Surely work can provide that?
Why Ubuntu specifically? Not even being snarky. Calling out a specific distro, vs. the operating system itself. I've had more pain setting up ML environments with Ubuntu than a MacBook, personally - though pure Debian has been the easiest to get stable from scratch. Ubuntu usually screws me over one way or another after a month or so. I think I've spend a cumulative month of my life tracking down things related to changes inn netplan, cloud-init, etc. Not to mention Ubuntu Pro spam being incessant, as official policy of Canonical [0]. I first used the distro all the way back at Warty Warthog, and it was my daily driver from Feisty until ~Xenial. I think it was the Silicon Valley ad in the MotD that was the last straw for me.
[0] https://bugs.launchpad.net/ubuntu/+source/ubuntu-meta/+bug/1...
I’m glad I have something in common with the smart people around here.
Yeah getting a good python environment setup is a very humbling experience
Docker is your friend
or pyenv at least
Yes, I’ve switched from conda to a combination of dev containers and pyenv/pyenv-virtualenv on both my Linux and MacBook machines and couldn’t be happier
Docker fixes this!
Oh my! This hits home. We have some test scripts written in python. Every time I try to run them after a few months I spend a day fixing the environment, package dependencies and other random stuff. Python is pretty nice once it works, but managing the environment can be a pain.
uv is a great drop-in replacement for pip: https://astral.sh/blog/uv
Just use Linux. Then you only have to fight the nvidia drivers.
Use standard cloud images
Really can't recommend Nix for Python stuff more, it's by far the best at managing dependencies that use FFI/native extensions. It can be a pain sometimes to port your existing Poetry/etc. project to work with nix2lang converters but really pays off.
I can recommend to try poetry. It is a lot more succesful in resolving dependencies than pip.
Although I think the UX of poetry is stupid and I do not agree with some design decisions, I have not had any dependency conflicts since I used it.