Keylogger discovered in image generator extension

Which is everybody in the world except for a handful of people.

Ain't nobody got time for that. LLMs should be capable of analysing code for anything malicious / suspicious.

Everyone runs code they have not inspected. For example, almost no one has read all of the code of in FreeBSD, Linux (kernel), MacOS, Open BSD, or Windows. I also doubt people are reading all of the code in their favorite Linux distribution.

Even inspecting the code is not enough because a lot of security vulnerabilities are not obvious. Basically, security is hard, and often there are not a lot of good solutions.

Here are some tricks I have found which have helped me minimize my risk:

1) Use different machines for different purposes. Basically, you should not use 1 PC (or Mac) for everything. I have one for my finances, one for gaming, and a general-purpose PC. If one gets hacked, the others are still fine.

2) Get software from trustworthy sources. Most of the major software companies are not going to ship malicious code. For open-source software, use software from popular projects which have a good reputation.

3) Ask yourself why is someone providing this software? Is it for money? Are they creating it because they enjoy it? How do they support themselves? For example, Google's business model is building a dossier on people so it can deliver ads they are more likely to click on. When Google gives you something for "free", they will probably use it to track you, or track visitors to your website.

4) Support the people who build the software you use. If its commercial software, pay for it, do not pirate it. If it's open source, donate time or money to the projects you use. Also, thank the people who work on the software, and ALWAYS treat them with respect.

5) Avoid pirated software, software from "free" porn web sites, etc. People who provide illegal software, or sketchy software are probably willing to put back doors in it.

Aside from not using passwords or using 2FA, sandboxing helps.

A VM with GPU passthrough set up would be one example (although this is usually a pain to set up and I expect most people aren't doing it).

As a more user-friendly example, if you install an iOS app (local-model LLM and image generation apps exist), the sandboxing provided by the OS ought to be more than enough to prevent keyloggers, short of 0day exploits.

Ideally, don't use passwords: Passkeys where supported, SSH Keys, client certificates, social login via a service that does support one of these methods.

Magic link emails can also work, but are potentially vulnerable if you copy/pasted it rather than clicking depending on the keylogger's capability and clipboard visibility, although the window for attack is small, it's a much more sophisticated attack that leaves more traces (good sites will reject reuse).

Second best, also use a second factor: U2F ideally, TOTP with the same caveats as magic link emails, and at the bottom of the barrel SMS which is better than nothing but known to be very flawed.

Honestly, if you are anything other than a casual user, and don't have devices with support baked in already, it's crazy not to spend ~£60 on a pair of security keys for passkey/U2F. It's not a lot of money and is just so much more secure.

"keylogger" may not be the right term here? I'm not familiar with how that term is broadly used for, but my definition of that term is a tool that logs your keypresses. Here, it seems like it was scraping your chrome/firefox data for login cookies?

Honestly there's quite a lot of malware that go against those files, I wonder if there's a way to require high privilege to accessing chrome/firefox appdata, or just block it entirely from other apps.

I mean, anything with root access can very easily use libevdev to get all keystrokes as well as mouse positions. (It's maybe 10 lines of code to do that).

So, don't run stuff as root. If it needs root access, run it in a virtual machine (personally I use qubes os for this).

Use 2FA I'd imagine.

I have not seen a statement from Nullbulge so it's not appropriate to say that they took over the repo.

The author of the repo is claiming that their repo is hacked, but this is an obvious lie, because their very first GitHub commit is the one where they push the malware. Nobody would hack an empty GitHub account.

I don't know if the author of the repo is lying when they say that Nullbulge is behind the attack (perhaps the author is part of Nullbulge, perhaps not).

Must be script kiddies. You have the opportunity to deploy anything to a machine that almost certainly has a powerful GPU, and choose a key logger that exists in signature databases? Genius.

Something is fishy here.

According to the original report, the “key logger” was in the custom wheels in the requirements.txt, but looking at that repository there has been only two commits, which according to Reddit both had malicious code in them.

Of course, proper discovery would be easier if the GitHub account still existed.

Well, Docker is great for this as long as you're not one of the unlucky few whose machine is bricked because of Docker. So, mostly yes, I suppose.

doesn't docker have this weird property where it bypasses your firewall?

https://www.techrepublic.com/article/how-to-fix-the-docker-a...

One basic measure (one part of a solution) would be to split Comfy into two parts: the part that does all the work (running plugins, generating images) should have access to nothing but read-only access to the files it needs, the GPU, and a socket to communicate with the other part.

Well, for one, the keylogger is detected by antivirus programs.

I keep coming across various projects whose executables trigger antivirus programs, and I think that when those triggers happen, "it's fine, don't worry" claims need to be treated with more skepticism.

At the same time, antivirus vendors need to stop being so lazy and using strings and such that are clearly part of an open source program/library for their signatures.

I think this is one of the use cases for a sandboxed WASM plugin system.

It's not specific to deep learning, practically every industry will look at security as a cost just not worth it. When we start throwing the CEO into jail instead of making them pay a 18.5M fine for losing the data of 41 million customers that's when things will change. Until then, it's just the cost of doing business.

From my outsider perspective, it's a field that moves very fast, there seem to be new tools being released every week so:

1) As the developer if you focus on hardening, you might be too late to release.

2) People downloading shiny new libs/files/programs constantly.

3) Influx of people not that versed in the basics of computer security playing around with local LLM models, image generators, etc.

"Security is not my field, I'm a stats guy": a qualitative root cause analysis of barriers to adversarial machine learning defenses in industry [0]

[0] https://dl.acm.org/doi/abs/10.5555/3620237.3620448

Isn’t this just one of the milestones that’ll eventually happen? Blind panic due to security always occurs at some point. There must be a ‘law’ defined for this somewhere.

I have a feeling that we'll be seeing some businesses, built, around exactly that.

I'm afraid a few simple tweaks, especially if the hackers themselves have access to the code LLM to try out their code, will be sufficient to evade detection.

If such a tool became commonplace, bad actors would just run it on their own malware and keep tweaking it until the LLM failed to detect it.

It was an extension for ComfyUI, which has 37k stars on GitHub. The way ComfyUI is commonly used is that a person shares a "workflow" file, which utilizes various obscure extensions (called "custom nodes") and then the people who want to run the workflow on their own computer will install all these obscure custom nodes that have like 40 stars on GitHub or so.

Using stars as popularity doesn't work.

I have personally never starred anything that I use. And 90% of the open source that I use isn't on github.

Hopefully this will be just the incentive they need to do something safer. Something similar happened before the move from PKL to SAFETENSOR for model files.

Comfy UI manager recently added some security levels so that by default you can't accidentally leave a public instance that allows remotely installing arbitrary python code https://github.com/ltdrdata/ComfyUI-Manager?tab=readme-ov-fi...

Adobe makes practical pipelines for creatives, not prototyping tools. ComfyUI is mostly for prototyping and ML nerds (I don't mean this in a bad way). There are more practical interfaces to get things done built on top of it, such as Krita Diffusion [1] and many others.

[1] https://github.com/Acly/krita-ai-diffusion

Yep, it's super powerful.

I would say that the "more secure way" is to just use ComfyUI without installing any obscure nodes from unknown developers. You can do pretty much anything using just the default nodes and the big node packs.

And this is the input for AI training.....