OpenAI is good at unminifying code

It’s not only their core strength — it’s what transformers were designed to do and, arguably, it’s all they can do. Any other supposed ability to reason or even retain knowledge (rather than simply regurgitate text without ‘understanding’ its intended meaning) is just a side effect of this superhuman ability.

One potential benefit should be that with the right tooling around it it should be able to translate your code base to a different language and/or framework more or less at the push of a button. So if a team is wondering if it would be worth it to switch a big chunk of the code base from python to elixir they don't have to wonder anymore.

I tried translating a python script to javascript the other day and it was flawless. I would expect it to scale with a bit of hand-railing.

The problem is the use case is where you don't care about the risk of hallucinations or you can validate the output without already having the data in a useful format. Plus you need to lack the knowledge/skill to do it more quickly using awk/python/perl/whatever.

Isn't this already their main use case for business? We use them primarily for extracting structured data from other forms.

Particularly those that are basically linear, that don’t involve major changes in the order of things or a deep consideration of relationships between things.

They can’t sort a list but they can translate languages, for instance, given that a list sorted almost right is wrong but that we will often settle for an almost right translation.

I left my Kleenex next to the Xerox.

There is a certain justice in the use of OpenAI as a name for their product, given that OpenAI has turned the generic technical GPT name into a brand.

more likely to get downvotes that way, potentially even downweighted

I agree, this would make the title more accurate.

It would have been a better title, yes.

I think it should not say the name of the company, but either ChatGPT or GPT-x.

The potential implications of this are huge. Not just open sourcing, but imagine easily decompiling and modifying proprietary apps to fix bugs or add features. This could be a huge unlock, especially for long dead programs.

For legal reasons I bet this will become blocked behavior in major models.

Unminifying isn't decompiling.

It's just renaming variable and functions and inserting line breaks.

Seems like we could render all code open source.

I agree. I think "AI generating/understanding source code" is a huge red herring. If AI was any good at understanding code, it would just build (or fix) the binary.

And I believe how it will turn out to be, when we really have AI programmers, they will not bother with human-readable code, but code everything in machine code (and if they are tasked in maintaining existing system, they will understand in its entirety, across the SW and HW stack). It's kinda like diffusion models that generate images don't actually bother with learning drawing techniques.

Seems like we could render all code open source

Unfortunately not really. Having the source is a first step, but you also need the rights to use it (read, modify, execute, redistribute the modifications), and only the authors of the code can grant these rights.

I think there's actually some potential here, considering LLMs are already very good at translating text between human languages. I don't think LLMs on their own would be very good, but a specially trained AI model perhaps, such as those trained for protein folding. I think what an LLM could do best is generate better decompiled code, giving better names to symbols, and generating code in a style a human is more likely to write.

I usually crap on things like chatgpt for being unreliable and hallucinating a lot. But in this particular case, decompilers already usually generate inaccurate code, and it takes a lot of work to fix the decompiled code to make it correct (I speak from experience). So introducing AI here may not be such a huge stretch. Just don't expect an AI/LLM to generate perfectly correct decompiled code and we're good (wishful thinking).

It can’t really compensate for missing variable and function names, not to mention comments.

Anyone working on decompiler LLMs?

Here is an LLM for x86 to C decompilation: https://github.com/albertan017/LLM4Decompile

There was a paper about this at CGO earlier this year [1]. Correctness is a problem that is hard to solve, though; 50% accuracy might not be enough for serious use cases, especially given that the relation to the original input for manual intervention is hard to preserve.

[1]: https://arxiv.org/abs/2305.12520

Seems like we could render all code open source.

That's not how copyright and licensing works.

You could already break the law and open yourself up to lawsuits and prosecution by stealing intellectual property and violating its owners rights before there were LLMs. They just make it more convenient, not less illegal.

JS minification is fairly mechanical and comparably simple, so the inversion should be relatively easy.

Just because a task is simple doesn't mean its inverse need be. Examples:

  - multiplication / prime factorization
  - deriving / integrating
  - remembering the past / predicting the future

  - bugs
  - unused or irrelevant routines
  - incorrect implementations that incidentally give the right results

JS minification is fairly mechanical and comparably simple, so the inversion should be relatively easy.

This is stated as if it's a truism, but I can't understand how you can actually believe this. Converting `let userSignedInTimestamp = new Date()` to `let x = new Date()` is trivial, but going the other way probably requires reading and understanding the rest of the surrounding code to see in what contexts `x` is being used. Also, the rest of the code is also minified, making this even more challenging. Even if you do all that right, it's at best it's still a lossy conversion, since the name of the variable could capture characteristics that aren't explicitly outlined in the code at all.

Converting a picture from color to black and white is a fairly simple task. Getting back the original in color is not easy. This is if course due to data lost in the process.

Minification works in the same way. A lot of information needed for understanding the code is lost. Getting back that information can be a very demanding task.

Random try (the first one) with Claude 3.5 Sonnet: https://claude.site/artifacts/246c1b1a-3088-447a-a526-b1e716...

I'm not on PC so it's not tested.

Yep, I've tried to use LLMs to disassemble and decompile binaries (giving them the hex bytes as plaintext), they do OK on trivial/artificial cases but quickly fail after that.

More tools should be built on ASTs, great work!

I'm still waiting for the AST level version control tbh

Does it work with huge files? I'm talking about something like 50k lines.

Edit: I'm currently trying it with a mere 1.2k JS file (openai mode) it's only 70% done after 20 minutes. Even if it works therodically with 50k LOC file, I don't think you should try.

how do you make an LLM work on the AST level? do you just feed a normal LLM a text representation of the AST, or do you make an LLM where the basic data structure is an AST node rather than a character string (human-language word)?

Would it be difficult to add a 'rename from scratch' feature? I mean a feature that takes normal code (as opposed to minified code) and (1) scrubs all the user's meaningful names, (2) chooses names based on the algorithm and remaining names (ie: the built-in names).

Sometimes when I refactor, I do this manually with an LLM. It is useful in at least two ways: it can reveal better (more canonical) terminology for names (eg: 'antiparallel_line' instead of 'parallel_line_opposite_direction'), and it can also reveal names that could be generalized (eg: 'find_instance_in_list' instead of 'find_animal_instance_in_animals').

Is it possible to add a mode that doesn't depend on API access (e.g. copy and paste this prompt to get your answer)? Or do you make roundtrips?

What kind of question does it ask the LLM? Giving it a whole function and asking "What should we rename <variable 1>?" repeatedly until everything has been renamed?

Asking it to do it on the whole thing, then parsing the output and checking that the AST still matches?

Looks useful! I will update the article to link to this tool. Thanks for sharing!

Finally someone else using ASTs while working with LLMs and modifying code! This is such an under-utilized area. I am also doing this with good results: https://codeplusequalsai.com/static/blog/prompting_llms_to_m...

Came here to say Humanify is awesome both as a specific tool and in my opinion a really great way to think about how to get the most from inherently high-temperature activities like modern decoder nucleus sampling.

+1

Thanks for your tool. Have you been able to quantify the gap between your local model and chatgpt in terms of ‘unminification performance’?

Unfortunately the comments that could be generated are exactly the ones that should never be written. You want the comment to explain why, the information missing from the code.

what if the comment doesn't match what you intended to write?

// This is an integer containing the accountId

int accountId;

We truly live in the future

This refers to the fact that ChatGPT generated version is missing some characters that are used in the original example. Namely, [looks like HN does not allow me to paste unicode characters, but I am referring to the block characters] can be seen in their version, but cannot be seen in the ChatGPT generated version. However, it very well might be that it is simply because I didn't include all the necessary context.

Discrediting the entire output because a few missing characters would be very pedantic.

Otherwise, the output is identical as far as I can tell by looking at it.

It does seem that the unminified code is very close to the original. In some cases ChatGPT even did its own refactoring in addition to the unminification:

    // ORIGINAL:
    j.useEffect(() => {
        function r() {
            n({ height: window.innerHeight, width: window.innerWidth });
        }
        if (typeof window < "u") return n({ height: window.innerHeight, width: window.innerWidth }), window.addEventListener("resize", r), () => window.removeEventListener("resize", r);
    }, []),

    // UNMINIFIED:
    useEffect(() => {
      const handleResize = () => {
        setSize({ height: window.innerHeight, width: window.innerWidth });
      };

      // Initial size setting
      handleResize();

      window.addEventListener('resize', handleResize);
      return () => {
        window.removeEventListener('resize', handleResize);
      };
    }, []);

You need to use another tool to do the actual renames, like HumanifyJS does:

https://github.com/jehna/humanify

He also told it to reimplement from JavaScript to TypeScript.

I would guess if he just told it to rename the variables and method first, it would have been closer to the original.

I'm bullish on AI, but I'm not convinced this is an example of what you're describing.

The challenge of understanding minified code for a human comes from opaque variable names, awkward loops, minimal whitespacing, etc. These aren't things that a computer has trouble with: it's why we minify in the first place. Attention, as a scheme, should do great with it.

I'd also say there is tons of minified/non-minified code out there. That's the goal of a map file. Given that OpenAI has specifically invested in web browsing and software development, I wouldn't be surprised if part of their training involved minified/unminified data.

I don’t claim expertise in AI or understanding intelligence, but could we also say that a pocket calculator really understands arithmetic and has superior intellectual performance compared to humans?

I think I’d agree with your statement, in the same sense that a chess simulator or AlphaGo are superior to human intellect for their specific problem spaces.

LLMs are very good at a surprisingly broad array of semi-structured-text-to-semi-structured-text transformations, particularly within the manifold of text that is widely available on the internet.

It just so happens that lots of code is widely available on the internet, so LLMs tend to outperform on coding tasks. There’s also lots of marketing copy, general “encyclopedic” knowledge, news, human commentary, and entertainment artifacts (scripts, lyrics, etc). LLMs traverse those spaces handily as well. The capabilities of AI ultimately boil down to their underlying dataset and its quality.

Yeah, ok. Now count the number of Rs in this word.

This is just transforming text.

There are not tons and tons of data pairs on the web of minimized code and unminified code for llms to learn from.

Are you sure about this? These can be easily generated from existing JS to use as a training set, not to mention the enormous amount of non-minified JS which is already used to train it.

Does not involve rote memorization. There are not tons and tons of data pairs on the web of minimized code and unminified code for llms to learn from.

GPT-4 has consumed more code than your entire lineage ever will and understands the inherent patterns between code and minified versions. Recognizing the abstract shape of code sans variable names and mapping in some human readable variable names from a similar pattern you've consumed from the vast internet doesn't seem farfetched.

Not exactly, because you still have to pay any distinct identifier present in your code. Also many minifiers do constant folding and inlining and remove comments, any of which almost surely remove redundant or unused information to compress.

Claude 3.5 Sonnet can natively speak double base64 encoded English. And I do mean it - you can double b64 encode something, send to it, and it'll respond as if it was normal English. Obviously base64 is a simpler transformation than rot13, but no GPT models can deal with double b64.

I asked Claude 3.5 Sonnet a question in Italian in rot13 and it replied in Italian in rot13, there are a few typos but it's perfectly understandable.

What language(s), out of interest?

This does seem to be a smart use of the tools available to skip the grind and get to the point of the leetcode questions.

However, I wonder about this: What will you do in a live interview situation? Will you pull up ChatGPT?

I was with you until:

If an LLM can interact with a browser and figure out how a UI works and take code hints from un-mimified code I think we could see all SAAS apps be commoditized. The backend would be proprietary, but it could figure out API formats and suggest a backend architecture.

whoooha! that's a lot of probing and testing of the SAAS that would be required in order to see how it behaved. SAAS aren't algorithms, they operate over data that's unseen on the front end as well...

All this makes me think AI's are going to be a strong deflationary force in the future.

I don't get this. I've literally never worked anywhere which had enough software engineers, we've been going on about software crisis for about 50 years and things are arguably worse than ever. The gap between the demand for good software (in the sense that allocating capital to producing it would be sensible) and the fulfillment of that demand is bigger than ever. We just don't have the mechanisms to make this work and to make it work at an economically viable level.

Then we get AI to help us and everyone thinks that the economy will shrink?

Honestly, the value in most business software isn't the actual technology. It's the customer base and data held by the platforms.

Someone could already easily clone HubSpot relatively cheaply even if they hired developers, but that doesn't mean it will be anywhere near successful.

I agree, it is fun!

LLM source recovery from binaries is thing. The amazing part is that they are pretty good at adding back meaningful variable names to the generated source code.

This is something you don't need AI for, there are many decompilers out there already as well.

AI cannot even lint properly right now and you want it to decompile? good luck, there's too much hype going on people really think this is possible this year?

In the end always remember it's just autocomplete, it's pretty terrible at translations that are not natural language to natural language. I worked on a natural language to SQL and it was impossible to make it consistently generate valid SQL for Postgres, and I'm talking about natural language to SQL not virtual machine instructions...

I think there are already decompilers and code analyzers at NSA like this. For 10 years or so.

Apologizing to a program seems rather silly though. Do you apologize to your compiler when you have a typo in your code, and have to make it do all that work again?

Can you provide more details? I'm curious about the performance and limitations of these models.

You should give it a try and report back! One easy way would be to take an open-source Android app, compile the APK, then decompile it and feed the bytecode to an LLM and ask it to write the java/kotlin equivalent and compare the source and LLM decoded one.

https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view

Likewise for css class names

If the training data's included both unminified and minified libs, then is it such a stretch?

Is it though? The developer tabs have an unminify button which yields similar results. JavaScript minification is not hard in any way and the guessing of variable names is not that hard given such a simple code example.

It's the title of the blog post.

Agents like Aider or Plandex wrap that up nicely. They do the automatic review and have a very verbose description of the edit format. If you do that often manually, it may be worth testing their prepackaged approach.

See Binary Ninja's Sidekick plugin

Almost likely you would be found guilty because the intent matters. It is easy to check that the generated code is much similar to the original code, and you surely had a reason to bypass the original license. The exact legal reasoning would vary but any reasonable laywer would recommend you to do not.

In the historic Google v. Oracle suit, the only actual code that was claimed to be copied was a trivial `rangeCheck` function, but Google's intent and other circumstances like the identical code structure and documentation made it much more complicated, and the final decision completely bypassed the copyrightability of APIs possibly for this reason.

The garbled text is included in the tree as relevant, pronounceable, and constantly changing text. Here's Chrome's accessibility tree: https://imgur.com/a/V1589Jr

(I'd love if a screen reader user could upload some audio of how awful this sounds, by the by)

Please use `aria-hidden="true"` for stuff like this, it just removes the element from the accessibility tree. I've also emailed Reactive a link to this thread.

Here is a decent intro to ARIA things: https://www.smashingmagazine.com/2022/09/wai-aria-guide/

Or learn to read minified JS code, which is actually not that difficult! (But you do have to take notes to track identifiers.)

A human would be even more expensive.

You should be. I am facepalming at the topic and every single comment in here. It's so full of holes, the Swiss dairy industry went out of business.

I recently learned this too, just a few months ago. Ended up making a frontend so I could do it automatically: https://decompiler.zeroday.engineering/

Beautifiers will restore whitespace, but they won’t rename variables by inferring their semantic meaning.