Deep Learning in JavaScript

Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does.

I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch.

JavaScript is so much faster than Python, and the package management experience is so much better, that I really think a JS library would have a chance to take market share from PyTorch in the long term if it had those features.

Many people seem to be unaware of tensorflow.js, an official JS implementation of TF

https://github.com/tensorflow/tfjs

I'd love to see PyTorch in JS, but I think unless you get it running on the GPU it won't be able to do much.

Do you plan to implement WebGPU acceleration to make it production-ready?

The nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disable the mask from attention. That said, very cool project.

Learning Machine Learning, I’ve always been interested in PyTorch and its Automatic Backpropagation.

In this project, I tried to reimplement most of PyTorch in Javascript. It is implemented from scratch in a well-documented, unit tested, and interpretable way, so it can help other JavaScript learners get into Machine Learning!

Hope you enjoy!

I like to see progress with multiple languages in the space. Julia, R, etc. It does seem though, that python really has a big and commanding lead. So much so that mojo is betting their business on it.

Very very cool to see!!!

So yours is SO MUCH MORE fleshed out than mine, but I have a similar (again very fledgling) one going here that uses Typescript:

https://github.com/Marviel/lab-grad

I think there are some really neat opportunities to increase the approachability of ML algorithms when you take advantage of the Typescript templating system to explicitly ensure you're not accidentally composing tensors of incorrect shapes.

Unfortunately, my above lib above doesn't yet handle tensors, it's still at the level of individual neurons. Hopefully someday I can bring it further along, use WASM or WebGPU, etc. :)

Why not using a faster language that compiles to WebAssembly?

BTW: you might want to add support for typed arrays.

See: https://github.com/xenova/transformers.js/blob/8804c36591d11...

This is really old, but added as part of the shape of the vector as well: https://github.com/nicolaspanel/numjs/blob/master/src/dtypes...

Edit: Great work! I'd love to have a nice alternative to PyTorch in JavaScript :)

Edit: formatting

Making JavaScript look like Python in this case could potentially bite you in the ass.

From the example:

    const torch = require("js-pytorch");

    // Instantiate Tensors:
    x = torch.randn([8,4,5]);
    w = torch.randn([8,5,4], requires_grad = true);
    b = torch.tensor([0.2, 0.5, 0.1, 0.0], requires_grad = true);

    class Transformer extends nn.Module {
      constructor(vocab_size, hidden_size, n_timesteps, n_heads, p) {
        //.....
        this.b1 = new nn.Block(hidden_size, hidden_size, n_heads, n_timesteps, dropout_p=p);

    /**
     * All of the arguments for this function are positional
     * and cannot be provided in a different order than defined
     */
    function performOperation(values, arg1 = false, arg2 = -1) {
        //....
    }

    /**
     * This works, but only because of the order
     */
    const result = performOperation([1, 2, 3], arg1 = true, arg2 = 10);

    /**
     * This does not work
     */
    const result = performOperation([1, 2, 3], arg2 = 10, arg1 = true);

    /**
     \* What is actually happening
     \*/
    arg1 = true; // global variable
    arg2 = 10;   // global variable

    const result = performOperation([1, 2, 3], 10, true);

This is great!

I was looking for something like this.

Now I just need a guide that tackles the principles of this, but from a typescript/javascript perspective.

PS: TyTorch (from Typescript) sounds nice!

Wow! Thank you for doing this. It looks like a great starting point for anyone approaching deep learning from the JS ecosystem. It is very plainly written and looks like it will be a joy to learn from. Thank you for adding JSDoc comments with type hints!

Are you open to pull requests? If I have the time I'd love to contribute. I'm sure others would as well.

You should write up a short article on this, even something really simple like one of the examples in the README but with some commentary and examples of output and then post it to a few places like https://dev.to/ or maybe https://hashnode.com/ or even Medium (even though I'm not a big fan). There aren't many newer implementations of PyTorch in JS and I've been looking for one to learn from for some time so I'm sure there are a lot of other JS/TS developers out there that would be interested. Getting to the front page of HN certainly helps but having an article somewhere will help everyone after this week find it through a Google search.

Again, thanks so much for doing this work! It's really helpful to have everything spelled out in JS for those of us who haven't used Python much (I'm sure Python devs can relate when they think about JS projects).

Is the goal here to do everything in Javascript, or it will also be Javascript for the glue and c++/c/fortran/other-compiled-language for heavy lifting? This is more of a node-based effort, or it will work both frontend and backend.

One of the learning tool is notebook. I even has a lisp backend to enabled it to be used. It will be nice if there is a notebook version and easy to run on web (like colab).

Cool project! I worked on something similar a while ago to learn about automatic differentiation: https://github.com/hexhowells/onegrad.js

Needs more layers to be really useful though.

I always thought examples using browser extensions would be neat since their built in JS and you only need to download the model once.

This is great. Code seems to have good comments / jsdoc from the bit studied, tests and all. Will take a closer look in the morning but congratulations on the release.

nice work but why code is in closures using ugly `var` statement?