MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

Reading emails, replying to emails, scheduling tasks, using apis for services.

Basically everything which doesn't need knowledge but actions.

"Tell my wife i'm late" and it will use some configured magic to talk to service xy and just does it.

Siri is very good in doing homeautomatistaion without the internet, the old google agent and alexa were absolutly not and i don't think they were ever available offline.

This basically gives you a local (local-first!) good working assistent

It could power simple agents like Siri under the hood. Helping with natural language understanding, intent classification, retrieval, and other agent tasks.

user cases are that of LLMs, from a mobile UI (so every AI use case there is), when you need privacy from big tech's AI APIs.

I'm just so amazed by statements like "LLMs can ONLY be used for autocomplete", like am I supposed to be impressed by the smirkiness?

You could possibly fine tune it for narrow domain tasks like they did with tiny-agent https://bair.berkeley.edu/blog/2024/05/29/tiny-agent/

I like the approach that Apple seems to be taking with fine tuned small models that handle routine tasks and then defer to larger off device models for things they can’t confidently do. I imagine you could construct a training set that contains examples that should produce low confidence answers where you could add an output that is essentially a “call for help” option so you could train it to choose that. Smaller models also means you could have more running in parallel and use another to route requests to the appropriate expert.

https://www.meta.com/smart-glasses/

I tested the Google AI on my phone, I had the browser open and asked it to read the page to me and it responded that it does not have access to the internet.

So I would like an AI assistant that:

1 can understand english and my native language

2 that is aware that runs on Android(or KDE/Linux) and can understand commands like "open the Android Settings , Application section " or "read the page that is opened in the browser" or "read the text in the popup that is now opened". Basically to be integrated with the OS via public and open APIs. Big AI companies could compete on selling us better assistants especially for multi lingual people.

3 the model should be small , it should not know geography, history, music bands etc, for tasks where the user asks question there should be an option for the model to forward the question to a search engine or even an online LLM.

Local agent like siri that can do simple tasks, and route more complex requests.

It can be fine tuned for device related actions. In other words, with all the capabilities of your device applications or services, the small model can virtually have the same capabilities. It can always dispatch a user request in way of “natural language” to those applications, and orchestrate the applications. It can dispatch user requests beyond the device capabilities to a cloud model. This is powerful since it changes how you interact with your devices.

Probably hacking foreign intelligence codes.

> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however.

An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.

Llama-3-8b runs fine on raspberry pi

But imagine if these models were baked into your Instagram app and then used for ad targeting using your own compute. Then Facebook gets to look at tons of other data and for less cost (and much less litigation risk) to them.

In this application it’s unfair to compare tiny models to cloud models. Moreover any incremental precision boosts to tiny models would be notable (and directly translate to revenue).

I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely).

I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1].

A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces.

However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant.

[1] https://github.com/mlc-ai/web-llm

It would be fascinating if NPCs had more backstory to them and more complex behaviors. Although I would imagine it would be near impossible to test since anything could influence their behavior.

Would it be interesting dialogue? You could generate more dialogue, but would it have anything underpinning it of interest to the player? i.e. you could suddenly have townspeople that would talk about local scenery or their relationships with other NPCs, but none of that stuff they describe would actually exist in the game. I would personally be weirded out if NPCs started making stuff up.

I can imagine training some sort of LLM on your game data such that NPCs are able to actually describe the game world, but I can't imagine what kind of scale you'd need to operate at for that to be cheaper than just paying someone to write the dialogue. Maybe at Ubisoft's scale where your team sizes are in the thousands (AFAIK, they have been investigating using AI for writing, but it's mostly for things like combat barks which are very repetitive and basically noise.)

wat

https://huggingface.co/mlc-ai

I have an (mlc-llm based) app on the App Store that supports over 2 dozen models, including some recent ones.

cnvrs runs GGUFs on iOS: https://testflight.apple.com/join/ERFxInZg

The Android apk for MLC is updated frequently with recent models built-in. And a Samsung S24+ can comfortably run 7-8B models at reasonable speeds (10ish tokens/sec).

https://llm.mlc.ai/docs/deploy/android.html

Dismissiveness like this tends to radiate ignorance, not insight.

Quests have shipped roughly ~1/2 PS5 sales. Certainly a scale only a handful of technologically advanced product lines outside of phones ever reach.

Incidentally, the enabling technology for the Quest? On-device ML that grew out of - you guessed it - developing on-device inference for their apps.

They have Oculus.

125M parameters feels very feasible to ship as part of apps -- even web-based apps.

That reminds me of the findings of Google’s paper on EfficientT5 (https://arxiv.org/abs/2109.10686). They refer to it as “DeepNarrow”.

Other research from Meta FAIR actually suggests that you should prune deeper layers if you want to improve performance while maintaining accuracy [1]. So there must be a cutoff point for smaller networks where this approach still works, otherwise the results are contradictory. Or we could drastically improve these new models even further.

[1] https://arxiv.org/html/2403.17887v1

The paper says they tried that: https://arxiv.org/abs/2402.14905

Deep link to the relevant snippet in html version: https://ar5iv.labs.arxiv.org/html/2402.14905#S3.SS5

"So far, we trained compact models from scratch using next tokens as hard labels. We explored Knowledge Distillation (KD)... Unfortunately KD increases training time (slowdown of 2.6−3.2×) and exhibits comparable or inferior accuracy to label-based training (details in appendix)."

If you mean that LM-head is just inverted embedding matrix then this was already done in GPT-2.

Unfortunately, the only thing I found out about this is that bigger models benefit from separate layer. But this was only mentioned somewhere in discord, so no paper to read and my personal hunch is that it should work for bigger models too. After all, GPT-3 was just scaled GPT-2.

From my personal experiments, models learn better if you give them harder task. And tied weights could be one of such things. Multi-token prediction could be another and bitnet could be also considered such... (and dropout too)

Training models is not OS dependend. RAM is dependend on the size and i would argue this should be a lot easier to finetune with less GPU Ram.

Nonetheless the endgoal will probably be downloading a model like this or paying for finetuning than downloading and using it through an optimized Neuralchip.

Its currently more a question of when this will happen. The newest Windows cert already requires some neuralchip and even my google pixel 8 pro can host small models (i know the pixel is not a cheap phone, but the coprocessor should still be much more affordable than a big GPU)

Tensor chips are not open enough for an optimized mobile LLM to be ran on them.