return to table of content

MemoryCache: Augmenting local AI with browser data

Jayakumark
29 replies
1d

Was just talking about this on reddit like two days ago

Instead of data going to models, we need models come to our data which is stored locally and stay locally.

While there are many OSS for Loading personal data, they dont do images or videos. In the future everyone may get their own Model but for now tech is there but product/OSS is missing for everyone to get their own QLORA or RAG or Summarizer.

Not just messages/docs: What we read or write, and our thoughts are part of what makes an individual unique. Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today.

Was just watching this yesterday https://www.youtube.com/watch?v=zHLCKpmBeKA and thought, why we still don't have a computer secretary like her after almost 30 years, who is one step ahead of us.

conradev
10 replies
21h8m

Instead of data going to models, we need models come to our data which is stored locally and stay locally.

We are building this over at https://software.inc! We collect data about you (from your computer and the internet) into a local database and then teach models how to use it. The models can either be local or cloud-based, and we can route requests based on the sensitivity of the data or the capabilities needed.

We're also hiring if that sounds interesting!

gardenhedge
4 replies
20h50m

Wow, nice domain. I'd work there for the name alone haha.

voakbasda
3 replies
20h43m

Am I cynical thinking the opposite? I can’t imagine they got that domain for a song. Spending a pile of cash on vanity such as that is a real turn off for me; it signals more flash than bang. Am I wrong to think this?

nickthegreek
1 replies
19h19m

I’d be worried about the ability to be in relevant searches with a name so generic.

otteromkram
0 replies
18h31m

I've never ever ran a query for "software inc" before. They should be okay.

Plus, search engines usually catch up based in click-throughs, bounces, financial kickbacks (cough), too.

Searching for Go programming language stuff was a pain a few years back, but now engines have adapted to Go or Golang.

I don't use Google, so ymmv.

conradev
0 replies
16h2m

You are not wrong to think this – spending a pile of cash on a name is a big decision that you want to approach with rigor.

We didn't do that, though. Our domain was available for like $4,000. The .inc TLD is intentionally expensive to discourage domain squatting :-)

herval
2 replies
20h42m

site's pretty funny, but would likely be more useful with more information and less clicking-around-nostalgia 8-)

smith7018
0 replies
20h39m

That's because the company is more or less in stealth/investigatory mode. It's the same team that built Workflow which was acquired by Apple and then turned into Shortcuts.

conradev
0 replies
17h46m

Here is the website with the same information and the same clicking around but less nostalgia: https://software.inc/html

I don’t think it is more useful, but it is certainly more functional (supports screen reading, text selection, maybe dark mode, etc)

thepra
0 replies
20h44m

As fare as I can see it's just a MacOS image, nothing is happening

bjord
0 replies
12h4m

your site is not loading at all for me on firefox (emulator error) and is totally non-functional on chrome (TCPCreate failed)

might be worth having some sort of automatic fallback to a static site after a certain amount of failed loading or an error

just saw the link to your html version in another comment and it took literally five minutes to load on firefox

butz
4 replies
23h46m

I assume that training LLMs locally require high-end hardware. Even running a model requires a decent CPU or, even better, a high end GPU, but it is not so expensive as training a model. And usually you have to use hardware that is available on the cloud, so not much of privacy here.

cjbprime
2 replies
23h44m

You don't need to train the model on your data: you can use retrieval augmented generation to add the relevant documents to your prompt at query time.

butz
0 replies
22h17m

Thank you for explanation. I see there is still a lot I have to learn about LLMs.

Art9681
0 replies
21h55m

This works if the document plus prompt fit in the context window. I suspect the most popular task for this workflow is summary which presumably means large documents. That's when you begin scaling out to a vector store and implementing those more advanced workflows. It does work even by sending a large document on certain local models, but even with the highest tier MacBook Pro a large document can quickly choke up any LLM and bring inference speed to a crawl. Meaning, a powerful client is still required no matter what. Even if you generate embeddings in "real-time" and dump to a vector store that process would be slow in most consumers hardware.

If you're passing in smaller documents then it works pretty good for real-time feedback.

smcleod
0 replies
21h7m

As someone else said you don't need to train any models, also - small LLMs (7b~) can run really well even on a base M1 Macbook air from 3 years ago.

simonw
2 replies
23h30m

"While there are many OSS for Loading personal data, they dont do images or videos"

Local models for images are getting pretty good.

LLaVA is an LLM with multi-modal image capabilities that runs pretty well on my laptop: https://simonwillison.net/2023/Nov/29/llamafile/

Models like Salesforce BLIP can be used to generate captions for images too - I built a little CLI tool far that here: https://github.com/simonw/blip-caption

orbital-decay
1 replies
23h22m

CogVLM blows LLaVA out of the water, although it needs a beefier machine (quantized low-res version barely fits into 12GB VRAM, not sure about the accuracy of that).

cinntaile
0 replies
21h14m

I have no actual knowledge in this area so I'm not sure if it's entirely relevant but an update from the 7th of December on the CogVLM repo says it now works with 11GB of VRAM.

amelius
2 replies
21h13m

Local compute is so 80s, when people moved away from dumb terminals and mainframes, to PCs.

simondotau
0 replies
21h9m

Yes but this time we call it “distributed computing”or “edge computing” instead.

gpderetta
0 replies
20h51m

remote computing is so late '90s when people moved away from PCs to servers (the dot in dot com).

Turns out this sort of stuff is cyclical.

nullc
1 replies
19h38m

Just having an archiver that gives you a tradition search over every webpage you've loaded-- forget the AI stuff, would be a major advance.

I don't know about everyone but a majority of searches are for stuff I've seen before, and they're often frustrated by things that have gone offline or are downranked by search engines (e.g. old documentation on HTTP only sites) or burred by SEO.

mclightning
0 replies
19h19m

you will be shocked when you try Rewind then...

timenova
0 replies
19h34m

I believe that's exactly what GitHub Copilot does. It first scans and indexes your entire codebase including dependencies (I think). So when it auto-completes, it heavily uses the context of your code, which actually makes Copilot so useful.

You're absolutely right about models coming to our data! If we could have Copilot-like intelligence, completely on-device, scanning all sorts of personal breadcrumbs like messages, browsing history, even webpage content, it would be a game-changer!

pradn
0 replies
19h44m

Yes, should have local models in addition to remote models. Remote ones are always going to be more capable, and we shouldn't throw that away. Augmentation is orthogonal - you can augment either of these with your own data.

jakderrida
0 replies
21h38m

Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today.

I was imagining something a little more ambitious. Like a model that uses our search history and behavior to derive how to best compose a search query. Bing Chat's search queries look like what my uncle would type right after I explained to him what a search engine is. Throw in some advanced operators like site: or filetype: or at least parentheses along with AND/OR. Surely, we can fine tune it to emulate the search processes of the most impressive researchers, paralegals, and teenagers on the spectrum that immediately factcheck your grandpop's Ellis Island story, with evidence he both arrived at first and was naturalized in Chicago.

diceduckmonk
0 replies
19h9m

Google already tried this 15 years ago

https://en.m.wikipedia.org/wiki/Google_Search_Appliance

csbartus
0 replies
21h47m

Instead of data going to models, we need models come to our data which is stored locally and stay locally.

That's the most important idea I've read since ChatGPT / last year.

I'll wait for this. Then build my own private AI. And share it / pair it for learning with other private AIs, like a blogroll.

As always, there will be two 'different' AIs: a.) the mainstream, centralized, ad/revenue-driven, capitalist, political, controlling / exploiting etc. b.) personal, trustworthy, polished on peer networks, fun, profitable for one / a small community.

If by chance, commercial models will be better than open source models, due to better access to computing power / data, please let me know. We can go back to SETI and share our idle computing power / existing knowledge

linsomniac
7 replies
1d

I would sure love a way to "chat" with my browsing history and page content. Is there any way to automatically save off pages that I've visited for later processing? I looked a decade or more ago and didn't really find a good solution.

kaynelynn
3 replies
1d

Rewind.ai is pretty much this - I just installed it and am very happy so far.

Alifatisk
2 replies
23h16m

Isn't it Apple devices only?

thekevan
1 replies
23h0m

"Coming soon to Windows"

https://www.rewind.ai/windows

emptysongglass
0 replies
22h22m

Just need a Linux version or an open source alternative now

wizardwes
0 replies
23h6m

Zotero might work, but only as a highly imperfect solution, since it is more focused on research

solarkraft
0 replies
23h49m

I think WorldBrain (https://github.com/WorldBrain/Memex) promises this. While I'm also excited by the idea, I think there was some reason I ended up not using it.

rhn_mk1
0 replies
20h26m
jml7c5
6 replies
23h50m

I hope this encourages Mozilla to focus more on page archiving support on the web. I feel as though they missed a huge opportunity by not making it easy to archive pages with DOM snapshots, or easy to snag videos or images. (Go to Instagram and try to right-click -> download the image; you can't.) Would have been a very good way to differentiate from Chrome, as Google wouldn't want that available for Youtube. And "our browser can download videos and images from anywhere" is pretty easy to sell for potential users.

eigenvalue
2 replies
23h42m

Agree, it seems like it’s insanely hard to back up a modern JS-enabled web page in a usable way that results in a single file which can be easily shared.

nekitamo
1 replies
23h39m

Have you tried SingleFile? It sounds like what you’re looking for:

https://github.com/gildas-lormeau/SingleFile

eigenvalue
0 replies
22h55m

Will check it out, thanks.

dralley
0 replies
16h52m

Isn't that basically Pocket, the service that people complain about endlessly as "bloat"?

Dwedit
0 replies
22h41m

"Save Page WE" will capture a DOM snapshot to a single HTML file. The only problem is that Data URLs encoded using Base64 are highly bloated.

BlueTemplar
0 replies
23h5m

I'm baffled that the support of single file, offline HTML is still so bad today :

https://www.russellbeattie.com/notes/posts/the-decades-long-...

(I'm suspecting because this goes against the wants of some of the biggest players who have the incentive of making us leave as many online footprints as possible ?)

Even here, Mozilla recommends converting to PDF for easier (?!?) human readability. Except PDF is a very bad format for digital documents, with no support for reflow and very bad support of multimedia. (PDF is perhaps good for archival of offline documents, even despite its other issues).

lofaszvanitt
5 replies
21h52m

What is happening at Firefox is quite strange. Like they are walking backwards.

smcleod
4 replies
20h59m

This seems like a sensible step in the right direction IMO, (optional) features such as local, privacy respecting LLMs will help to augment peoples online research, bookmarking, contextual search etc....

It's important that we have Firefox working on such experiments otherwise as Google adds more of their privacy invading features to chrome / chromium it will likely impact negatively on peoples desire to find alternative browsers.

lofaszvanitt
3 replies
20h51m

Yeah, but maybe, if you are constantly losing market share... maybe you should work on things that appeals to a wider audience. Except if you have a trump card and intend to use it as a deus ex machina to suddenly show people you are THE browser, the way forward.

nullc
2 replies
19h30m

You don't gain market share by doing the same stuff the other FREE alternative does.

You gain market share by doing what they refused to do, no matter how much it's in the user's interest, because their business is stealing the user's data and yours isn't.

mcbits
0 replies
18h5m

In a just world, that's a way to gain market share. In our world, people concede their data for marginal improvements in the quality of a feature because they can't conceive of how giving up control of their data could come back to harm them. It doesn't feel like there is a downside.

lofaszvanitt
0 replies
18h0m

What free alternatives? All the browsers look & feel the same. Zero innovation.

danielovichdk
5 replies
22h36m

My usage with browsing is not relevant for this. I don't want to "chat" with my browsin g history. I would simply love my browser would index my bookmarks on my OS so I could search the actual content of those bookmarks.

The feedback loop coming gained from chatgtp will I assume always be way better than my local gpt equivalent.

But often I bookmark pages where I know the information on there are important enough for me to come back to more than once.

So I have started crafting out a solution for this. It crawls your bookmarks on your local browser storage, downloads those pages and adds them to your search index on your OS.

That's been an itch for me for years.

jval43
2 replies
21h46m

Didn't Chrome do this at the very beginning, when it was initially released? I faintly remember that being a feature.

Personally I would already be content if my browsers didn't forget their history all the time, both Firefox and Safari history is way too short-lived.

ayewo
1 replies
12h58m

You were probably thinking of Google Desktop which could search almost anything on your machine.

https://en.wikipedia.org/wiki/Google_Desktop

jval43
0 replies
11h0m

I went looking and it was indeed Chrome that could do it. See screenshot from 2009 here: https://superuser.com/a/42499

Google removed the feature intentionally in 2013: https://bugs.chromium.org/p/chromium/issues/detail?id=297648

Apparently Opera supported it too at the time, and from the comments Safari as well.

Performance reasons seem to have killed it. I'd think that after 10 years now any modern computer would be able to handle it.

overstay8930
0 replies
21h12m

Isn't this just Safari? *using a modern chip

groestl
0 replies
22h15m

Small data sets suffer from bad recall in full text search. So a bit of smart fuzzyness added to the search by AI could improve the experience on locally indexed bookmarks quite well.

bloopernova
5 replies
1d

Regarding PrivateGPT, if I have a 12GB Nvidia 4070 and an 11GB 2080ti, which LLM should I run?

Edited to add: https://www.choosellm.com/ by the PrivateGPT folks seems to have what I needed.

SkyMarshal
4 replies
1d

There's a big community discussing exactly that over at https://www.reddit.com/r/LocalLLaMA/.

smcleod
3 replies
21h2m

+1 r/localllama, 23GB should allow you to run 30b~ models, but honestly some of the new smaller models such as Mistral & friends (Zephyr etc..) are really interesting. You could also Give Mixtral a try if you get a low quant format such as this q3 - https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-G...

dr_kiszonka
2 replies
17h38m

Do you know if there is a website or spreadsheet where I could enter my RAM, GPU, etc. and see what models I could run locally?

unstuck3958
0 replies
15h13m

While I agree a website/spreadsheet would be convenient, it's not that complicated. As long as GPU is handling 50-75% of the LLM layers, you should get a decent tok/sec speed (unless you're running really really large models).

SkyMarshal
0 replies
17h5m

Yes, https://www.reddit.com/r/LocalLLaMA/. Just ask there and a bunch of Non-Artificial Intelligent agents will give you that answer ;).

reqo
4 replies
1d

What I would love to see is this model being able to learn ti automate some tasks that I usually do! e.g. sign up for events/buy tickets etc! If this has access to your login details, and could log in, it could be a great assistant!

candiddevmike
2 replies
23h21m

Teach it to press the skip ad button

lacker
1 replies
23h6m

Or it could click "hide" on cookie banners for me!

k1t
0 replies
21h59m

They actually already added this, but it's still in a limited trial phase.

https://support.mozilla.org/en-US/kb/cookie-banner-reduction

redblacktree
0 replies
23h50m

Or shoe bot

CollinEMac
4 replies
1d

I'm confused by the example they gave.

What is the meaning of a life well-lived?

Is the response to this based on browser data? Based on the description I was expecting queries more like:

What was the name of that pdf I downloaded yesterday?

What are my top 3 most visited sites?

What type of content do I generally interact with?
ipaddr
2 replies
23h33m

That information is already available. You want a better search interface.

lacker
0 replies
23h6m

Yes, exactly, I want a search interface that's an LLM instead of a bunch of menus.

ape4
0 replies
23h12m

They could private a local url called "about:wrapped" that gives a summary of your usage like Spotify Wrapped. The top 100 sites, you can click on a site for more info like what pages did you visit, when, how often, etc.

atomicUpdate
0 replies
19h54m

One thing you'll see from a lot of these LLM examples and demos are intentionally subjective queries, so they can't be judged on pass/fail criteria.

For example, you'll see things like "where should I visit in Japan?" or "how should I plan a bachelor party?", because they are a huge variety of answers that are all "correct", regardless of how much you disagree with them. There is also a huge number of examples from them to draw from, especially compared to something as specific as your browsing history.

orbital-decay
3 replies
22h59m

Classic bookmarks have failed because mnemonic organization doesn't scale. This kind of interface does, and can replace it entirely if done right.

Thinking of it, something like this can be used for all your local files as well, acting as a better version of the old filesystem-as-a-database idea. Or for a specific knowledge base (think LLM-powered Zotero).

wintogreen74
0 replies
22h31m

Sounds like you just invented the modern version of Windows Longhorn

ElectricalUnion
0 replies
15h2m

Wasn't this the idea behind "Networked Environment for Personal, Ontology-based Management of Unified Knowledge" (Nepomuk) Semantic Desktop?

Assuming that you can coerce the LLM to fill in the RDF correctly, and that we now have much more memory and faster storage, it might work.

AureliusMA
0 replies
22h35m

Something like Orbit would be perfect

https://withorbit.com/

ath3nd
2 replies
21h41m

They might be onto something here.

Instead of doing lots of back-n-forth with the giants, enriching them with each prompt, you get a smaller local model that's much more respectful of your privacy.

That's an operating model I am willing to do some OSS contributions to, or even bankroll.

Gotta love the underdogs, even if admittedly, I am not a big Mozilla org fan.

visarga
0 replies
21h16m

In the future their AIs are going to talk to our AIs. Because we need protection.

altairprime
0 replies
21h27m

It’s what Apple’s been doing for a few years, though it remains unclear how much of that is “AI”. So it makes sense that someone else would enter that niche.

no_time
1 replies
21h53m

Good idea. Mozilla gets a lot of rightful hate for their mishandling of FF and their political preaching, but I believe they are still capable of developing tech that is both privacy preserving and user friendly at the same time.

I use the offline translator built into FF regularly and It's magic. I would've never thought something like that can run locally, without a server park worth of hardware thrown at it.

Here's hoping this experiment turns out the same way.

pixxel
0 replies
21h31m

Well said; I agree wholeheartedly.

avallach
1 replies
1d

PrivateGPT repository in case anyone's interested: https://github.com/imartinez/privateGPT . It doesn't seem to be linked from their official website.

nightski
0 replies
1d

It's linked from the MemoryCache repo listed at the bottom of the article: https://github.com/Mozilla-Ocho/Memory-Cache

Dwedit
1 replies
1d

Very misleading name. The word "Memory" has a distinct meaning in relation to computing, but this is more about human memories.

Sai_
0 replies
22h8m

I was going to ignore this as a troll comment because computer memory has its antecedents in human memory but the commenter is right - the combination of memory and cache to talk about human memory seems misleading.

yeukhon
0 replies
23h4m

Maybe it is just me, since I lived through the Firefox OS era as a past intern: this feels like a possible re-entrance of offering a Mozilla-built OS in the future. They said Internet was born to connect people - but building everything into a browser is not the most optimal way of adding all these fancy stuff. Firefox OS was basically a small linux kernel plus Gecko plus HTML5 for rendering. So much like iOS and iPadOS Mozilla could offer similar OS for devices/platform. I mean, for the past 5 years they have been invested in AR and VR. So I won’t be surprised if they eventually bet on another Firefox OS…

tesdinger
0 replies
20h40m

I wish they would fix basic features such as downloading pictures on Firefox for Android. Often long pressing the image on your screen opens a context menu that does not allow download, only following the link associated with the image.

stainablesteel
0 replies
19h24m

this is a pretty cool idea, i'd like to be able to choose which pages i want to cache

politelemon
0 replies
22h23m
nektro
0 replies
20h44m

you're better than this mozilla. hopping on the ai trend is disgusting given your alleged morals

huy77
0 replies
20h40m

so this is how growth hacking look like, building a landing page for a imaginary product to test market-fit idea?

emsign
0 replies
11h34m

Does it ingest the ads containing in the web pages as well? This would be a major concern, ads will pollute the model at least with unwanted information. At worst they'd be a security concern when used for indiscriminate or even targeted manipulation of the model. Advertisers do that to our brains, it's all they do. So why shouldn't they try that with LLMs scraping/being fed from the web?

SpaceManNabs
0 replies
21h22m

This seems completely overkill.

I don't even like having to clear my history and wtv regularly. I use incognito mode most times.

Now I have monitor what my local AI collects?

"through the lens of privacy" my ass, man.

Why would I ask my browser what the meaning of a life well lived is?

Ringz
0 replies
19h0m

Doing this for a long time.

https://news.ycombinator.com/item?id=38421121

The FF solution is just more automated.

ChrisArchitect
0 replies
20h12m

Could barely get a sense of what any of this meant from the shared link.

Went back a bit further/to the official site:

MemoryCache is an experimental development project to turn a local desktop environment into an on-device AI agent.

Okayy...

And this from November

Introducing Memory Cache

https://memorycache.ai/developer-blog/2023/11/06/introducing...