Was just talking about this on reddit like two days ago
Instead of data going to models, we need models come to our data which is stored locally and stay locally.
While there are many OSS for Loading personal data, they dont do images or videos. In the future everyone may get their own Model but for now tech is there but product/OSS is missing for everyone to get their own QLORA or RAG or Summarizer.
Not just messages/docs: What we read or write, and our thoughts are part of what makes an individual unique. Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today.
Was just watching this yesterday https://www.youtube.com/watch?v=zHLCKpmBeKA and thought, why we still don't have a computer secretary like her after almost 30 years, who is one step ahead of us.
We are building this over at https://software.inc! We collect data about you (from your computer and the internet) into a local database and then teach models how to use it. The models can either be local or cloud-based, and we can route requests based on the sensitivity of the data or the capabilities needed.
We're also hiring if that sounds interesting!
Wow, nice domain. I'd work there for the name alone haha.
Am I cynical thinking the opposite? I can’t imagine they got that domain for a song. Spending a pile of cash on vanity such as that is a real turn off for me; it signals more flash than bang. Am I wrong to think this?
I’d be worried about the ability to be in relevant searches with a name so generic.
I've never ever ran a query for "software inc" before. They should be okay.
Plus, search engines usually catch up based in click-throughs, bounces, financial kickbacks (cough), too.
Searching for Go programming language stuff was a pain a few years back, but now engines have adapted to Go or Golang.
I don't use Google, so ymmv.
You are not wrong to think this – spending a pile of cash on a name is a big decision that you want to approach with rigor.
We didn't do that, though. Our domain was available for like $4,000. The .inc TLD is intentionally expensive to discourage domain squatting :-)
site's pretty funny, but would likely be more useful with more information and less clicking-around-nostalgia 8-)
That's because the company is more or less in stealth/investigatory mode. It's the same team that built Workflow which was acquired by Apple and then turned into Shortcuts.
Here is the website with the same information and the same clicking around but less nostalgia: https://software.inc/html
I don’t think it is more useful, but it is certainly more functional (supports screen reading, text selection, maybe dark mode, etc)
As fare as I can see it's just a MacOS image, nothing is happening
your site is not loading at all for me on firefox (emulator error) and is totally non-functional on chrome (TCPCreate failed)
might be worth having some sort of automatic fallback to a static site after a certain amount of failed loading or an error
just saw the link to your html version in another comment and it took literally five minutes to load on firefox
I assume that training LLMs locally require high-end hardware. Even running a model requires a decent CPU or, even better, a high end GPU, but it is not so expensive as training a model. And usually you have to use hardware that is available on the cloud, so not much of privacy here.
You don't need to train the model on your data: you can use retrieval augmented generation to add the relevant documents to your prompt at query time.
Thank you for explanation. I see there is still a lot I have to learn about LLMs.
This works if the document plus prompt fit in the context window. I suspect the most popular task for this workflow is summary which presumably means large documents. That's when you begin scaling out to a vector store and implementing those more advanced workflows. It does work even by sending a large document on certain local models, but even with the highest tier MacBook Pro a large document can quickly choke up any LLM and bring inference speed to a crawl. Meaning, a powerful client is still required no matter what. Even if you generate embeddings in "real-time" and dump to a vector store that process would be slow in most consumers hardware.
If you're passing in smaller documents then it works pretty good for real-time feedback.
As someone else said you don't need to train any models, also - small LLMs (7b~) can run really well even on a base M1 Macbook air from 3 years ago.
"While there are many OSS for Loading personal data, they dont do images or videos"
Local models for images are getting pretty good.
LLaVA is an LLM with multi-modal image capabilities that runs pretty well on my laptop: https://simonwillison.net/2023/Nov/29/llamafile/
Models like Salesforce BLIP can be used to generate captions for images too - I built a little CLI tool far that here: https://github.com/simonw/blip-caption
CogVLM blows LLaVA out of the water, although it needs a beefier machine (quantized low-res version barely fits into 12GB VRAM, not sure about the accuracy of that).
I have no actual knowledge in this area so I'm not sure if it's entirely relevant but an update from the 7th of December on the CogVLM repo says it now works with 11GB of VRAM.
Local compute is so 80s, when people moved away from dumb terminals and mainframes, to PCs.
Yes but this time we call it “distributed computing”or “edge computing” instead.
remote computing is so late '90s when people moved away from PCs to servers (the dot in dot com).
Turns out this sort of stuff is cyclical.
Just having an archiver that gives you a tradition search over every webpage you've loaded-- forget the AI stuff, would be a major advance.
I don't know about everyone but a majority of searches are for stuff I've seen before, and they're often frustrated by things that have gone offline or are downranked by search engines (e.g. old documentation on HTTP only sites) or burred by SEO.
you will be shocked when you try Rewind then...
I believe that's exactly what GitHub Copilot does. It first scans and indexes your entire codebase including dependencies (I think). So when it auto-completes, it heavily uses the context of your code, which actually makes Copilot so useful.
You're absolutely right about models coming to our data! If we could have Copilot-like intelligence, completely on-device, scanning all sorts of personal breadcrumbs like messages, browsing history, even webpage content, it would be a game-changer!
Yes, should have local models in addition to remote models. Remote ones are always going to be more capable, and we shouldn't throw that away. Augmentation is orthogonal - you can augment either of these with your own data.
I was imagining something a little more ambitious. Like a model that uses our search history and behavior to derive how to best compose a search query. Bing Chat's search queries look like what my uncle would type right after I explained to him what a search engine is. Throw in some advanced operators like site: or filetype: or at least parentheses along with AND/OR. Surely, we can fine tune it to emulate the search processes of the most impressive researchers, paralegals, and teenagers on the spectrum that immediately factcheck your grandpop's Ellis Island story, with evidence he both arrived at first and was naturalized in Chicago.
Google already tried this 15 years ago
https://en.m.wikipedia.org/wiki/Google_Search_Appliance
That's the most important idea I've read since ChatGPT / last year.
I'll wait for this. Then build my own private AI. And share it / pair it for learning with other private AIs, like a blogroll.
As always, there will be two 'different' AIs: a.) the mainstream, centralized, ad/revenue-driven, capitalist, political, controlling / exploiting etc. b.) personal, trustworthy, polished on peer networks, fun, profitable for one / a small community.
If by chance, commercial models will be better than open source models, due to better access to computing power / data, please let me know. We can go back to SETI and share our idle computing power / existing knowledge