26 replies
19h18m
Tangential question : did anyone ever use GPT4-V in production in visual tasks? It's never consistent enough for me to be useful
14 replies
20h18m
This "matching gpt-4" catchy phrase has lost its meaning to me. Everytime an article like this pops up, I see marketing buzz and unrealistic results in practice.
9 replies
20h14m
Of course, it's nothing else. Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to figure out they could also do it with $500.
2 replies
19h38m
It's llama 3 training cost + their cost. Meta "kindly" covered the first $700M.
We add a vision encoder to Llama3 8B
1 replies
17h7m
They didn't train the vision encoder either, it's unchanged SigLIP by Google.
0 replies
6h33m
“We finetuned billions of dollars of research by Google and Meta.”
2 replies
19h41m
You have clearly not read the article. $500 is the cost of fine tuning.
1 replies
16h48m
Fair enough. Is it now safe to say that OpenAI could have done with a 8B model + $500 of fine tuning instead of running a (much) larger model on their GPU cluster?
0 replies
16h37m
Maybe they did
0 replies
19h42m
it would have been a lot cheaper for oai if they had access to llama3 in 2018
0 replies
19h25m
While that may be true, the opposite has also happened to hundreds of companies in other areas:
https://news.ycombinator.com/item?id=39136472
Many companies also optimize for tools, like Python, that have boost productivity more than price/performance ratio. OpenAI had billions of other people's money. They might just keep using tools which worked before.
Lastly, there are tons of papers published on techniques that claim to reduce cost. Most of them aren't good. Their benchmarks aren't good. Even reviewing most of them is more time than a lot of AI researchers have. Those that make it to established communities usually have gotchas that come with the benefits. So, they could also simply miss a needle in a large haystack.
I think you're right that they'd be using whatever really worked with no loss in model performance. It's just that they might not for a number of reasons. The rational choice is for others to keep experimenting with those things in case they get a competitive advantage.
0 replies
19h40m
Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to figure out they could also do it with $500.
People upvoting the post??
Not really sure? But PT Barnum said there's always a lot of them out there.
Pretty sure they mean fine tuning though?
But even that is total tripe.
These guys are snake oil salesmen. (Or Sylvester McMonkey McBean is behind it.)
1 replies
20h13m
For me it's become a signal the person making the claim is unserious.
0 replies
8h3m
If "beats GPT 4" is in the title it's almost a guarantee that it's a bold faced lie that includes benchmark overfitting.
The first time a model that actually matched GPT 4 launched (i.e. Command-R+) there was no mention of it at all. If your results speak for themselves, there's no need to shout.
0 replies
16h45m
Starting to sound like the "iPhone Killer" we've all heard about... for the past 15+ years
0 replies
17h41m
Sadly the front page is often riddled with posts like these.
6 replies
20h38m
- Llava is not the SOTA open VLM, InternVL-1.5 is https://huggingface.co/spaces/opencompass/open_vlm_leaderboa...
You need to compare the evals to strong open VLMs including this and CogVLM
- This is not "first-ever multimodal model built on top of Llama3", there's already a Llava on Llama3-8b https://huggingface.co/lmms-lab
2 replies
19h39m
Very curious how it performs on OCR tasks compared to InternVL. To be competitive at reading text you need tiling support, and InternVL does tiles exceptionally well.
1 replies
11h14m
I think CogVLM2 is even better than Intern at OCR (my usecase is extracting information from an invoice)
0 replies
6h1m
After some superficial testing I with bad quality scans you can find on kaggle I can not confirm that. CogVLM2 refuses to handle scans that InternVL-V1.5 still can comprehend.
0 replies
18h24m
I’m going to be saying First Ever AI something for the next 15 years for clout and capital, not going to be listening to anybody’s complicated ten step funnel if they’re not doing the obvious
0 replies
20h24m
Like InternVL, no llama.cpp support severely limits its applications. Close to GPT4v performance level and runnable locally on any machine (no need for a GPU) would be huge for the accessibility community.
0 replies
15h56m
Thank you for the link! Our initial testing suggests MiniCPM outperforms InternVL for GUI understanding: https://github.com/OpenAdaptAI/OpenAdapt/issues/637#issuecom...
(InternVL appears to hallucinate more.)
6 replies
15h44m
If I had a nickel for every outrageous "matches/beats GPT-x" claim, I'd have more money than the capital these projects raise from VC.
This absolutely is not the first Llama3 vision model. They even quote it's performance compared to Llava. Hard to take anything they say seriously with such obviously false claims
4 replies
12h47m
This absolutely is not the first Llama3 vision model. They even quote it's performance compared to Llava.
Although this is true, there have been earlier Llama3 based vision releases, none of the latest Llava releases are Llama3 based.
3 replies
9h26m
https://ollama.com/library/llava-llama3 llava-llama3
1 replies
8h8m
This appears to be a Llava model which was then fine-tuned using outputs from Llama 3. If I understand correctly, that would make it Llama-2-based.
0 replies
7h57m
fine-tuned using outputs from Llama 3.
Llama 3 outputs text and can only see text, this is a vision model.
that would make it Llama-2-based.
It's based on Llama 3, Llama 2 has nothing to do with it. They took Llama 3 Instruct and CLIP-ViT-Large-patch14-336, train the projection layer first and then later finetuned the Llama 3 checkpoint and train a LoRA for the ViT.
0 replies
5h51m
That is someone else who has just used the Llava name.
It is not by the original group who have published a series of models under the Llava name.
0 replies
11h55m
All models surely write 'its performance'.
6 replies
20h13m
Is there a reason Phi Vision is omitted?
5 replies
19h55m
Is there any place that currently hosts phi3 Vision and provides API access to it? I cannot run it on my local machine, unfortunately.
3 replies
19h9m
Nvidia has a 1000 free credits API for phi3 vision.
You only need an email address.
https://build.nvidia.com/microsoft/phi-3-vision-128k-instruc...
1 replies
17h52m
Was also looking for something like this - I can't find pricing listed anywhere for their API usage, only the free 1,000 credits - or am I completely misunderstanding how this works?
0 replies
6h27m
I can’t find the pricing either. I’m interested, the demo worked well.
0 replies
18h49m
Beautiful. Thank you.
0 replies
18h41m
Azure has a playground, I haven't tried to use it with an API though. https://ai.azure.com/explore/models/Phi-3-vision-128k-instru...
3 replies
20h52m
Shouldn't CogAgent be in this comparison?
2 replies
20h30m
CogVLM should be, not sure how CogAgent plays into this. This isn't an agent.
1 replies
19h57m
You would use CogAgent in VQA mode. Why would someone downvote suggesting to test one of the most powerful multimodal LLMs? Because it doesn't have "V" in its name? CogAgent is improved on many tasks compared to CogVLM.
0 replies
15h31m
I didn't downvote, only replied.
CogAgent is also CogVLM modified to handle documents and larger images. CogVLM is better for VQA.
1 replies
16h7m
How does it compare with MiniCPM-Llama3-V 2.5 [0]? Based on what I see it seems much better than Llama 3-V on the benchmarks. Also it can directly be tried on Huggingface Spaces to check the performance [1]. It has the dataset, code and fine-tuning details with screenshots of it running on Xiaomi 14 pro. It has strong OCR performance and supports 30+ languages.
[0] https://github.com/OpenBMB/MiniCPM-V
[1] https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5
0 replies
6h31m
Woah, this actually did quite well on table data extraction. I wonder how this could be used for long documents. Maybe paired with some kind of hybrid rag approach.
0 replies
20h9m
Don't see a license listed in the repo; presumably needs to be the same as Meta's Llama 3 license?
0 replies
6h27m
Would love to see Ollama support for this - seems promising given my experience with LLaVA so far and would love to get some hands on head to head experience
0 replies
18h50m
Oh wow. I was expecting it to be the 70B one as base given those stats
0 replies
15h26m
Is there a local small llm that can OCR images or haabdwritten invoices ?
Traditional OCR do not handle multiple invoice formats or handwritten ones.
I would like to train one locally with as many invoices it wants
Don’t use it for anything OCR related that needs perfect accuracy. Stuff where some errors are ok, we’ve had great success. Depending on your budget, you can also run it multiple times to catch errors.
How does it compare to Tesseract?
Edit: Thank you!
I’ve done a lot of OCR work and tesseract is nearly a decade out of date at this point. It is not a serious technology for anything requiring good accuracy or minor complexity. From what I’ve seen, GPT-4V completely smokes tesseract, but then again, most modern OCR systems do. If you want fast and pretty powerful OCR, check out paddle. If you want slower but higher accuracy, check out transformer based models such as TrOCR.
Running PaddleOCR in production now, I would suggest contrasting Tesseract v4 and v5, since v5 is a lot better(but until recently has not been available on Linux) - PaddleOCR does still smoke it though, you are right (especially for concurrency and fairly easily just setting different workers to different GPUs for best concurrent batching).
How is Paddle on complex data tables? This is my biggest challenge at the moment.
What format? The entire data table in one image, or a PDF for example printed off with 8 pages where the user choose to only put the header on the first page etc? Or decent formatting, font size 8+ on an image with decent resolution? With the latter you are probably fine although you will need some manual implementation for parsing the output. You get bounding boxes at word level. One thing if I started nowadays I would do is use basic columns (x coordinates) to add '|' inbetween the outputs(including detecting empty span positions), keep items with similarish y coordinates together on lines, and put it into ChatGPT to format as desired, I suspect this would avoid misreading.
I would say PaddleOCR is good in general for tables - it's much better (in terms of recall rate) at recognising numerical digits / symbols than Tesseract although I notice it often misrecognises "l" in "Lullaby/ml/million" etc as "1" sometimes.
The cloud providers have better table extraction iff you can guarantee the same format each time for the document.
A wide variety of PDFs (both in length and content) that can have a variety of different tables, real estate related with a lot of financial content. And I need to be able to run on local models / software (no parsing as a service, no OpenAI, etc).
Here's just one example: https://www.totalflood.com/samples/residential.pdf (I struggle getting accurate data out of the Sales Comp section - basically all approaches mix up the properties.
Tesseract's true value is being one apt-get command away (i.e. opensource). Does Debian host more modern OCR systems in their repos?
Tesseract the tool is one apt-get away but the trained models are not, and I've found that they are a starting point, not a final destination. You still have to do more training on top of them for anything that isn't black text on a crisp white background.
Big mistake on my part; I should clarify I fine-tuned both PaddleOCR and TrOCR on large amounts of data specific to my domain. I cannot speak on the best out of the box “ready to go” solutions (besides cloud ones, which were quite good with the right pre and post processing).
See this for a comparison of PaddleOCR, TrOCR, and various cloud ones (note: on documents of typed and handwritten text):
https://news.ycombinator.com/item?id=32077375
Caveat that being from 2022, the Tesseract version used was almost certainly v4 (if Linux), rather than v5 which is much better (and widely available on Windows in 2022, but not Linux yet).
However Tesseract is quite behind still as you note, even with v5.
Does this require a slight offset and/or rotation to the image, or just literal rerun, with seed seed/whatever giving a different result?
the coolest use case i've seen is this https://github.com/ddupont808/GPT-4V-Act
I feel like this is the beginning of the end for all captchas
Image based or any kind of visual captchas will never be extremely effective. I think we will see more of PoW captchas in the upcoming years (just like cloudflare's turnstile captcha)
I'm not suer about that, can't you give GPT4 a math problem in an image already and have it solve it correctly most of the time?
And these haven't even been trained to defeat captchas/logic problem captchas yet, if it was fine tuned on the general pattern of them I imagine any form of captcha is bust.
Twitter / X has a very interesting captcha: you get to see 10 objects that have weird colors and are slightly deformed, and then you have to match them (1 at a time) with another row that has the same objects but seen from a different angle.
Of course eventually this will be defeated too, but for now it seems to work pretty well.
Nope, I tried it for graph and diagram understanding and it wasn't good enough. Planning to repeat the evaluation with 4o when I have time.
I'm using 4o to convert visual diagrams into mermaid, and it's been almost perfectly accurate in my experience.
This is the out of the box thinking I love about HN. What do you do with the mermaid?
The resulting mermaid is used for... more LLM processing. Converting to mermaid first is more cost-effective, consistent, and accurate for my purposes.
It's very reliable for GUI segment understanding; see e.g. https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (scroll down to `gpt-4-vision-preview`).
Can it be used for automatic annotation?
As in: you tell it that these and these parts should be masked such and such, and then it does that?
We have not had success with that unfortunately.
Thank you, your comment will save me some trouble ;)