LLaVA is a revolutionary large language and vision assistant, making significant strides in the field of multimodal AI. This innovative model brings together the best of language and vision understanding, offering a unique, comprehensive understanding of both visual and textual data. LLaVA represents a novel end-to-end trained large multimodal model that combines a vision encoder and Vicuna for general-purpose visual and language understanding, achieving impressive chat capabilities mimicking spirits of the multimodal GPT-4 and setting a new state-of-the-art accuracy on Science QA.

LLaVA: Blending Vision and Language

LLaVA combines two major components: a vision encoder and Vicuna, another large language model. Through this combination, LLaVA is capable of understanding visual information and generating content both visually and textually. But what sets LLaVA apart is its unique approach to visual instruction tuning – it uses machine-generated instruction-following data to enhance large language models’ capabilities in understanding and generating content in multimodal domains.

Surprisingly, though LLaVA is trained with a small multimodal instruction-following dataset (~80K unique images), it demonstrates quite similar reasoning results with multimodal GPT-4 on these two examples, as stated in the arXiv paper.

The authors show impressive results on multimodal reasoning and instruction following with just this small dataset, further illustrating the effectiveness and efficiency of the LLaVA model.

The Underlying Mechanism: CLIP Image Encoder and LLaMA Decoder

LLaVA is built upon a CLIP image encoder and a LLaMA decoder, which is a recently-developed large language model by Meta, revered for its exceptional text understanding capabilities. LLaMA is fine-tuned for the new task involving images, where image and word tokens are passed to the LLaMA decoder for output.

The development of LLaVA involved an extensive data collection process, where language-image instruction-following samples were collected based on the COCO dataset. The performance of LLaVA and GPT-4 was then evaluated using a two-stage instruction-tuning procedure. Impressively, LLaVA achieved an 85.1% relative score compared to GPT-4, underlining the effectiveness of the self-instruct method in multimodal settings.

The project also provides detailed information about the data files used, along with usage and license notices. LLaVA further leverages language-only models to generate language-image instruction pairs, enabling effective instruction following in the multimodal domain.

Constantly Evolving: Open-source and Regularly Updated

LLaVA isn’t just a static model – it’s constantly evolving. As an open-source project, it allows contributions from a wide array of developers and AI enthusiasts. It has already set new state-of-the-art accuracy on science question answering tasks. Notably, when combined with GPT-4, LLaVA achieves even more impressive results. The arXiv paper reports, ‘Surprisingly, GPT-4 is able to provide consistent improvement over all question classes, and achieves a new SoTA accuracy of 92.53%.’ This speaks to the potential and adaptability of LLaVA, showing how it continues to evolve and adapt for better performance.

LLaVA has also shown impressive results with unseen images and instructions, further attesting to its robust capabilities.

How does LLaVA-1.5 deal with OCR?

Here we tested LLaVA’s optical character recognition capabilities by using a screenshot of the LLaVA paper on arXiv as input. Overall, LLaVA’s OCR performed very well – it was able to correctly extract nearly all of the plain text from the paper. I would estimate its accuracy at around 95-98% on normal body text without any special formatting or characters.

The few errors LLaVA made were primarily in extracting text from in-line citation brackets and numeric superscripts. It seems citations and special symbols like brackets still pose challenges for the model’s OCR system. Additionally, in some cases LLaVA omitted or merged together punctuation and spaces between words. However, it robustly recognized all standard letters and numbers in paragraphs, section headings, and figure captions.

Compared to other state-of-the-art multimodal AI models, LLaVA’s raw OCR abilities appear on par with, if not better than, similar large language models that incorporate computer vision. The high accuracy on plain body text suggests it has learned strong optical recognition abilities.

While there is still room for improvement, especially with specialized text, LLaVA’s overall OCR performance is impressive given a simple screenshot as input. As multimodal models continue to advance, extracting high-quality text from images to enrich language understanding will only become more important. LLaVA sets a strong foundation in this regard and points toward future OCR enhancements.

LLaVA Explaining Graphs & Flowcharts

So here it appears the LLaVA model seemed to have struggled a bit. The chart actually has 6 different time frames and four funds on the left. The next paragraph, again appears to hallucinate. It mentioned that there are 4 sections, even though we can clearly see 6. It stated that the 12 months section was on the top left, even though the table does not have a “12 months” section at all. Overall I would not trust LLaVa to read tables, graphs or complex diagrams.

Comparison with Bing GPT-4

I compared the same image and prompt with Bing GPT-4.

It basically came to the same conclusion. Both Models seemed to have missed the “1 Month”, “3 Month” & “YTD” sections. Bing also came up with the name Bing Global Growth Fund. Both models did say things that are correct, but there are also too many errors which still makes it an un-trusty assistant for visually analyzing data.

Comparing to Bard

And for those of you wondering how it compares to Bard, this is what we got.

It honestly did better than expected. It got all the timestamps correct and most of the numbers. However it mistook the MIWO with the SPX column. Still not too bad overall.

Conclusion

As we journey deeper into the world of AI, we encounter incredible innovations like LLaVA that continue to push the boundaries of what is possible. LLaVA is more than just another AI model. It’s a game-changer, a stride towards the future, bringing language and vision understanding into a seamless, potent blend.

With an uncanny knack for mimicking spirits of AI giants like GPT-4 and making strides with relatively small instruction-following datasets, this revolutionary tool has swiftly set a new gold standard in multimodal AI. When it’s coupled with GPT-4, the duet manage to hit an astonishing SoTA accuracy of 92.53%. Impressive, isn’t it?

But what truly sets LLaVA apart is its adaptability. In a rapidly evolving technological landscape, it isn’t merely a static invention. It grows, learns, and adapts, just like us. As an open-source project, it invites the collective genius of developers and AI enthusiasts to keep refining and improving it.

What’s more, the real-life applications of this tool are boundless. Imagine having a digital assistant that doesn’t just hear you, but sees your world as well. With LLaVA, we’re edging closer to that reality. In the end, LLaVA symbolizes a step into a future where AI doesn’t just understand our words but also sees our world.