TL;DR:
- Large language models like Flamingo, GPT-4V, and Gemini have excelled in various tasks.
- Open-source models like LLaMA and Vicuna have accelerated vision language model development.
- LLaVA-Phi, powered by compact language model Phi-2.2, combines with LLaVA-1.5 for exceptional performance.
- LLaVA-Phi’s three billion parameters outperform larger multimodal models.
- Impressive results on diverse academic benchmarks, including ScienceQA and MMBench.
- Outperforms competitors, including MobileVLM, across all five key performance metrics.
- Limited multilingual capabilities due to Phi-2’s codegenmono tokenizer.
- Future improvements focus on enhancing small language model training and optimizing visual encoder size.
Main AI News:
Large language models have made significant strides in executing instructions, engaging in multi-turn conversations, and solving image-based question-answering challenges. Prominent examples include Flamingo, GPT-4V, and Gemini. The rapid evolution of open-source Large Language Models like LLaMA and Vicuna has significantly propelled the development of open-source vision language models. These advancements are primarily focused on enhancing visual comprehension by harnessing language models with a minimum of 7 billion parameters and seamlessly integrating them with a vision encoder. Notably, industries requiring real-time interactivity, such as autonomous driving and robotics, stand to benefit from increased inference speed and reduced test times.
In the realm of mobile technology, Gemini has emerged as a trailblazer in adopting multimodal approaches. Gemini-Nano, a streamlined iteration, boasts 1.8/3.25 billion parameters and is compatible with mobile devices. However, critical details concerning the model’s architecture, training datasets, and training methodologies are held under tight confidentiality and remain undisclosed.
A groundbreaking study conducted jointly by Midea Group and East China Normal University introduces LLaVA-Phi, a vision-language assistant fueled by a compact language model, Phi-2.2. This innovative development combines the prowess of Phi-2.2, the most efficient open-sourced tiny language model, with the robust capabilities of LLaVA-1.5, a versatile open-source multimodal model. The researchers employ LLaVA’s meticulously curated high-quality visual instruction tuning data in a meticulously crafted two-stage training pipeline. LLaVA-Phi’s performance is nothing short of exceptional, rivaling or surpassing that of larger multimodal models while comprising a mere three billion parameters.
The research team subjected LLaVA-Phi to a battery of rigorous evaluations, encompassing a diverse range of academic benchmarks tailored for multimodal models. These evaluations included VQA-v2, VizWizQA, ScienceQA, and TextQA, covering general question-answering capabilities, as well as specialized assessments such as POPE for object hallucination and MME, MMBench, and MMVet, which comprehensively gauge multimodal abilities like visual understanding and visual commonsense reasoning. Astonishingly, LLaVA-Phi consistently outperformed its larger counterparts, including models like IDEFICS relying on 7-billion parameter or larger Large Language Models.
One standout achievement was LLaVA-Phi’s exceptional performance on ScienceQA, particularly in answering math-based questions. This can be attributed to the Phi-2 language model’s training on mathematical corpora and code production domains. In the extensive multimodal benchmark of MMBench, LLaVA-Phi surpassed numerous prior art vision-language models based on 7-billion-parameter Large Language Models.
Additionally, the study juxtaposed LLaVA-Phi with MobileVLM, another parallel initiative aimed at constructing an effective vision-language model. In a resounding victory, LLaVA-Phi consistently outperformed all competitors across all five performance metrics.
The research team acknowledges that LLaVA-Phi, while excelling in its current capabilities, is not fine-tuned for multilingual instructions. It is unable to process instructions in various languages, including Chinese, due to Phi-2’s utilization of the codegenmono tokenizer. Future endeavors are poised to enhance the training procedures for small language models and explore the impact of visual encoder size through methods like RLHF and direct preference optimization. These initiatives are geared toward further enhancing performance while reducing model size, promising a bright future for vision-language assistants.
Conclusion:
LLaVA-Phi, driven by Phi-2 technology, signifies a significant advancement in the vision-language assistant market. Its remarkable performance and potential for improvement indicate a promising future, especially for industries demanding real-time interactivity and enhanced visual understanding. This innovation is poised to reshape the landscape of vision-language assistants and drive market growth in the coming years.
