Ollama LLaVA 1.6 Vision Model Support

Ollama has integrated the LLaVA (Large Language-and-Vision Assistant) 1.6 model collection, providing users with enhanced image resolution, improved reasoning capabilities, and a wider range of model sizes. This update allows for more detailed image analysis and better text recognition within the Ollama ecosystem.

LLaVA 1.6 Technical Enhancements

LLaVA 1.6 introduces three primary improvements over previous versions:

• Increased Image Resolution: The models now support up to 4x more pixels, enabling the assistant to capture and interpret finer details within images.
• Enhanced Reasoning and OCR: The model has been trained on additional datasets specifically focusing on documents, charts, and diagrams to improve text recognition and logical reasoning.
• Permissive Licensing: LLaVA 1.6 is distributed under the Apache 2.0 license or the LLaMA 2 Community License.

Available Model Sizes

Ollama provides LLaVA 1.6 in three different parameter sizes to balance performance and hardware constraints:

• 7B: ollama run llava:7b
• 13B: ollama run llava:13b
• 34B: ollama run llava:34b (New size)

Implementation and Usage

Vision models in Ollama can be interfaced via the Command Line Interface (CLI), Python, or JavaScript.

CLI Integration

Users can pass image files (supporting .jpg and .png formats) directly as file paths in the command line:

ollama run llava "describe this image: ./art.jpg"

Programmatic Access

For developers using the Ollama Python or JavaScript libraries, or the REST API, images can be provided as file paths or as base64-encoded files within the images parameter of the chat request.

Python Example:

import ollama

res = ollama.chat(
	model="llava",
	messages=[
		{
			'role': 'user',
			'content': 'Describe this image:',
			'images': ['./art.jpg']
		}
		]
)

print(res['message']['content'])

JavaScript Example:

import ollama from 'ollama'

const res = await ollama.chat({
	model: 'llava',
	messages: [{
		role: 'user',
		content: 'Describe this image:'
		images: ['./art.jpg']
	}]
})

console.log(res.message.content)

Vision Capabilities and Use Cases

LLaVA 1.6 demonstrates proficiency in two primary vision tasks:

• Object Detection: The model can identify subjects and describe their actions and emotions, such as identifying a person using VR gaming devices and noting their immersion in the experience.
• Text Recognition (OCR): TheL LLaVA 1.6 model can accurately identify bolded or stylized text within images, such as identifying the word "ollama" in a high-contrast image.