TL;DR:
- Organizations should train their AI systems using their own data to enhance generative AI effectiveness.
- Using foundation models as a starting point provides relevant context and mitigates risks.
- Accuracy is crucial, especially in industries like agritech, where AI powers critical applications.
- Jiva, an agritech vendor, trains its AI models using thousands of annotated images collected from its field teams and farmers.
- Localization is essential, and Jiva has achieved better results by training models specific to crops common in Indonesia and India.
- Finetuning AI models with internal data lead to better outcomes and alignment with business objectives.
- Data quantity doesn’t necessarily equate to data quality; context and annotation are crucial.
- Generative AI has great potential in call centers, improving customer service through better responses.
- Adding domain-specific content to existing language models through finetuning requires less data and compute time.
- Retaining control over data used for AI training ensures transparency and responsible AI adoption.
- Challenges include the lack of standardized prompt methodologies and the need for improved cross-language support.
- Human expertise remains invaluable in addressing unique challenges and augmenting AI capabilities.
Main AI News:
Artificial intelligence (AI) has become a pivotal tool for organizations seeking to unlock its generative potential. To maximize its effectiveness, it is crucial for businesses to train their AI systems using internal data as a foundation, while still building upon existing models. By doing so, organizations can contextualize their AI models, addressing concerns related to accuracy, intellectual property, and potential risks.
Jiva, a leading agritech vendor, understands the significance of accuracy when it comes to AI implementation. Their flagship mobile app, Crop Doctor, utilizes AI-powered image processing and computer vision to identify crop diseases, enabling precise treatment recommendations. Moreover, Jiva leverages AI to assess the creditworthiness of farmers seeking cash advancements, ensuring reliable loans. The company employs a range of AI and machine learning tools, including Pinecorn, OpenAI, scikit-learn, TensorFlow, and Vertex AI. With operations spanning Singapore, Indonesia, and India, Jiva has established itself as a prominent player in the agritech space.
Tejas Dinkar, CTO of Jiva, emphasizes the importance of training AI models with large volumes of annotated images specific to each disease. Jiva’s extensive network of field teams and farmers contributes hundreds of thousands of images through their app, AgriCentral, available in India. The initial collection and annotation of these images involve both field experts and agronomy specialists, resulting in a robust training model for accurate plant disease identification. For unfamiliar crops, Jiva collaborates with platforms like Plantix, leveraging their extensive datasets for enhanced image recognition and diagnosis capabilities.
In an interview with ZDNET, Dinkar emphasized the vital role of delivering accurate information, as it directly impacts farmers’ harvests and livelihoods. To ensure data veracity, Jiva relies solely on meticulously sourced and vetted datasets, disregarding any pretrained farming data that may exist within the AI models. This approach mitigates the risk of providing vague or unreliable responses to farmers seeking assistance.
Localization is another crucial aspect of AI model development highlighted by Dinkar. While existing models like Plantix provide a solid foundation, they may not be adequately trained on region-specific data. Jiva’s expertise in crops common to Indonesia and India, such as corn, has yielded superior performance compared to off-the-shelf products. This localization demonstrates the significance of tailoring AI models to specific regions and markets.
Refining AI Models through Data Finetuning
While using foundation data models accelerates the adoption of generative AI, it poses challenges when the data’s relevance to the specific industry is in question. Olivier Klein, Amazon Web Services’ (AWS) Asia-Pacific chief technologist, emphasizes the importance of finetuning AI models with organization-specific data. Properly incorporating internal data into AI models expedites implementation and ensures alignment with business objectives. Standalone generative AI, when combined with an organization’s data strategy and platform, delivers more compelling results.
Klein acknowledges that the availability of sufficient internal data is a common challenge for companies. However, he stresses that data quality is more important than sheer quantity. Accurate data annotation and contextualizing AI training models to the industry are critical. By labeling individual components of the training data and adding context, organizations empower AI systems to generate responses tailored to their industry’s specific needs.
It is crucial to dispel the misconception that all AI systems are identical, according to Klein. Organizations must tailor AI models to their use cases and verticals. In the realm of call centers, LLMs have sparked numerous discussions about enhancing the experience for call agents. By granting access to better responses in real-time, generative AI enables call agents to deliver improved customer service. Call center operators can train AI models using their knowledge base, which includes chatbot interactions and customer engagements.
Building upon Existing Models and Ensuring Data Control
Adding domain-specific content to existing LLMs, already trained on general knowledge and language-based interactions, is a viable approach that requires less data than building a foundational model from scratch. A report by Business Harvard Review highlights the effectiveness of this finetuning approach, which involves adjusting parameters and utilizing hundreds or thousands of documents. This method reduces both compute time and data requirements compared to starting from square one. However, it is worth noting that not all LLM providers permit finetuning on top of their models, limiting the options available.
Harnessing internal data addresses a significant concern in generative AI—the need for organizations to maintain control over the training data. Klein emphasizes that retaining data control ensures transparency, responsible AI adoption and eliminates the “blackbox” effect. Organizations have a clear understanding of the data used to train their AI models, fostering trust and compliance with ethical guidelines. AWS actively collaborates with regulators and policymakers to identify policies that prevent the blackbox effect and extends this support to customers. Amazon Bedrock, a suite of foundation models including Amazon Titan, AI21 Labs’ Jurassic-2, Anthropic’s Claude, and Stability AI, can detect bias and filter content that violates AI ethical guidelines.
Klein envisions the emergence of more foundation data models in the future, including vertical-specific base models, offering organizations a wider range of training options.
Key Challenges and Human Expertise
While generative AI holds immense promise, there are situations where human expertise is invaluable. In cases of rare or highly specific crop issues, Jiva’s team of agronomy experts collaborates with local researchers and field teams to resolve them. By overlaying data generated by AI systems with additional information, such as on-site visits, Jiva ensures comprehensive credit assessments. Human involvement complements AI capabilities by amplifying adaptive thinking, an area where machines are yet to excel.
Dinkar points out challenges faced by Jiva in their generative AI adoption journey, including the lack of a standardized prompt methodology across different software versions and providers. Achieving “true omni-lingualism” and addressing hallucination remain ongoing concerns in LLMs. Refining prompt engineering has allowed Jiva’s agronomy bot to seek clarification when it struggles to infer the referenced crop based on context. However, it is important to note that this specific prompt’s performance varied across different LLM versions, highlighting the need for bespoke prompt techniques for each platform. As tooling and best practices continue to evolve, cross-platform prompts may become a reality.
Improvements in cross-language support are also essential, as chatbots sometimes generate out-of-context responses, leading to strange outcomes.
By leveraging internal data, organizations can enhance the effectiveness of their AI models while mitigating risks associated with accuracy, data ownership, and intellectual property. Finetuning AI models with context-specific data allow for more relevant and targeted responses. Human expertise remains vital in overcoming unique challenges and ensuring the responsible and effective use of generative AI. With ongoing advancements and collaborations across the AI landscape, businesses can look forward to continued progress and further opportunities to train their AI models effectively.
Conclusion:
Leveraging internal data for AI training offers businesses the opportunity to enhance the effectiveness of generative AI while mitigating risks. By using their own data, organizations can contextualize AI models, improve accuracy, and address industry-specific needs. The localization of AI models and the process of finetuning with internal data further optimize outcomes. However, challenges in prompt methodologies and cross-language support persist. Combining human expertise with generative AI will continue to drive innovation and deliver superior results in the market.
