Freshmen URD Track; Entrepreneurship and Innovation
Problematically, young people's spending habits are on the rise, which is having an effect on their financial stability. This disparity between the ideal situation of fiscal stability and the current level of overspending is a major issue that requires immediate resolution.
Project Solution Approach
The utilization of cutting-edge technology, specifically the advanced LayoutLM Question Answering model, offers a potential opportunity to empower young individuals in efficiently controlling their financial behavior and fostering a sound financial lifestyle. This technique seeks to streamline financial education and decision-making processes for young individuals by utilizing image-based interactions and a chatbot interface. The LayoutLM Question Answering approach promotes proactive financial behavior, including consistent saves, budget planning, and informed investment decisions, fostering a more robust financial lifestyle.
Project Objectives
The team leverages innovative technology solutions such as the LayoutLM Question Answering model to create user-friendly interfaces. The objective is to simplify financial education by allowing youth to interact visually and ask questions via chatbots, making complex financial concepts more understandable and accessible.
AI Framework
Offer a unique value proposition for recording their data by taking a picture. They can even ask for any questions from the receipt. We delved into various visual image questioning datasets, encompassing FUNSD, Squadv2, DocVQA, and an additional dataset curated through collection efforts. These diverse datasets serve as the foundation for training and fine-tuning AI models.
Training Results
This is the training result of fine-tuning the model of Layout LMv2 on the FUNSD dataset for document understanding. We finetuned the Layout LM model on the FUNSD dataset, and we have reached a big milestone by earning an amazing F1 score of 0.909539.
The impressive outcome highlights the effectiveness of our method in utilizing the Layout LM architecture for problems related to interpreting documents, specifically in the context of the FUNSD dataset.
Summary of Accomplishments
The team have accomplished significant milestones including research, AI model, gathering dataset for training, fine tuning the model and a working question answering model from receipt. We have an impressive result with the accuracy of 97.09% on Fine tuning the state of the art model Layout LM v3 on the DocVQA. This model inspires a “purpose - driven” on document image analysis while improving the financial lives for youth with a streamlined data collection system.
We have designed a prototype for the app and incorporated features such as a receipt scanning system, chatbot, streamlined loan process, visualizations for spending, and budgeting tools.
