DU_SERIES_Session1.pdf
- 1. Document Understanding Series Session 1 : Introduction Speaker :Mukesh Kala
- 2. 2 YouTube : Tutorials by Mukesh Kala https://mukeshkala.com Mukesh Kala ❑ 3 Times UiPath Most Valuable Professional ❑ UiPath Delhi Chapter Lead ❑ Certified Trainer | Speaker ❑ RPA Manager at Boundaryless Group
- 6. 6 UiPath Community MVP The Most Valuable Professional (MVP) Award is the highest recognition that we offer to our community members for their outstanding contribution, innovation, and evangelism shown in the larger automation community. • Stand out as a leading contributor in the AI-Powered world! • Envision the automation platform together! • Become the next UiPath Community MVP: Accelerate Your Automation Impact • Get recognized among the top contributors in the AI-Powered community! APPLY NOW!!!
- 7. 7 Introduction • Document is form of Information stored used and shared among set of user • It is a record of Information which is saved and can be retrieved later • Docs are used across Business such as Law , Media , Education , Government etc
- 8. 8 Documents and Types Structured Semi Structured UnStructured • Have Precise Format • User has specific areas to fill details • Fix Formats Always • Always have a Key Value Format • Does not Follow a Strict Format • Can be Changed. • Have a Common Enough Format • Contain Paragraphs • May contain Key Value with additional Info. • Info. Is not organized in a clear and structured manner. • Might have Key value with free text such as Paragraphs Surveys , Questionnaire , Registration forms , Passports Invoices , Receipts, PO, Bank Statements, Utility Bills Contracts , Reports , Leases , Annual Reports, News Documents are oftentimes combinations of the three categories above. A file might have a structured top part, followed by unstructured free- form content, or it might contain unstructured content but have specific pieces of information that always appear in very structured, repeating contexts.
- 9. 9 • Forms and low-diversity documents generally focus on collecting information in a precise format, guiding the person who is filling them with precise areas where each piece of data needs to be entered. • These come in a fixed format and are generally called forms. These forms can contain handwritten text, signatures, and checkboxes. These contain exclusively key-value pairs and tables. • Examples of structured documents: Surveys, questionnaires, tax forms, passports, licenses, and time sheets. Forms and Low Diversity Documents
- 10. 10 • Semi-structured documents are documents that do not follow a strict format the way structured forms do and are not bound to specified data fields. • These don't have a fixed form but follow a common enough format. They contain fixed and variable parts like tables. They may contain paragraphs as well, but data is mainly to be found in key-value pairs. • Examples of semi-structured documents: Invoices, receipts, purchase orders, healthcare lab reports, bank statements, and utility bills. Semi Structured Documents
- 11. 11 • Unstructured documents are documents in which the information isn't organized according to a clear and structured model. They have no fixed format. • These files are all easily comprehensible by human beings, yet much more difficult for a robot. Some documents may contain key-value pairs and tables, but much of the data is in the unstructured form inside the text. • Note: For unstructured text data (e.g. emails, chat conversations, support tickets) use Communications Mining(opens in a new tab). • Examples of unstructured documents: Contracts, leases, annual reports, agreements, and news. UnStructured Documents
- 12. 12 Rule Based Data Extraction • Relies on the set of Rules for Extraction. • Templates are created and Data is extracted • Works on Patterns. • Applies to Structured Doc. • String Manipulation / Regex can be Used
- 13. 13 Challenges - Rule Based Data Extraction • Closely Tied up to document layout • Always works on Fix Formats. • Any Change in layout can break and require Configuration.
- 14. 14 Model Based Data Extraction • Uses Machine Learning (ML) • Requires a Pre Trained ML Model • Requires Further Retraining Challenges • Time and Expertise to develop the ML Skills
- 15. 15 Introduction to OCR Digitize and Obtain the Text • Certain applications are not compatible with the usage of normal scraping or UI automation technologies. • OCR technology scan the entire screen of the machine, finding all the characters that are displayed. • Enables to create automations based on what can be seen on the screen, simplifying automation in virtual machine environments. • OCR is a method that reads text from images, recognizing each character and its position. • It comes handy in the Digitize step of the process when dealing with non- native documents, like scanned files.
- 16. 16 Document understanding is the ability to extract and interpret information and meaning from a wide range of document types, storage formats (e.g., images, PDFs), and objects (e.g., handwriting, signatures, checkboxes, logos). What is Document Understanding
- 17. 17 Challenges does document understanding solve? Problems Solutions Manually Extract, Interpret, and Take Action Delegation to robots using AI to understand documents. This involves automating the extraction and interpretation process using artificial intelligence. Varied Types of Documents and Low-Quality Images Automatic Processing using classification. The solution includes implementing algorithms that can classify and process various types of documents and handle low-quality images effectively. Increased Cost and Time Consumption Humans in the Loop to Validate. While robots handle the initial processing, human validation is introduced to ensure accuracy. This reduces the chances of errors and rework, ultimately saving time and costs. Human Work Varies, Rework Incurring Additional Cost DU (Document Understanding): Fast and Time-Efficient Solution. The use of advanced Document Understanding techniques ensures faster and more reliable document processing, minimizing variations in human work. Most Solutions are Partial and Not End-to-End Robots Are Accurate, Fast, and Consistent. The overall solution aims to provide end-to- end document processing by relying on robots that are accurate, fast, and consistent in handling various tasks.AI Infused Complete Solution. By infusing AI into the entire document processing workflow, a comprehensive and efficient solution is built, addressing the limitations of partial solutions.
- 18. 18 The Document Understanding Framework combines both Rule Based and the Model Based approach Document Understanding Taxonomy Digitize Classify Extract Validate Export Train What to Extract Obtain Text Identity Document Type Obtain Info. Confirm Extracted Results Export Validated Data Improvise OCR
- 19. 19 In this pre-processing step, we can add multiple document types and the fields we are interested in extracting. For example, we can work with Invoices, wanting to extract the vendor and the total amount, and With medical forms, wanting to extract insured ID number and patient name. Steps in the Framework Taxonomy (What to Extract) Digitization (Obtain Text) • As the documents are processed one by one, they go through the digitization process. • The difference for non-digital (scanned) documents is that we need to apply the OCR engine of our choice. • The outputs of this step are the Document Object Model and a string variable containing all the document text and are passed down to the next steps.
- 20. 20 After digitization, the document is classified. Need to know what type of document you're working with. Multiple classifiers can be used in the same scope and Trained Steps in the Framework Classification (Identify Document Type) Extraction (Obtain Information) • Extraction is getting just the data you are interested in. • In this framework, we can use different extractors, for the different document structures, in the same scope application. • The extraction results are passed further for validation.
- 21. 21 • The extracted data can be validated by a human user through the Validation Station. • Build logic around the decision of adding or not a human validation step, with rules depending on the specific use case to be implemented. • Validation results can then be exported and used in further automation activities. Steps in the Framework Validation (Confirm Extracted results) Export (Export Results) Export the results say in Excel Training Classifiers and Extractors • Document wasn’t classified properly; it means it was unknown to the active classifiers. • The Framework provides the opportunity to train the classifiers and the extractors, to improve recognition of the documents and fields.
- 22. 22
- 23. 23 Processing Native vs Scanned Documents • All documents that are to be processed (native and scanned) must pass through this step in order for the robot to understand the kind of document it's working with and what data is relevant. • while OCR is not the same as Document Understanding, we still need to use an OCR engine. One is expected when working with scanned documents or images. • The OCR engine will be used only if the incoming documents require OCR processing, and the decision gets taken on a page by page basis. • Fortunately, applying the Framework means using a single activity to digitize both scanned and native documents.
- 24. 24 What is Digitization • Digitization is the process of obtaining the machine-readable text from a given file, through the use of Optical Character Recognition (OCR). It is performed using the Digitize Document activity. • It detects all the words in the document and their x-y coordinates, whether by doing OCR - Optical Character Recognition (for scanned PDFs) or by getting the text directly from the native PDF documents. • When documents are in native PDF, OCR is typically not required. • It can also detect other things on documents, such as handwritten text, checkboxes, signatures, or barcodes/QR codes, depending on the OCR engine used. • OCR is able to extract text and metadata from the documents.
- 25. 25 How is OCR Deployed ? OCR is usually the most compute-intensive operation in the Document Understanding framework, and for this reason, we offer a number of different deployment options. This way, you are able to optimize both your costs and the latency of your automation, i.e. the time it takes to process each document. The OCR engines that are available by default are the following: UiPath Document OCR: Nearly all customers use UiPath Document OCR for printed text, checkboxes, handwritten text, signatures, barcodes, and QR codes. UiPath Chinese-Japanese-Korean OCR Print text; Handwritten text. Kofax Omnipage activity packs (available as Studio activity package) for printed text. In addition to these, the Document Understanding product also integrates with other OCR engines, including but not limited to Google Cloud Vision OCR and Microsoft Read OCR.
- 26. 26 ApplyOCRonPDF Setting Consider a PDF which has to be digitized. The default value of ApplyOCRonPDF is set to Auto. This determines if the document requires applying the OCR algorithm depending on the input document. If the ApplyOCRonPDF = Yes, the logic takes the branch to the right where the OCR is applied to all PDF pages of the document. If ApplyOCRonPDF=No, the logic takes the branch to the left, without calling the OCR, so only the digitally typed text is extracted. Digitization outputs the words and the x-y coordinates of their size and position on each page, but it also returns the raw text in the reading order. • This involves complex postprocessing including detecting lines, paragraphs, columns, headers, and footers. • The raw text is a key output especially in the context of downstream processing using Generative AI capabilities such as GPT Large Language Models
- 27. 27 Capabilities of the available OCR engines Engine Print Handwriting Checkboxes Signature Barcode Languages UiPath Document OCR Cloud ✔ ✔ ✔ ✔ ✔ Print: Latin based Handwriting (EN, FR, DE) UiPath Document OCR LocalServer ✔ ✔ ✔ ✔ ✔ Print: Latin based Handwriting (EN, FR, DE) UiPath Chinese-Japanese- Korean OCR ✔ ✔ ✘ ✘ ✘ Chinese Japanese Korean Kofax Omnipage activity packages ✔ ✘ ✘ ✘ ✘ https://docs.uipath.com/activities/docs/ omnipage-ocr(opens in a new tab) Google Cloud Vision OCR ✔ ✔ ✘ ✘ ✘ https://cloud.google.com/vision/docs/la nguages(opens in a new tab) Microsoft Azure Computer Vision Read OCR ✔ ✔ ✘ ✘ ✘ https://learn.microsoft.com/en- us/azure/cognitive-services/computer- vision/language-support
- 28. 28 • After digitization, the document is classified. In a project with multiple document types, you need to know what type of document you're working with, in order to extract data properly. • The key thing here is that you can use multiple classifiers in the same scope, you can configure the classifiers and, later in the process, even train them. The classification results help in applying the right strategy in extraction. What is classification? Why is classification important? • In a typical enterprise scenario, there will be many documents and they will typically be scanned into one. • With classifiers, the robot can identify the document types and split the file to process the documents accordingly.
- 29. 29 Overview of various classifiers Classifier name Where is it? What does it do well? Can it split document packs? Is it retrainable? Keyword Classifier Studio IntelligentOCR.Activities package Classify documents exhibiting specific words which are known in advance. No Yes Intelligent Keyword Classifier Studio IntelligentOCR.Activities package Classify documents of different types (e.g. invoices versus passports). Yes Yes Document Classifier/ML Classifier Document Understanding in Automation Cloud/Suite (AI Center must be enabled too) Classify documents of different types (e.g. invoices versus passports). No Yes Pre-trained classifiers UiPath provide a classifier that is trained on all of the document types available in the Pre-trained Extractors. This is available as an ML package which can be deployed in Automation Cloud or Automation Suite on-premises, or as a public endpoint that can be invoked directly in an automation workflow. The public endpoint is https://du.uipath.com/classify/MLclassification
- 30. 30 Keyword-based to ML-based classification.
- 31. 31 Extraction is getting just the data you're interested in. For example, extracting specific data from a 5-page document is quite troublesome if you want to do it with string manipulation. But in the Document Understanding framework, you can use different extractors for the different document structures, in the same scope application. The extraction results are passed further for validation. What is extraction? There are four extraction methods available: • Regex Extractor • Form Extractor • Forms AI • Semi-structured AI.
- 32. 32 Comparison of the available extraction methods Extractor name Where is it? What does it do well? RegEx Extractor Studio IntelligentOCR.Activities package Extracts single values with a known format on the page. Requires knowledge of Regular Expressions. Form Extractor Studio IntelligentOCR.Activities package Extracts regular values or simple tables from fixed forms. Requires some training to use effectively. Forms AI Document Understanding in Automation Cloud/Suite (AI Center must be enabled too). Requires DocumentUnderstanding.ML.Activities in Studio Extracts regular values or simple tables from fixed forms. Easy to use without special training. Semi-structured AI Document Understanding in Automation Cloud/Suite (AI Center must be enabled too) Requires DocumentUnderstanding.ML.Activities in Studio Extracts regular or complex values or tables from complex documents. Ease of use depends on the complexity of the documents.
- 33. 33 Pre-trained out-of-the-box extractors The out-of-the-box ML extractors are models that have already been trained and deployed by the UiPath Data Scientists. They can be leveraged by simply referencing them from the ML Extractor, inside the Data Extraction Scope activity, and don't require any additional setup. Advantages of using the out-of-the-box models • The biggest advantage of out-of-the-box models is that you can start processing documents with them right away. • The extractors accept files in PDF, PNG, JPEG, and TIFF formats and can automatically determine the location of critical pieces of information even if the format of the document changes. • They don't require any templates and can automatically identify and extract a wide range of entities from your documents. Even if the documents include "noise," the models will still find and extract the relevant information. Out of the many such models available, the following are the two of them: Invoices ML Extractor: This model is suitable for processing invoices in English, Spanish, Portuguese, German, and Romanian languages. Receipts ML Extractor: This model is suitable for processing receipts in English, Spanish, German, French, Norwegian, Finnish, and Romanian languages. The Invoices ML Extractor and Receipts ML Extractor models were trained on documents coming out of their respective countries, but they are suitable for any document using any of the Latin, Cyrillic, or Greek alphabets.
- 35. 35 • Document Understanding Process is a fully functional UiPath Studio project template based on a document processing flowchart. Its aim is to offer a Document Understanding tool at hand that is easy to use and easy to execute for any developer who wants to start working either on a simple demo or on large-scale implementation. • The process provides logging, exception handling, retry mechanisms, and all the methods that should be used in a Document Understanding workflow, out of the box. • The Document Understanding Process is preconfigured with a series of basic document types in a taxonomy, a classifier configured to distinguish between these classes, and extractors to showcase how to use the Data Extraction capabilities of the framework. Create a New Template from Studio Document Understanding Process is compatible with Studio version 21.4.4 or higher. For previous Studio versions, you can download the NuGet package from https://github.com/UiPath-Services/StudioTemplates/releases
- 36. 36 The workflow has an architecture decoupled from other connected automations: • it does not matter where the files to be processed are coming from or what triggers the execution, this is the responsibility of an upstream process; • it does not matter where the extracted information should be used, this is the responsibility of a downstream process. The architecture is common for both attended robots and unattended robots running Document Understanding and having human-in-the-loop validation via Action Center. • Items are not processed in bulk. Instead, there is one job for each input file. Indeed, a dispatcher is required every time but this flexible approach makes it simpler to get started, to develop, debug, and scale Document Understanding processes. • The process supports Visual Basic flavored projects. The default dependencies in a Document Understanding Process are: UiPath.DocumentUnderstanding.ML.Activities, UiPath.Excel.Activities, UiPath.IntelligentOCR.Activities, UiPath.PDF.Activities, UiPath.System.Activities, and UiPath.UIAutomation.Activities. Document Understanding Process
- 38. 38 How does licensing work? Licensing has to cover the following components: • Automation Developer licenses in order to build automation projects in Studio. • Unattended/attended Robot licenses to run the automations. • An Enterprise (or Enterprise Trial/Pro Trial) license for Automation Cloud/Automation Suite, with the Document Understanding, AI Center and Action Center third-party services enabled. • AI units.
- 39. 39 AI Units AI units are the licensing units consumed in projects involving document understanding, machine learning, as well as any other AI capabilities. We can buy packages of AI units, and they are consumed typically when doing classification and extraction in a AI unit/page system. AI unit consumption • Digitization: the use of OCR engines doesn't result in AI unit consumption, although it may require a Document Understanding license (for the UiPath proprietary OCR engines) or external licenses for engines like Abbyy, Microsoft Azure Computer Vision OCR or Google Cloud Vision OCR. But there are also free OCR engines, like Omnipage OCR or Tesseract OCR. • Classification: it may result in AI unit consumption for documents with more than 24 pages. Above this threshold, the consumption is incremental, but the maximum consumption will be 5 AI units no matter the number of pages. Keep in mind that the Keyword classifier is free. • Extraction: it typically results in AI unit consumption, with the basic logic being 1 AI unit per page. Keep in mind that there are extractors which are either free (RegEx extractor) or cheaper (0.2 AI units/page). • Validation and Human in the loop: this has no AI unit consumption attached.
- 40. 40 AI Units Consumption Examples A customer has a 3-page document, for which they use RegEx-Based Extractor and Form Extractor: RegEx-Based Extractor: free Form Extractor applied to 3 pages: 3 x 0.2 AI unit = 0.6 AI units. In this example, 0.6 AI units would be charged for processing a 3-page document. A customer has a 10-page document, for which they use Intelligent Keyword Classifier and combine Form Extractor and ML Extractor: • Intelligent Keyword Classifier: free (<24 pages) • Form Extractor applied to 6 pages: 6 x 0.2 AI units = 1.2 AI units • ML Extractor applied to 4 pages: 4 x 1 AI unit = 4 AI units In this example, 5.2 AI units would be charged for a 10-page document. A customer has a 100-page document, for which they use ML Classifier, RegEx-Based Extractor, Intelligent Form Extractor, and ML Extractor (several extractors can be applied to the same page): • ML Classifier: 4 AI units for 100 pages • RegEx-Based Extractor: free • Intelligent Form Extractor applied to 23 pages: 23 x 1 AI unit = 23 AI units. • ML Extractor applied to 81 pages: 81 x 1 AI unit = 81 AI units. In this example, 108 AI units would be charged for a 100-page document.
- 41. 41 DU Use Cases to relate to Case Study Industry Challenge Solution Impact Invoice Processing Automation Manufacturing High volume of manual invoice processing, prone to errors Implement Document Understanding for automated extraction of key invoice details Reduced processing time by 80%, minimized errors, improved efficiency Insurance Claims Processing Insurance Backlog of claims due to manual review processes Deploy Document Understanding to extract information from claim forms and supporting documents Accelerated claims processing by 50%, improved accuracy, enhanced customer satisfaction Legal Document Review Legal Services Time-consuming legal document review during discovery phases Implement Document Understanding for automatic identification and extraction of key details Reduced legal document review time by 60%, increased accuracy, enabled focus on high-value tasks HR Document Management Corporate Difficulty in managing and extracting insights from a large volume of resumes Use Document Understanding for automated extraction of relevant information from resumes Improved recruitment efficiency by 70%, enhanced candidate screening Compliance and Regulatory Reporting Finance Heavy burden on compliance officers for document analysis due to strict regulatory requirements Implement Document Understanding for automated extraction of key information from regulatory documents Reduced compliance review time, ensured accuracy in reporting, minimized regulatory risk
- 42. 42 DU + GenAI Use Cases Scenario Document Understanding Impact Gen AI Impact Healthcare Records Extracts structured data from medical records, improving efficiency Gen AI interprets complex medical research, aiding diagnosis Legal Document Review Automates extraction of key legal terms and clauses for analysis Gen AI helps lawyers by providing insights from vast legal databases Financial Statements Streamlines extraction of financial data, improving analysis Gen AI predicts market trends based on historical data Customer Support Emails Automates categorization and response to customer queries Gen AI generates personalized responses for complex issues Research Paper Analysis Extracts key findings and trends from research papers Gen AI generates summaries and identifies potential gaps in knowledge Invoice Processing Automates extraction of invoice details, reducing manual effort Gen AI predicts cash flow trends and provides financial insights HR Resume Screening Analyzes resumes to identify suitable candidates for a job Gen AI predicts employee retention and recommends training Insurance Claims Extracts information from claims forms, expediting processing Gen AI analyzes historical claims data to assess risk