Document Automation Help and User Guide

1. Migration from DA v1 to DA v2

IMPORTANT INFORMATION: if the user is not already on Sicon v21.1 or higher, please make sure they upgrade to at least Sicon v21.1, preferably Sicon v22.1​.

DA v2 features can be added whilst DA v1 is still functioning, meaning testing can be carried out on the new version whilst the old version is still being used; therefore reducing disruption in the workplace

1.1. Enabling Document Automation v2

Sicon Documents v 21.1, 22.0 or 22.1 is required for this version of Document Automation. For versions 21.1 and 22.0, the build will need to be either 211.0.53 or 220.0.53 or higher. For 22.1, any build is fine, but we would always recommend that latest builds.

There are separate menu items for this version of Document Automation. The previous version is now in a menu folder called Legacy Document Automation. If you are using the previous version of Document Automation, you can still use it alongside version 2, but this is only recommended for a short time while you are in transition. The two versions are entirely separate, so invoices ready for processing in one version will not automatically appear in the other version.

You will also need an enable string for Document Automation. You will also need an API key which needs to be requested from Sicon. These then need to be entered in the Settings screen, found within Sicon Documents\Document Automation\Automation Settings. You can trial the system with 100 free documents once you have an API key.

Clicking Automation Settings will open the Sicon Document Automation Settings screen.

To enter the Enable String or Api Key, tick the box to the left of those fields and then enter the appropriate text. Click Save, and then assuming you have entered a valid value, a green tick will appear to the right of it.

1.2. Document Automation v2 New Features

• Improved table extraction, line recognition and order .
• Improved document recognition time; now just a few seconds rather than 15-30 minutes in some.​
• Supplier data is no longer stored outside of your client’s sage database, (limited info such as account reference is stored as a tag on the model but not things like address, vat number etc). All supplier recognition is performed locally with instant feedback on matching issues and the ability to fine tune and resolve in.​
• Data and documents leave the sage database for a few seconds during model training and are then deleted when this is.​
• Support for handwritten and scanned documents, (must be saved as a PDF to be processed).​
• Support for multiple buying/pricing.​
• Multiple regional formats.​
• Improved validation detail.​
• Ability to update supplier accounts directly from DA v2 screen.​
• Ability to query and assign documents to specific users. 

1.3. Migration Timeframe

The Cogidocs platform will be shut off on 30th September 2024, so it is imperative that all migrations are completed by this date to avoid any disruptions.

The Sicon Document Automation v2 platform on Azure is now performing very well with many satisfied customers following their migrations. As usual, we are happy to assist where needed to get your customer migrations completed.

1.4. Migration Steps

1.5. Migration FAQ's

What version do I need to upgrade to?

You can upgrade your customer to the latest iteration of Sicon documents within Sicon v21.1 or Sicon v22. The code and functionality is exactly the same in all versions. If their document version is older than this, you should upgrade them to Sicon v22.1. Use the Sicon auto installer for this process.

How do I access DAv2 once it’s installed? New features need to be assigned to the relevant user roles within the Sage System Admin to give access to the new DAv2 screens.

How long will I need to stop using Document Automation for during the migration? You can use both DAv1 and DAv2 at the same time while migrating them over, allowing super users to trial DAv2 in the live system before moving all users over to v2.

How do I start the extraction process? One main feature in v2 is the ability to drag & drop invoice attachments from emails onto a drop panel in Sage 200 which starts the extraction process. This may seem like a backward step but 80% of sites prefer this approach as it avoids the cost of the automation engine having to process junk email.

Is Email Integration available in DAv2? Email integration is available for Office 365, but will require the Sicon Web Api to be installed on the server and give the Sicon addons Enterprise Application in Azure permission to access your exchange 365 instance. This will require a onetime login as an azure global admin for the tenant to generate an OAuth token which can be revoked at any time. We do no not want to know the details, just that someone from IT will need to enter them at the required time. Shared mailboxes plus addressing/subdomains are supported.

Do I need to re-train the new system to read document content? Training of documents will need to be carried out again from scratch. However, one document training layout will be sufficient rather than 5 or more and as it is field based rather than coordinate based, it is much faster, easier and more accurate to complete.

How and where is the document identification and data extraction carried out? All Document identification and data extraction is carried out in the UK on the Microsoft Azure UK South platform.

What do I do if I still have document bundles to use up on DAv1? All remaining document bundle balances on DA v1 with Cogidocs will be transferred over to the new Sicon Azure DA v2 platform.

2. Processing Invoices

Once you have a document that you are ready to process, click onto Sicon Documents\Document Automation\Process Invoices & Credit Notes. This will open the Purchase Invoice and Credit Note Processing screen.

You can click New if you want to scan a new document now to display the Automation Document Drop screen. Alternatively, if you have configured the Exchange settings to use a mailbox to email the documents to, then you can click straight into the Process Invoices & Credit Notes screen to see the document there.

As an alternative to adding documents via the Purchase Invoice and Credit Note Processing screen, you can also open the Automation Document Drop screen via the Sicon Documents\Document Automation\Document Drop menu option.

You use this panel to add the new document. Currently supplier invoices, credit notes and customer purchase orders are the only document types, but more are planned with future development.

You can either drag and drop the document onto the relevant button (Credit Note, Invoice or Purchase Order), or click on the button itself to browse to the desired location and add the document that way.

If you drag and drop using the right mouse button, then you are presented with the Analyse Options screen at the time of adding the document. See the Analyse and Recognize section for further details.

After you have added a document, the processing of that document then happens which can take a short time. After this processing, the document will then be available in the Purchase Invoice and Credit Note Processing screen.

Check boxes at the bottom of this panel are available for the following:

Preview Documents After Analysis: If you tick this option then when the document is finished scanning, the Document Result window opens immediately allowing you to quickly progress the matching of fields scanned on the document.

Show Trader Identification Forms: If the supplier account wasn’t identified based on details scanned from the document, then instantly display the window to assist with supplier identification.

2.1. Purchase Invoice and Credit Note Processing Screen

This is the main screen from which you process the documents once they have been scanned.

This screen is split into three main sections. The pane in the top left displays the documents that are waiting to be processed. While the invoices/credit notes appear in this list, they have not yet been posted into the purchase ledger in Sage. The pane below this in the bottom left displays the item lines that relate to the document that is highlighted in the pane above. The pane on the right-hand side displays the scanned document for the document that is highlighted in the pane on the top left.

The documents are only displayed when you click the Load Documents button in the bottom left hand corner. This is by design, to enable users to configure and apply the filters above the top left pane before a potentially large number of documents is displayed.

The Refresh button updates the list of documents.

Documents are applied a Validation Status once they have been validated, and these filters at the top can be used to toggle whether documents with the relevant statuses are displayed. Once the documents are loaded, then these status filters will also display a numerical value equal to the number of documents with that status.

The Queried filter behaves differently to the other filters, in that if you have it toggled on then you only see documents with a Document Status of Queried. This is a different field to the other filters, which work using the Validation Status. This also means that a document in the Needs Attention count, will also be counted on the Queried tab. It also results in not necessarily seeing the same number of documents as the count on a tab would suggest, i.e. you have 4 documents with a Validation Status of ‘Receipt Issues’, so that shows a count of 4. If you queried one of them, then you would still see a count of 4, but would only see 3 documents if you toggle the filters so that you are only looking at your Receipt Issues filter. You would only see the 4^th one if you toggled the Queried filter, and then you would only be looking at that 1.

The Multiselect button can be toggled to add checkboxes to the left of the Document ID values. These can then be used if you wish to perform the same action against multiple ticked documents at the same time, i.e. delete, validate or process 10 documents.

The Filters button toggles the appearance of additional filter options, that enable you to filter for particular supplier etc.

Changing these options will cause the counts to change next to the validation statuses to reflect the filter. The Filters button will display a numerical value to reflect how many filters are being applied once the Apply Filters button is clicked. It is also the only filter that is remembered when you close the screen and then re-open it.

The fields are a mix of text fields and drop downs that allow selection of the following:

Filename of the desired document.

Document Id number (displayed as the first column in the list).

Max Records: Entering a number here will restrict the document list to only show up to the number entered. You would only do this if experiencing some kind of performance issue. Note that this is the only filter that is not cleared by the Clear Filters button, you would need to empty this manually.

Created By: Allows selection of a Sage user.

Assigned To: You can filter the list to just display documents assigned to a selected Sage user.

Supplier: Allows selection of a purchase ledger account.

Customer: Allows selection of a particular sales ledger account, not relevant for Purchase Invoices/Credit Notes.

Document Types: Allows selection of invoices and/or credit notes.

Sender Name: If the document was received via email, this will be their name as it was extracted from the email.

Sender E-Mail: If the document was received via email, this is the email address of the sender.

Once you have selected your desired filter options, then click Apply Filters to update the Documents list to reflect the filter. You can toggle off the filters box without losing the filter if you don’t want to lose the space taken up by the filter options. Click Clear Filters to remove the filter.

The four buttons at the end can be used to access the Document Automation Settings screen, display the settings for the document processing list, and toggle on/off the bottom panel and the right-hand side panel.

Depending upon your resolution or whether you don’t have the screen maximised, you may only see one of these buttons as follows:

This button at the end of the Filters icons row enables you to show/hide the right-hand panel that shows the scanned document. Clicking on the little down pointing arrow to the right of this gives access to the three further buttons:

The button on the left enables you to show/hide the bottom left panel, which is the one displaying the item lines for the current document.

The button on the left enables you to show/hide the bottom left panel, which is the one displaying the item lines for the current document.

The button in the middle accesses the Column settings for the Documents list.

This can be used to enable/disable columns as desired.

The Restore Column Order button can be used to reset the columns back to default order if you have changed them.

The Fade List Items on Refresh will remove the fade effect applied to the lines when they are being revalidated. You may want to do this for performance, or as desired.

The button on the right opens the Document Automation Settings screen, which is also accessible via its own Menu option in the Document Automation folder in the Sage menu.

Columns can be reordered by dragging the headings to the left or right, and these changes are stored per user meaning that different users can have them in different orders. You can also click the column headings to reorder the documents list by the values in that column.

Underneath the panels are the following buttons:

Load Documents: You need to click this button when first opening the screen to display the scanned documents.

New: Opens the Automation Document Drop Panel, when you want to add a new document.

Validate: Validates the currently selected document (or multiple if checked and using the Multiselect option).

Validate All: Validates all of the documents in the list.

Process Automatically: This will post the currently selected document (or multiple if checked and using the Multiselect option) using the automatic option, which will attempt to post without any further interaction needed by the user. Only recommended for documents with a Validation Status of ‘Ready for Processing’.

Process Manually: This will post the currently selected document (or multiple if checked and using the Multiselect option) using the manual option, which will open the Sage Record Purchase Invoice/Credit Note screen if the invoice is linked to a purchase order/return, otherwise it will open the Sage Enter Purchase Invoice/Credit Note screen as it’s posting direct to the Purchase Ledger.

Edit: Opens the Document Result window for the currently selected document (even if using the Multiselect option) which allows you to update the matching of fields scanned on the document.

View: Opens the same screen as the previous option, but in View only mode.

Delete: Deletes the currently selected document (or multiple if checked and using the Multiselect option) if you do not want to process it further.

The pane on the right-hand side six tabs above it.

Document

Shown as default, displays the scanned document.

These buttons enable you to Process, Train, Query, open the Supplier Recognition Details window, Edit, View, Delete, Analyse or Recognize. You may not see the Recognize button initially (depending upon your screen resolution), it may be accessible behind clicking on a downward pointing arrow to the right of the Analyse button (or whichever is the last button that you can see). You can drag the dividing line between the panels on the left and the document preview on the right-hand side. Dragging it to the left will then give you more space and cause the missing buttons to appear.

The first three buttons beneath this can be used to zoom in or out of the document, or resize the whole document. Click the downward arrow to see options for fit to width, fit to page or actual size.

The next two buttons will rotate the document anti-clockwise or clockwise.

The button that looks like a capitalised I toggles on/off a selection mode, which when on makes it possible for you to highlight text on the document if you then wish to copy it in order to paste somewhere else.

The remaining buttons toggle whether the following elements of a scanned document are highlighted:

Tables: Areas of the document that look like a table will be highlighted with this enabled.

Selection Marks: Highlights scanned items on the document that look like tick boxes to read them as true or false. Unlikely to be relevant for purchase invoices/credits.

Paragraphs: Scanned text that is grouped together can be identified as a paragraph. Not relevant for purchase invoices/credits.

Lines: Scanned text that is grouped together can be identified as a line. Not relevant for purchase invoices/credits/.

Words: Highlights each scanned text item separately to confirm what has been read.

Recognized Fields: Highlights the scanned values that have been matched to fields in Sage.

Identification Fields: Highlights the scanned values that have been used to identify the supplier account.

Tooltips: When hovering over any of the above enabled highlighted values on the document, a tooltip will be displayed confirming what is highlighted.

Validation

Shows the validation status of the various fields on the document. This tab should be referred to when reviewing documents that are at a different validation status than Ready for Processing.

This button will revalidate the document. Once this validation check is complete, you will see that the Last Validated date and time updates.

Items that need attention are identified by the presence of an orange exclamation mark to their left. Valid items will instead have a green tick to their left.

There is a Line Description Matching Threshold displayed to the right, which is a numerical value between 0 and 100. The higher the number, the stricter the matching on the line description is. You can adjust this if you are seeing validation errors informing you that the line cannot be matched to a purchase order line. You can lower this value and revalidate to see if the line can then be matched. The issue with lowering the number too far is that it could then start matching the wrong lines. This value will be remembered against each document separately as those documents are validated again.

Fields

This tab shows the values that have been matched to fields in Sage.

Edit Document: Enables you to Edit the document if you need to change any values that have been read from the document. E.g. The supplier has quoted the wrong order number on the document. Note that it is not possible to edit the read values directly in the list, you need to use this Edit facility to make any manual changes before processing the document.

View Document: Opens the same screen but in view only mode.

Email

If the document was received via email, then the email will be displayed on this tab.

Bundle Usage

This tab tracks how many bundle credits have been used in the processing of this document. For most documents, the Bundles Used figure will be just 1. But if additional Analyse or Recognize operations have been manually carried out on this document, then they will show on this tab and each will incur an additional bundle credit. The Cognitive Credits can be ignored.

In the example above, the document has used 3 bundle credits.  You can see the date and time each step has happened, and which Sage user did it.

Model

This tab shows the model information for this supplier. See the section on Models for further information.

Train Model: Opens a prompt asking if you want to train this model. See the section on training models for more information.

Edit Model: Opens the Model Maintenance window showing which documents are linked to the model.

View Model: Opens the Model Doc Types window showing the confidence the model has with each identified field.

Log

This tab shows a log of what has happened with this document within the time that this screen has been open. You are only likely to refer to this if directed as part of a support investigation.

Export allows you to save the displayed log entries as a csv file.

Clear resets the log.

Right Click Options

You can click the right mouse button in the first pane on it to access the following options.

Setting added to allow for suppliers to be copied between free text lines on project requisitions.

2.2. Document Tab

Shown as default, displays the scanned document.

These buttons enable you to Process, Train, Query, open the Supplier Recognition Details window, Edit, View, Delete, Analyse or Recognize. You may not see the Recognize button initially (depending upon your screen resolution), it may be accessible behind clicking on a downward pointing arrow to the right of the Analyse button (or whichever is the last button that you can see). You can drag the dividing line between the panels on the left and the document preview on the right-hand side. Dragging it to the left will then give you more space and cause the missing buttons to appear.

The first three buttons beneath this can be used to zoom in or out of the document, or resize the whole document. Click the downward arrow to see options for fit to width, fit to page or actual size.

The next two buttons will rotate the document anti-clockwise or clockwise.

The button that looks like a capitalised I toggles on/off a selection mode, which when on makes it possible for you to highlight text on the document if you then wish to copy it in order to paste somewhere else.

The remaining buttons toggle whether the following elements of a scanned document are highlighted:

Tables: Areas of the document that look like a table will be highlighted with this enabled.

Selection Marks: Highlights scanned items on the document that look like tick boxes to read them as true or false. Unlikely to be relevant for purchase invoices/credits.

Paragraphs: Scanned text that is grouped together can be identified as a paragraph. Not relevant for purchase invoices/credits.

Lines: Scanned text that is grouped together can be identified as a line. Not relevant for purchase invoices/credits/.

Words: Highlights each scanned text item separately to confirm what has been read.

Recognized Fields: Highlights the scanned values that have been matched to fields in Sage.

Identification Fields: Highlights the scanned values that have been used to identify the supplier account.

Tooltips: When hovering over any of the above enabled highlighted values on the document, a tooltip will be displayed confirming what is highlighted.

2.3. Validation Tab

Shows the validation status of the various fields on the document. This tab should be referred to when reviewing documents that are at a different validation status than Ready for Processing.

This button will revalidate the document. Once this validation check is complete, you will see that the Last Validated date and time updates.

Items that need attention are identified by the presence of an orange exclamation mark to their left. Valid items will instead have a green tick to their left.

There is a Line Description Matching Threshold displayed to the right, which is a numerical value between 0 and 100. The higher the number, the stricter the matching on the line description is. You can adjust this if you are seeing validation errors informing you that the line cannot be matched to a purchase order line. You can lower this value and revalidate to see if the line can then be matched. The issue with lowering the number too far is that it could then start matching the wrong lines. This value will be remembered against each document separately as those documents are validated again.

2.4. Fields Tab

This tab shows the values that have been matched to fields in Sage.

Edit Document: Enables you to Edit the document if you need to change any values that have been read from the document. E.g. The supplier has quoted the wrong order number on the document. Note that it is not possible to edit the read values directly in the list, you need to use this Edit facility to make any manual changes before processing the document.

View Document: Opens the same screen but in view only mode.

2.5. Email Tab

If the document was received via email, then the email will be displayed on this tab.

2.6. Bundle Usage Tab

This tab tracks how many bundle credits have been used in the processing of this document. For most documents, the Bundles Used figure will be just 1. But if additional Analyse or Recognize operations have been manually carried out on this document, then they will show on this tab and each will incur an additional bundle credit. The Cognitive Credits can be ignored.

In the example above, the document has used 3 bundle credits.  You can see the date and time each step has happened, and which Sage user did it.

2.7. Model Tab

This tab shows the model information for this supplier. See the section on Models for further information.

Train Model: Opens a prompt asking if you want to train this model. See the section on training models for more information.

Edit Model: Opens the Model Maintenance window showing which documents are linked to the model.

View Model: Opens the Model Doc Types window showing the confidence the model has with each identified field.

2.8. Log Tab

This tab shows a log of what has happened with this document within the time that this screen has been open. You are only likely to refer to this if directed as part of a support investigation.

Export allows you to save the displayed log entries as a csv file.

Clear resets the log.

2.9. Right Click Options

You can click the right mouse button in the first pane on it to access the following options.

Setting added to allow for suppliers to be copied between free text lines on project requisitions.

2.10. The Scanning Process

There are various stages that an invoice goes through before it is ready for processing.

The first step is to add the invoice to the system via the Automation Document Drop Panel. As part of the scanning process, the system is identifying the supplier, and then looking for a model for this supplier to use to match fields.

If the supplier has not been used previously with this version of Document Automation and the supplier has been identified, then you will see this message confirming that no model has been found for them for this document type:

Click OK on this message. At this point the document is added to the Purchase Invoice and Credit Note Processing screen with a status of ‘Not Validated’.

If the supplier has been identified and a model found, then you won’t see the previous message, and the document is added to the Purchase Invoice and Credit Note Processing screen with a status of ‘Not Validated’.

If the supplier has not been identified, then no message is displayed, and the document is added to the Purchase Invoice and Credit Note Processing screen with a status of ‘Not Validated’. When viewing the entry in the list, it should be obvious to see that the Supplier No. and Supplier Name columns are empty.

See the Supplier Identification section if the supplier isn’t identified.

The next step is to match the scanned details from the invoice to the fields that they relate to when posting the invoice/credit note into Sage. If an existing model was found then these fields will be matched already. For any supplier being used for the first time, then you will need to match the fields for the model to be generated. See the Training the Invoice/Credit layout section for further details.

Once the layout has been trained, then you will need to train the model. See the Model section for further details. Once the model is updated, then you need to validate the document again in order for the details in the model to be applied to this document.

At this stage, the document has been analysed and recognized. See the Analyse and Recognise section for further details, and when you might need to do either of these again.

Once you have validated the document again, it’s Validation Status will be updated to one of the following statuses:

Ready For Processing: The document has passed validation and is ready to be processed.

Needs Attention: The document has failed validation, and the reason for this should be investigated by referring to the Validation tab in order to find the details that should be investigated. See the troubleshooting validation section for more details. It is still possible to process these invoices, but you should investigate first, otherwise you could continue to see the same status on all future invoices from this supplier.

Receipt Issues: The document has failed validation, as it is linked to a purchase order where some items have not been sufficiently received. See the troubleshooting validation section for more details.

Once you are happy to post the invoice into Sage, then you can do this using either the Process Automatically option, to post it with no further input, or to use the Process Manually option to then be presented with either the Sage Record Purchase Invoice/Credit Note screen if the invoice is linked to a purchase order/return, or the Sage Enter Purchase Invoice/Credit Note screen as it’s posting direct to the Purchase Ledger.

Supplier Identification

Supplier identification is done by extracting fields from the scanned document, and comparing them to the following fields in the Purchase Ledger:

• VAT Number
• Bank Account Number
• IBAN
• Email Address
• Website

You can see which information has been used to identify the supplier using the I button toggle above the document preview.

Each of these values is assigned a score of 1. Whichever supplier has the highest score is the account that is then selected. Within the General Settings tab of Sicon Document Automation Settings, there is a setting where you can change the minimum score that needs to be reached for a positive supplier identification. You may want to increase this score if you have multiple purchase ledger accounts with either the same bank account number, or website for example.

It is also possible to configure specific supplier accounts with custom fine-tuning fields if there are other values on their documents that you wish to use for supplier identification.

The Supplier Recognition Details screen can be accessed via the Account button within the Document tab in the Purchase Invoice and Credit Note Processing screen. This will show which fields have been read from the document, and which accounts they are matched to. So, in the example above, there are three numbers in the document that could potentially be a VAT Number, and it has been able to match the final one to the KIT002 purchase ledger account. There are two numerical values that could have potentially been bank account numbers, but it hasn’t matched either to a bank account number in the purchase ledger. This has resulted in a score of 1 for KIT002, which is enough for a positive identification.

The Copy Document Value button copies whatever line you have highlighted in this screen. This is to make it easier to the just paste this value in the account via the Amend Supplier screen. So, for example, their VAT number has been read from the document but is not on the supplier. It will show in this screen, you can highlight it and click Copy Document Value. Then you can open the Amend Supplier screen, and paste it into the VAT number field in Sage.

Note that when comparing VAT numbers, spaces are ignored. However, if the supplier quotes GB (or other country text) as part of the VAT number being displayed in the Actual Value column above, then this text will also need to be in the VAT number field on the supplier account in the Purchase Ledger.

The Options button gives access to the standard Sage Amend Supplier and View Supplier screens, as well as new options for Update Supplier Fields and Update Fine Tuning Fields.

The Update Supplier Fields button will update the account with the relevant ticked values. So, in the above example, if the bottom value identified as a possible Bank Account Number was correct, you could tick that and then selecting this option would update these fields on the account without you needing to open the Amend Supplier screen to do so.

The Update Fine Tuning Fields button should be used instead if you have ticked the option on this supplier account to use fine tuning instead of the standard fields on the supplier record. If this is the case, then the fine-tuning values would be updated instead of the standard Sage fields.

Training the Invoice/Credit Layout

The act of training the document layout is what you do to match the scanned information to the relevant fields in Sage. When you click the Training button on the Document tab within the Purchase Invoice and Credit Note Processing screen in Sage, then the Document Labeller screen is opened.

Once you match the fields, these are then applied to a model which is used for future documents from this supplier. So, you will need to do this for the first document from that supplier, for each supplier.

To match fields, click onto the highlighted text on the left-hand side, and then click onto the relevant label on the right-hand side. In the example above, the invoice number is 090123001, this has all been identified as a single piece of text. Clicking onto this will highlight it in a darker colour. Then, over on the right-hand side, click anywhere on the words ‘Invoice Number’. This will then match that text to the Invoice Number label.

If the values on the left that you are matching contain multiple highlighted words, then you can draw a box around each word so that they are all highlighted or click onto the separate words one at a time.

The labels on the right each have a colour next to them. This is the colour that the matched element will be highlighted in when viewing the document later. You can click onto these colours here if desired to change them.

You cannot edit the values that have been scanned within this screen, training only allows for matching scanned information into the fields. If the values then need to be manually changed, then you need to use the Edit option from the Document tab back in the Purchase Invoice and Credit Note Processing screen.

If posting only to the purchase ledger, you will need all of the fields that you would expect to populate when completing the Sage Enter Purchase Invoice/Credit Note screen.  So, invoice number, invoice date, net value, tax value and total value.

Three Custom fields are available for you to match scanned text to. These should only be used to match some text that then needs to be split into multiple fields. See the section on Field Extraction for more information on this.

If matching the invoice/credit to a purchase order/return, then you will also need the order number. If you are trying to match the invoice lines to the lines on your purchase order/return, then you will also need to match the elements of the document lines to the relevant boxes in the Invoice Lines section. Note that you only need to do this for either invoices/credits that are part of a purchase order/return, or potentially have document lines that are spread across multiple purchase orders/returns. If neither of these apply to documents from this supplier (i.e. they only ever invoice/credit the entire purchase order/return on a single document), then you don’t need to match the Invoice lines fields as if the value of the document matches the outstanding value of the purchase order/return, then the system will match the lines automatically.

In the screenshot above, we have matched Line Quantity, Line Description (by drawing a box around all of the relevant text on the left-hand side), Line Unit Price and Line Net Value.

When matching invoice lines to the lines on a purchase order, it will match on quantity, unit price and net value before trying to match on description. It will also ignore lines on the purchase order that are already complete.

Our invoice though had three item lines on it. The buttons below give us a couple of choices as to how we add the other lines to our table of invoice lines.

The first button clears the table, by first displaying a warning where you have to click Yes before the table is cleared.

The middle button applies the identification applied to the first row to each other row identified on the document. There is a warning prompt displayed first, if you click Yes on this then the auto labelling will be carried out. With the example above, this is what was displayed after clicking this button and confirming on the warning:

The third button adds a blank row to the bottom of the Invoice Lines table. You would need to do this if the table on the document cannot be read as easily as in the example above, in which case you may need to build up each Invoice Line separately.

This button at the end of the row can be used to clear any table rows that are not needed. A warning prompt will first be displayed when you click on this button before the row is removed. You will still see the empty rows while you are in this screen, but they won’t be there the next time you view this document.

The following buttons are in the top right-hand corner of the screen and control which panels are displayed:

Each of these toggles on or off the following:

The bottom panel which displays the invoice lines table in a much wider form

The status bar which displays the cursor co-ordinates and which elements you are hovering over

The Summary fields displayed at the top of the panel on the right (Supplier account, Document Type Etc.)

The panel on the right which displays the fields that you are training against

The following buttons are available at the bottom of this screen:

Save: Saves the field matching against this document.

Account Identification: Opens the Supplier Recognition Details screen for this document.

Save & Train Model: Saves the field matching and then immediately trains the model. As with training the model from other screens, this can take as long as between 1-5 minutes for this training to happen.

Model Details: Opens the Model Summary window that this document is linked to.

Close: Closes the screen without saving.

Models

Once you have trained a document layout, this information can then be used to create or update a model. A model is stored per supplier, and effectively is the template used for future documents for that supplier. If you don’t have a model for a supplier, which would be the case for the first document processed for a supplier, then you will see a message advising that there is no model for this supplier. Separate models are maintained for invoices and credit notes from the same supplier.

Creating the model for the supplier is easy through the software. Once you have finished the training of the document, you can click the Save and Train Model button in the Document Labeller screen.

Alternatively, from the Model tab on the document in the Purchase Invoice and Credit Note Processing screen, you can click the Train Model button.

When you train the model, you will receive the following warning message:

Click Yes to train the model.

Once a model has been trained, then this information is visible on the Model tab of the document.

Further documents received from the same supplier will automatically have the details stored against the model applied to them. The model will become more accurate with more documents being used to update it. A combination of single page and multiple page documents from the same supplier (if applicable) would be ideal. For some suppliers, you might only ever need to have the one document for the model to be reliable, for others it could take five or even more.

Adding more documents to the model is done by clicking the Training button on more documents from that supplier; when you then save them the information from this document is then stored ready for the next time that the model is trained.

So, the model is only updated when the train model option is selected, using either method specified previously. When it is trained, the model is deleted and then recreated with all the documents linked to it being used for their information. This does mean that the training time of a model could take longer if it has more documents linked to it.

Clicking the Edit Model button on the Model tab opens the Model Maintenance screen, which displays the documents that are linked to this model and will update it the next time it is trained.

You can delete documents from the model if desired, such as if they reflect a departure from the usual layout for documents from that supplier.

The Model Type drop down should be left on Template as the default. The other option of Neural allows for much more sophisticated models that take much longer to train, and this option should only be selected if directed by the Sicon support department.

Clicking the View Model button on the Model tab opens the Model Doc Types screen.

Clicking the View button on here shows the Model Doc Type Details screen, which displays the confidence that the model has for each field within it:

Querying Documents

You can manually assign a Queried Document Status to a document to reflect that there is something that you need to look into or are waiting for someone else to confirm that the document is okay. Note that a document with a Document Status of Queried cannot be posted, you need to clear the query first.

This can be done by either using the Query button on the Document tab, or by right clicking on the document and choosing the Query Document option in that selection list.

When you do either of these, the Query Document window opens.

Enter a letter of your choice as the Query code, and then enter a note describing your query. Click the Query button to close the screen and save the query, or just the Close button if you don’t want to save it. If desired, you can also assign this query to a Sage user. This is a filter that is then available via the Filters at the top of the Purchase Invoice and Credit Note Processing screen.

Once the document has a Document Status of Queried, then these can be filtered to display using the Queried filter button at the top of the Purchase Invoice and Credit Note Processing screen.

The details of the query can be viewed via View Query, which is available on the Query button on the Document tab or as an option when right clicking on the document.

The query can then be cleared via Clear Query, which is also available on the Query button on the Document tab or as an option when right clicking on the document.

Troubleshooting Validation

If you have documents with a Validation Status of Needs Attention, then you should review the Validation tab for details on this document has this status. You are looking for entries that have an orange exclamation mark on their left.

The text to the right of this message should clarify what the issue is.

Example reasons to look for include:

• The invoice number exists in Sage already for this supplier.
• The invoice is matched to a purchase order that is complete.
• There is a variance in pricing from the purchase order that is greater than the acceptable variance values specified in Automation settings.
• No Order number has been matched, and Order Number matching is enabled for this supplier (see Supplier Settings for how to resolve this).
• Fields are missing and haven’t been trained before the document was validated.
• Total Value is not the total of Net Value and Tax Value.
• Lines on the document cannot be matched to lines on the order/return. Try lowering the Line Description Matching Threshold displayed in the top right hand corner and validate the document again.

Matching a Document to Multiple Orders/Returns

You can train the document to match the order/return number from fields. If there are multiple order/returns involved, then hopefully the supplier is quoting each relevant order/return number against each line on the document.

If the document has multiple lines on it that each quote the relevant order/return number, then these can be used to train each line to the relevant order/return number.

If the supplier is quoting multiple order numbers in a single field, then you will only be able to match to one order number. Then you will need to process the invoice via the Process Manually option in order to be able to select the other order/returns when you get through to the Record Purchase Invoice/Credit screen.

This can then be done by using the Amend Selected Orders button that appears between the Order Items and GRNs boxes on the left-hand side. After selecting this button, you can select one or more of the current orders/returns for this supplier, which will then populate all of the outstanding lines into the Order Items section. You can then manually match up the relevant items and quantities to the document.

Analyse and Recognize

These steps are done automatically when a document is added to the Purchase Invoice and Credit Note Processing screen, so you wouldn’t need to do these again for most documents. The automatic processing of these steps uses one bundle credit for this document. Manually requesting further analyse or recognize processing steps for the same document incurs an additional bundle credit for each of these operations.

Analyse – This is the step of reading the text from the document. You can have the system redo this step via the Analyze button on the Document tab. This will use another bundle credit to do this.

You can rerun this with alternative options via clicking on the little downward pointing arrow next to the Analyze button and choosing the Analyze With Options choice. This will use another bundle credit to do this.

This will display these Analyze Options:

The Pages option can be used to restrict which pages are being scanned. In the example above, we only want to scan the first page, so 1-1 is the correct value.

The remaining options would only be altered as recommended by Sicon Support department.

Recognize – this step is when the model is applied to the document. You can use the Recognize button to reapply the model to the document, this would be applicable if the model has been updated since the document was added to the screen and field matching would be improved by reapplying the model. This will use another bundle credit to do this.

You can rerun this with alternative options via clicking on the little downward pointing arrow next to the Recognize button and choosing the Analyze With Options choice. This will display the same options as above. This will use another bundle credit to do this.

Posting the Documents

Once you are happy to post the invoice into Sage, then you can do this using either the Process Automatically option, to post it with no further input, or to use the Process Manually option to then be presented with either the Sage Record Purchase Invoice/Credit Note screen if the invoice is linked to a purchase order/return, or the Sage Enter Purchase Invoice/Credit Note screen as it’s posting direct to the Purchase Ledger.

If processing via the manual option against a purchase order, the Accept and Post button will post the invoice. You can safely click the Close button to return to the Purchase Invoice and Credit Note Processing screen without actually posting the document if desired.

If processing via the manual option to post to the purchase ledger, the Save button will post the invoice. You can safely click the Close button to return to the Purchase Invoice and Credit Note Processing screen without actually posting the document if desired.

When a document is posted, it will initially remain in the Purchase Invoice and Credit Note Processing screen with a Document Status of ‘Completed’. But the next time the screen is refreshed, or closed and reopened, then the document will be removed from this screen.

3. Processing Sales Orders

Coming soon

4. Settings

The settings used by Document Automation v2 are all available via the Sicon Documents\Document Automation\Automation Settings menu option. A further button gives access to Automation Models.

There are also supplier/customer specific settings available on the Sicon tab against the supplier/customer record accessed via the normal Sage Amend Account Details screen.

4.1. Automation Settings

This settings screen is made up of six tabs.

Licence & Api Key

This tab shows your Document Automation enable string, as well as the Document Automation Api Key. The Usage section shows your bundle, as well as how many bundle credits have been used.

The numbers that matter on here are the Bundles, and the Bundles Used. So, in the example above, we have a bundle of 600 credits, and have used 140 of them.

You can use the ‘Warn if less than’ option to set a number that will then generate a warning message to users when breached.

General Settings

Here you can specify the Minimum Score for Trader Identification. See the section on Supplier Identification for more details.

You can also enable which formats of VAT Numbers that Document Automation should be looking for when trying to identify the supplier\customer from a document. So, you will need to activate matching on VAT Numbers from Ireland separately to VAT Numbers from the UK, or elsewhere in the EU. The EU options are initially hidden and can be displayed via the little arrow icon on the right-hand side.

Exchange Online

This is where you can configure the Exchange email settings if you would prefer to email the documents to a mailbox instead of dropping them onto the Automation Document Drop Form screen. This works by checking one or more mailboxes in your Exchange organisation, so the emails are not sent to an external email address.

If you wish to use the Exchange Integration, then you will need to have the Sicon WebAPI installed.

WARNING: Be aware that once you have configured a mailbox to be read by Document Automation, then any unread emails with attachments will be processed into the Process Invoice screen (and will use bundles in doing so). It is suggested that the mailbox should be checked first to ensure that there aren’t already unread emails in there that you don’t want this to happen with.

For this to work, you need to have Exchange Online, and have the details of a domain account that has full access to be able to manage your Azure domain applications.

There are two sub-tabs to configure here.

Authentication

Click Auto Configure and then sign into an account with enough permissions. The expectation is that whoever is responsible for the support on the Exchange organisation will need to be present for this step to ensure that the correct authentication details can be entered. They are only needed to generate the Directory ID and carry out the configuration that is done via this button, they are not stored or needed going forward.

This process will then result in the Directory (tenant) ID field being populated.

E-Mail Address for Testing: Once the Directory ID field is populated, then you can specify an Email address in your Exchange organisation that you wish to test that it can be read, and then click the Test button. All being well, the test will return this message:

E-Mail Addresses

Here you can specify the email addresses in your Exchange organisation that will be receiving the documents for the different entities, which are currently just Purchase Invoice, Purchase Credit and Sales Order. So, you need to specify the email address that is receiving Purchase Invoices, and the email address that is receiving Purchase Credits so that Document Automation knows which mailboxes to scan. You can use the same mailbox for both if desired, and just prefix it with a plus and either “invoices” or “credits”.

For example, you have a mailbox setup within your organisation called documentautomation@yourdomain.co.uk

You do not want to have separate mailboxes for invoices and credits. Instead you can use the same mailbox by using the following address format for the two entities:

Documentautomation+invoices@yourdomain.co.uk

Documentautomation+credits@yourdomain.co.uk

You would enter these two email addresses into the Inbox Email Address field for the relevant entity row. Then, these are the email addresses that would be used for whoever is sending the documents into that mailbox for automatic reading by Document Automation. Note that Document Automation would only be looking for emails sent specifically to one of these two addresses, i.e. not ones addressed to documentautomation@yourdomain.co.uk.

The Admin Email Address is optional and is only used with the Notify Admin checkboxes that follow towards the end of each row.

Notify Sender of Receipt: Check this if you want the sender of the email to receive a receipt email when sending documents through to the specified mailbox.

Notify Sender of Invalid: Check this if you want the sender of the email to receive an email when Document Automation can’t read any of the attachments on a scanned email.

Notify Admin of Receipt: Check this to send a receipt email to the Admin Email Address when a document is read from the relevant mailbox(es).

Notify Admin of Invalid: Check this to send a notification email to the Admin Email Address when Document Automation can’t read any of the attachments on a scanned email

Notify Admin of Error: Check this to send a notification email to the Admin Email Address when a more serious error has occurred when Document Automation was reading the attachments on a scanned email.

Purchase Invoice & Credits

These settings apply to both purchase invoices and credit notes, but the text of the settings mainly refers to invoices and orders for screen clarity.

Validate Invoice Age: Enable validation on the Invoice Age and set the number of days i.e. Flag as Needing Attention any Invoices that are dated more than 60 Days old.

Validate Invoices have a Second Reference: If this option is enabled, then all invoices will be validated to ensure they have a second reference field identified. Any that don’t will be flagged as Needing Attention. This can be overridden for individual suppliers via the Supplier Settings.

Validate Purchase Orders have valid GRN: Available if order number matching is enabled.  Will validate that the Purchase Order has been GRN’d in Sage, and flag them as invalid but they could still be processed. This can be overridden for individual suppliers via the Supplier Settings.

Validate Invoices have an Order Number: This will attempt to match the Invoice against an existing order within Sage.  The order number must appear on the Invoice or it will be flagged as Needing Attention. This can be overridden for individual suppliers via the Supplier Settings.

Validate Invoice Lines: Sicon Document Automation will read each line on the Invoice and these will be generated when processing the Invoices to attempt to match them to the lines on the purchase order. This can be overridden for individual suppliers via the Supplier Settings.

Trim Non-Numeric Characters from PO numbers: Tick this box if you would like the data matched as the PO number to only pass through the numerical digits.

Validate Invoices have a Project: This will attempt to match the Invoice against an existing Sicon Project within Sage.  The Project number must appear on the Invoice. Any that don’t will be flagged as Needing Attention. This can be overridden for individual suppliers via the Supplier Settings.

Order Line Analysis Code for Stock Item: This enables you to specify an analysis code on a purchase order line that Document Automation will use to match the invoice line to the PO line.

Order Analysis Code for Order No: This enables you to specify an analysis code on a purchase order that Document Automation will use to match the order number to.

Acceptable Variance %: Specify an acceptable variance percentage. If the invoice matches the value of the purchase order within this percentage then it will be considered valid. Any that don’t will be flagged as Needing Attention.

Acceptable Variance Value: Specify an acceptable variance value. If the invoice matches the value of the purchase order within this difference, then it will be considered as valid.  Any that don’t will be flagged as Needing Attention.

Ignore variance if the invoice is less than the purchase order: Do not use either of the above settings if the invoice is less than the purchase order. This would be relevant where either the invoice is only a part invoice of the purchase order, or the supplier is invoicing you less than you were expecting. This setting is designed to be used as a simpler alternative to using invoice lines validation.

Nominal Account for Variance: This enables you to specify a nominal account where any variance should be posted to. So, if you receive an invoice that differs from a purchase order by a value that falls within the acceptable options above, then you will see an extra nominal line when the invoice is posted using the nominal code set here for the variance.

Vat Code for Zero Tax Invoices: Invoices would ordinarily have the default tax rate from the supplier account applied to them. If the invoice is found to have zero VAT, then the tax code specified here would be used instead, to enable to you to select Exempt or Zero Rated (or another similar tax code). This can be overridden for individual suppliers via the Supplier Settings.

Validate against existing transactions and file images only: With this setting enabled, then no transactions would be created from Document Automation. Instead, the scanned invoices are matched to transactions already posted into Sage. So, you are using Document Automation just to add the images to the already posted transactions. This can be overridden for individual suppliers via the Suppler Settings.

Process Gross Value as Goods Amount: This is to disassociate VAT from the scanned Invoices for companies that don’t record VAT, so the total Invoice Value will be recorded as the goods amount

Disable Processing of Orders with no GRN: Prevent the posting of invoices that match to order lines that are missing their GRN details.

Purchase Orders & Returns

Enable Call-off Orders: Enabling this will activate a tick box on purchase orders, where you can then indicate that they will be call off orders. Ticking this on a purchase order will then flip the quantity and line value the other way around. This is useful when you are raising a free text purchase order with a value that will be invoiced in multiple invoices. This will then mean that partial receipts against that purchase order are not considered invalid when matching invoices to them. The expectation for all off orders is that there will only be a single line item and it will be free text. This means that any invoices matched to a call off order with more than one line will always register as Needing Attention once validated.

Sales Order

Treat no free Stock as invalid when processing Purchase Orders to Sales Orders: If this is ticked and the purchase order is for a stock item that has no free stock, then flag it as Needing Attention.

Treat no Price Book price as invalid when processing Purchase Orders to Sales Orders: If this is ticked and the purchase order is for a stock item that has no price setup in Price Book, then flag it as Needing Attention.

Always use Price Book prices: If this is ticked, then use the prices read from Price Book instead of the prices quoted by the customer on their purchase order.

4.2. Automation Models

This button opens up the Automation Models screen which displays all of the models that have been created for the different suppliers.

From here you can train, edit or delete any of the existing models.

Editing the model opens the Model Maintenance screen. See the previous section on Models for further details.

4.3. Supplier Settings

There are supplier specific settings that can be configured per supplier account if desired.

These can be found on the Sicon tab within the standard Sage Amend Supplier Details screen. If you have multiple Sicon addons enabled, then you need to click the Automation screen on the left-hand side.

The first check box enables you to disable Document Automation for this supplier. This would result in them not being included in searches during the supplier identification step of adding a new document. Note that this defaults to unchecked for all suppliers, so they are all included by default in Document Automation v2.

Fine Tuning

Enabling this setting will activate this section. This enables you to specify values for VAT Number, Sort Code, (Bank) Account Number, IBAN and Website for this supplier so that they match the ones that the supplier has on their documents.

You only need to do this if you do not want to change these details on the purchase ledger account itself. If you change them there, then those details would be used for supplier identification. Alternatively, you might want to blank out one or more of these fields on this tab to prevent them causing this account to be matched, if you have multiple accounts for the same supplier for example.

But if, as an example, for some reason you don’t want to change the bank details on the account, and they are different to the ones on the invoice, then follow these steps:

• Enable fine tuning for this supplier
• Click the Copy From Supplier button to populate the activated fields initially
• Click Yes on the prompt displayed
• Then make manual amendments to the bank account number and sort code here
• Going forward, the details on the Fine Tuning tab would be used for supplier identification when new documents are added. Changing the fields elsewhere on the account will have no effect on the supplier identification, unless you untick the Enable Fine Tuning checkbox for this supplier.

Custom Fine Tuning

This feature enables you to add additional fine-tuning options to assist with supplier identification. An example of this could be where you have multiple accounts for the same supplier, but they have some kind of value on their invoices which lets you know which account the invoice should be posted to. Each custom fine-tuning option added to a supplier is worth a score of 1 when matched, so might not be enough on its own depending upon the setting for Minimum Score for Trader Identification.

With fine tuning enabled, click the New button to add an additional option. The Custom Fine Tuning window opens:

Enter a description in the top field, then from the Type dropdown you can select either Word, Line or Cell. If the value you are looking for is a single field on the document that doesn’t contain any spaces, then you would select Word. Line would be appropriate for text containing spaces, or multiple words. Cell means that what you are looking for is contained within a table.

If you are unsure which of these options you should select, view the document in the Process Invoices & Credit Notes screen. On the Document tab, click the Hide/Show Tables button to highlight what text on the invoice is contained within tables. Click the Hide/Show Lines button to highlight which text is contained within single lines and lastly click the Hide/Show Words button to highlight which text is considered a separate word.

If you custom fine-tuning entry is based in either Word or Line, then select that in the dropdown, and then enter the text in the Content field.

If your custom fine-tuning entry is based on Cell, then you will need to identify the necessary values for the Table, Column and Row fields. These can be obtained by viewing the document in the Process Invoice and Credit Note screen, clicking the Hide/Show Tables button and then hovering over the field. The pop-up text will give you the Table, Column and Row values. You would then select Cell as the Type in the dropdown in the Custom Fine-Tuning screen and enter them in the fields in the Cell Options section. Finally, you would enter the text of the value itself in the Content field.

If your custom fine-tuning entry is based on Cell, then you will need to identify the necessary values for the Table, Column and Row fields. These can be obtained by viewing the document in the Process Invoice and Credit Note screen, clicking the Hide/Show Tables button and then hovering over the field. The pop-up text will give you the Table, Column and Row values. You would then select Cell as the Type in the dropdown in the Custom Fine-Tuning screen and enter them in the fields in the Cell Options section. Finally, you would enter the text of the value itself in the Content field.

The Exact Match checkbox means match only the text in the Content field in its entirety. If you don’t check this, then you are doing a ‘contains’ search for the text. i.e. Searching for 123 would match on a read word of ABC123. Checking the box only searches for 123 being read as its own full value.

So as an example, you have four accounts for the same supplier, let’s call then ABC001, ABC002, ABC003 and ABC004. In the purchase ledger, they all have the same values for VAT number and the same bank details. But they quote an account number of either A – 001, A – 002, A – 003 or A – 004 on their invoices, and this is what can be used to decide which of the accounts are correct. These values are not held in a table, just as text somewhere on the invoice.

On each of the four supplier accounts, you will need to enable fine tuning for that supplier. When you do this, you will then need to click the Copy From Supplier button, as with the fine tuning option enabled, Document Automation is now only using the details on this tab instead of reading the fields from the other tabs on the supplier account.

Then on each of the accounts, you will need to add a custom fine-tuning entry. This will need to be set to type Line, as it can’t be a single Word due to the presence of spaces within it. The text entered into each Description field can be whatever text you like, the important entries are selecting Line as the Type, and then entering the relevant text (A – 001 etc.) into the Content field for each of the four accounts.

When future documents are added for this supplier, the matching of the standard fields (VAT number, Bank details etc) will lead it to these four accounts. But if Document Automation finds a line entry of A – 003 (as an example), then the custom fine-tuning entry will lead it to account ABC003.

Field Extraction

On this tab, you can create Field Extraction rules for this supplier. These would be required where you need to extract multiple fields from a single item of text on their invoice. For example, they quote a reference number of 1234/6789 where the bit before the slash is the invoice number, and the bit after the slash is the order number.

Click the New button to display the Module, Entity and Document Type Selection screen.

Currently Document Automation only supports Purchase Invoices, Credit Notes and Sales Orders. So, the relevant Module selection for Invoices/Credit Notes is Purchase Ledger. From here you can select either Purchase Invoice or Purchase Credit as the Entity. Then choose the relevant Document Type for that Entity, so Invoice for Purchase Invoices or Credit Note for Purchase Credits.

Once you have populated these selections and click OK, then you are presented with the Field Extraction Maintenance screen.

In the Input Field Name, select the field that has been matched to the text that needs to be split. This matching will have happened as part of the training of the model for this supplier. Three custom fields are available to be selected, as it would make most sense that you would train a field that needs splitting into one of these.

The Field relates to document lines checkbox will change the fields available in the Input Field Name dropdown. Each field on a document is either classified as being a Header field (the default list displayed), or a Line field. So, tick the checkbox if the field that needs to be split is categorised as being a line field.

The Split Regex field is where you enter the character(s) that are where the value splits. Typically, this would be a hyphen or a slash.

Click the Add button to add the Output Fields, as many times as there are fields to be extracted.

So, for example, you have matched a field to the Custom 1 field in the training screen. This is some text with a hyphen in the middle, and the text before the hyphen is the invoice number, and the text after the hyphen is the order number. You would add two lines using the Add button, the first would have the Output Field set to Invoice Number and the second would be set to Order Number. You can then enter some example test matching this pattern into the Testing field and click the Test button. The results are displayed as follows:

Click Save to save your extraction rules for this supplier. You can have as many different extraction rules for the same supplier as required.

These field extraction options are applied when a document for this supplier is analysed – which happens automatically when you add a new document to the system. If you wish to redo the field extraction on a document, such as when you have updated the field extraction rules since adding the document to the system, then use the right click option of Local Field Extraction on the document itself in the Purchase Invoice and Credit Note Processing screen. This will avoid incurring the bundle cost of running the analyse operation again on the document.

Culture & Formats

If the supplier is from a different country, then if necessary you can populate this tab with details of their location. Choose the relevant culture from the dropdown, in this example we have selected that they are from the USA. This will assist with problems where the invoices use different separator symbols as either part of their numbers (i.e. commas instead of full stops) or different date formats.

Purchase Invoices & Credits

These options act as overrides for this supplier account for the settings specified in the Automation Settings, on the Purchase Invoices & Credits tab found there.

There are three possibilities for each setting with a checkbox:

1. Filled square – use the system default specified in Automation Settings.
2. Empty square – do not validate the element specified in the setting (or don’t receive the invoice image only).
3. Ticked square – validate the element specified in the setting (or receive the invoice image only).

4.4. Customer Settings

Coming Soon

5. Technical Information

Sicon Document Automation v2 uses Azure Cognitive Services to perform the scanning of documents and training of templates (referred to as models) in the cloud. All documents are stored locally within the Sage network environment, but there are a couple of points at which documents are temporarily uploaded, and then removed again when the relevant process completes.

This document will address when these moments happen, and how the process flows, and the communication methods used.

The communication happening between the Sage client and all the elements in the cloud is happening through a site’s regular internet connection. This connection is using HTTPS so that the information being sent and received is encrypted and secure. In terms of what ports need to be open for the client to access these, it’s using the standard port 443 for this communication so this port needs to be open. The expectation is that if a site can browse web pages on the internet, then this port is open. If specific URLs need to be permitted in the firewall, then these are the addresses that need to be accessed:

• azurewebsites.net
• sicon.co.uk

It is not currently configurable to access these via an HTTP proxy.

The servers hosting all the processes in the cloud are based in the UK. The models are all stored in Form Recognizer within Azure Cognitive Services. A SQL database in Azure is used to store all site API keys, bundle information and usage. All the communication between the elements in the cloud is also using HTTPS.

A copy of Microsoft’s Service Level Agreement for their Azure Cognitive Services can be found at:

https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services?lang=1

The live service status from them can be found here: https://azure.status.microsoft/en-gb/status

Note: Document Automation works with multiple document types. For ease of reading, the term trader is used in this document to refer to either customer or supplier. When the term Ledger is used, this would be Sales Ledger for customer documents and Purchase Ledger for supplier documents.

5.1. The Document Process

Adding documents into the Document Automation processing screen can be done by one of two methods:

1. Via the Automation Document Drop Form
2. Via being emailed to a configured mailbox and being read as an attachment in there. This mailbox is within the end user’s own Exchange Online organization

If you were using the second of these two options, and there was an issue with the status of the Microsoft azure services, then the emails would remain unread in the mailbox until the service was back up and running again.

The outcome of either of these options is that the document is now stored in a subfolder of the company attachments folder, as specified in Sage 200 System Administration.

At this stage, the document needs to be read by Azure Cognitive Services to identify the potential fields that can be used to match it to a trader. A copy of the document is sent from the Sage client to the Sicon Document Automation Web API in the cloud. This has a security step where the Azure SQL database is checked to ensure that they have a valid API key and that their usage does not exceed their purchased bundles.

This then communicates with Azure Cognitive Services to read the document fields. When this process is complete, then the fields that have been read are returned to the Sage client in the form of encrypted JSON values. At this point, the uploaded document is deleted by the Sicon Document Automation Web API.

Back in the Sage client, the JSON information is used to either identify the trader from matching details in the Ledger, or else flag it as not identified. If this was the case, then the user can use features of Document Automation in Sage to assess the details that have been read and update the relevant trader account with the missing details, to re-check the read trader details for the trader to now be identified. Note that all the trader information being used to match to a Ledger account is happening locally, i.e. the matching is not happening in the cloud.

With the trader identified for the document, the document now needs to be re-analysed to match it to a model, if one exists previously for that trader. This requires the document to be again uploaded to the Sicon Document Automation Web API, for further communication with Azure Cognitive Services. Once again, the Azure SQL database information about the site comes into play to ensure that they have a valid API key and have not exceeded their purchased bundle. The model for the trader is stored in Form Recognizer and is used to extract named fields from the content of the document. These are then returned to the Sage client, and the document is again deleted in the cloud by the Sicon Document Automation Web API.

This information retrieved to the Sage client will then trigger a validation of the document in Sage. This will result in a validation status of ‘Ready to Process’ or ‘Needs Attention’ or similar. Unless the document is re-analysed at this point, or used to train the model further, it will not be sent up into the cloud again.

5.2. The Model Training Process

Once a document has had the trader identified, it can be used to either create a new model, or else update an existing one if desired. This is triggered via the Training option in the Process screen, and once fields are matched in that Document Labeller screen, then the users can click a button marked ‘Save and Train Model’.

A model in Sicon Document Automation can have multiple documents associated with it, as each document that is used to train the model is linked to it. If a model doesn’t exist, then the first document for a trader that is used to train the model will be stored against it. There is one model stored per trader per document type, e.g. separate models for purchase invoices and purchase credit notes for the same supplier.

If training a model for the first time, then that document will be uploaded. If updating an existing model, then all documents linked to the model will be uploaded at the same time (there is a maximum limit of 100 documents against a model). If updating an existing model, then that model in Form Recognizer is deleted, and a new one is created.

When all the documents are uploaded, the Azure SQL database is checked again to ensure that their API key is valid and that they have not exceeded their purchased bundles.

Once the model has been created in the cloud, then the documents are deleted from the cloud. The new model details are returned to the local Sage client, where we maintain a link between the trader and its model for that document type.

This means that there are very few trader details held in the cloud. We only really hold the trader account number and their name in the Azure SQL database against the site id number to enable monitoring of how many models have been created to ensure that we maintain enough resources in the cloud services.

5.3. Exchange 365 Integration

Exchange 365 integration is handled by the windows service component of the Sicon Web API Platform. The service must be configured to run as a user account that is a member of the ‘Sage 200 Users’ group, and a member of the ‘Local Administrators’ group of the server its installed on.

The user account must be a member of the Sage 200 Users group as this is a requirement of the Sage 200 SDK and controls access to the companies in Sage 200 System Administrator. When E-Mails are processed from Exchange 365 the data is written back into the specified Sage 200 Company for Processing.

The user account must be a member of the Local Administrators group in order for the runtime log data to be written to the ‘Sicon’ event log in event viewer. This is useful for monitoring and troubleshooting.

In addition to the above groups. Its important the user account is in a group that has internet access. Without internet access, the service is unable to connect to the Sicon Document Automation App Service in Azure, or the Exchange 365 Api.

5.4. Security

The exchange integration is performed via the ‘Sicon Addons’ Enterprise application that registered in your Microsoft Entra tenant by creating an instance of a service principal by performing the auto configuration via the Document Automation settings.

As the application is running as a service and authenticating via OAuth with ‘app-only’ auth the ‘full_access_as_app’ scope is required (as opposed to delegated auth).

This scope by default will give access to all mailboxes in Exchange 365 that are then subsequently set up in Sage 200.

5.5. Configuring ApplicationAccessPolicy to limit mailbox access for Document Automation

Administrators who want to limit app access to specific mailboxes can create an application access policy by using the New-ApplicationAccessPolicy PowerShell cmdlet. This document covers the basic steps to configure access control. These steps are specific to Exchange Online resources and do not apply to other Microsoft Graph workloads.

Document Automation accesses Exchange 365 using its own identity and not on behalf of a user. This is common to background servers or deamon apps that run on the server without the presence of a signed-in user. Document automation uses OAuth 2.0 client credentials grant flow to authenticate and by default is granted access to all mailboxes on Exchange Online.

There are scenarios where administrators may want to limit an app to only specific mailboxes and not all Exchange Online mailboxes in the organization. Administrators can identify the set of mailboxes to permit access by putting them in a mail-enabled security group. Administrators can then limit third-party app access to only that set of mailboxes by creating an application access policy for access to that group.

5.5.1. Manage mail-enabled security groups in Exchange Online

A mail-enabled security group can be used to distribute messages and to grant access permissions to resources in Active Directory.

For more information visit: https://learn.microsoft.com/en-us/exchange/recipients-in-exchange-online/manage-mail-enabled-security-groups

We would recommend creating a mail-enabled security group to limit access to mailboxes as its easier to manage adding and removing users from the group, than to maintain access policies for individual mailboxes.

5.5.2. Use the Exchange Admin Centre to create a mail-enabled security group

1. In the Exchange Admin Centre, click Recipients > Groups > Mail-enabled security.
2. Click Add a group and follow the instructions in the details pane.
   1. In the Choose a group type section, click Mail-enabled security and click Next.
   2. In the Set up the basics section, enter the details and click Next.
3. In the Assign owners section, click + Assign owners, select the group owner from the list, and click Next.
4. Click Add members > + Add members, select the group members from the list, and click Next. The members are the list of Inboxes you would like Document Automation to have access to.
5. In the Edit settings section, enter the group email address, configure the following and then click Next:
   1. Privacy: Set it to either public or private.
   2. Add Microsoft Teams to your group: Select this to create a Team for your group.
6. In the Review and finish adding group section, verify all the details, click Create group, and then click Close.
7. Take a note of the email address used for the mail-enabled security group. For this example the email address used is dasecuritygroup@domain.co.uk

5.5.3. Configuring Application Access Policy

Application Access Policies can currently only be configured from a cloud shell.

The cloud shell can be accessed by logging into the Microsoft 365 Admin Centre and clicking the button to open a cloud shell.

1. Open a cloud shell
2. Connect to Exchange Online Powershell

Input the following commands into the cloud shell to connect to Exchange Online Powershell

a. Install-Module ExchangeOnlineManagement

b. Import-Module ExchangeOnlineManagement

c. Connect-ExchangeOnline

3. Create a New Application Access Policy

The ‘Application ID’ of the Sicon Addons Enterprise Application in Microsoft Entra is:

9ff16b33-d848-4a5c-af6d-1d9613c6d218

This can be verified by logging into Microsoft Entra Admin Centre, and finding the Sicon Addons application in the ‘All Applications’ section within ‘Enterprise Applications’

Input the following command into the cloud shell

a. New-ApplicationAccessPolicy -AppId 9ff16b33-d848-4a5c-af6d-1d9613c6d218 -PolicyScopeGroupId dasecuritygroup@domain.co.uk -AccessRight RestrictAccess -Description “Restrict Sicon Addons to members of the Document Automation Security Group”

Please Note: Replace ‘dasecuritygroup@domain.co.uk’ with the email address of the security group you created earlier. The ‘Description’ can be amended to anything that helps identify what the security group is used for.

5.5.4. Testing the Application Access Policy

The access policy can be tested via the cloud shell by passing the Application ID, and an email address to check access for.

1. Test a an email address that the application should be allowed to access

Input the following into the cloud shell

1. Test-ApplicationAccessPolicy -AppId 9ff16b33-d848-4a5c-af6d-1d9613c6d218 -Identity validemail@domain.co.uk

Replace ‘validemail@domain.co.uk’ with a valid email address for a member of the mail-enabled security group.

You should see an output containing:

AccessCheckResult : Granted

1. Test-ApplicationAccessPolicy -AppId 9ff16b33-d848-4a5c-af6d-1d9613c6d218 -Identity invalidemail@domain.co.uk

Replace ‘invalidemail@domain.co.uk’ with an email address that is not a member of the mail-enabled security group

You should see an output containing:

AccessCheckResult : Denied

5.5.5. Listing Application Access Policies

To list all currently configured Application Access Policies input the following command into cloud shell.

Get-ApplicationAccessPolicy | Format-Table -Auto

5.5.6. Removing Application Access Policies

To remove an Application Access Policy, first execute the command to list Application Access Policies. This will output a table containing the current policies. Take notes of the value in the  ‘Identity’ column, this will be required to remove the policy in this step

To remove the policy input the following command into a cloud shell

Remove-ApplicationAccessPolicy -Identity “Access Policy Identity”

Replacing Access Policy Identity with the identity obtained when listing Application Access Policies.

5.5.7. Important Information

Although testing the application access policies gives immediate results. The policy can take an hour or more to propagate over to Exchange 365. Once the changes have propagated across, if Document Automation is configured to access a mailbox that is not in the mail-enabled security group, the Exchange Web Services will start returning an http 403 error (Forbidden).

If new mail boxes are created in future for use with Document Automation. They will need to be added to the mail-enabled security group and may take an hour before they can be accessed by Document Automation.

6. Release Notes

The release notes page shows which release of the system includes new features or issues resolved.

At the release of Sicon v21 we announced that going forward, we will only be supporting Sage 200c and as such we are able to drop the year from our version numbers. We moved from 201.201.0.0 to 210.0.0.

New features detailed in the Release Notes relating to Pre-Release versions will not be detailed in the current Help and User Guide until the end of development phase.