Setting up an Arcana

Info

This service is currently in a beta phase and can change regularly. The same applies to the documentation.

The process screen shots contain blank blocks. These are in position that will be replaced with the username.

Table of content:

First login

Go to the main Arcana page and click on register. You will need to log in with your Academiccloud account.

The register page of the RAG Manager welcomes the user, shows the Academiccloud profile and shows a button for registering to the service.

Once this is done, you are now able to go to the dashboard:

Once the register button is clicked, an additional green box appears stating that the registration was successful and that the account is ready to be used. Below is a button redirecting the user to the Dashboard.

which looks like this

The Arcana manager dashboard has a main part and a sidebar. The sidebar contains the Arcana manager tab, a profile tab, and near the bottom a link to the Data privacy and terms of use. The main part contains a list of arcanas, which is currently empty. Two buttons are on top, one for adding a new arcana and on to reload.

You can find an overview of your profile under the profile tab. This is also the place where you can delete your profile data.

The user profile tab shows the account information and a red box that reads danger zone and contains a button to delete the account.

Creating an arcana

Once you have the dashboard open, clicking on “+ Add New Arcana” will create a new instance. First, you need to specify a name for this arcana

The main Arcana manager, once pressed on the new Arcana button, displays a window in which a name can be set and a new Arcana created.

Once this is created, you should have it in your list of Arcana

The Arcana manager now contains a new Arcana which is not indexed and has no files. The actions section contains a button for edit, delete, and access link.

By clicking on it, you can open the detailed dialog

The menu for this new Arcana has three buttons up top, upload a new file, reload, and restart a failed conversion. The table below does currently not contain any files.

and click on upload to process a file

The upload a new file window has a button for choosing a file, as well as a cancel and done button.

Select a file and upload it. It is possible to upload PDF, Text, and Markdown files. Once you have clicked done, the file will be processed in the background

Once uploaded the menu for this Arcana now shows the file, which is not indexed and the Docling is in process.

and the “Conversion status” will read “Docling in Progress”. This will switch to completed once it is done

The Docling process has finished and the status changed to completed.

You can check the detail by clicking on the details button

The details menu shows the file information as well as the status. Additionally, it gives the option to download the processed Markdown plus file as well as a JSON file. Also, a new Markdown plus file can be uploaded.

This is also the page, where you can download the Docling as a JSON file or download the use material in the Markdown format. Also, you have the option to upload a new version of the file and delete it entirely. More importantly, you can download the Markdown file and update the markers mentioned in the Docling process. This is very useful to fine tune the splitting and marking of the file for better indexing and retrieval. Once these changes are made, uploading it will start a check process to make sure the content still matches the uploaded material.

Now that the file or files are prepared, you can go back to the main dashboard and check the full Arcana

The full list of Arcanas now also has the Action Generate Index available.

Here you can see that this Arcana is currently not indexed, meaning it is unlisted. If you have updated the material or the Markdown file, you also need to regenerate the Index. You will find the Index status to be “outdated” in this case. Clicking on “Generating Index” changes this

Clicking on generate index opens a new small window prompting the user to confirm that the index should be generated.

Which will change the status to a yellow “pending” and finally a green “indexed”.

The index status has changed from not indexed to pending. The index status has changed to Indexed.

Once your Arcana is index, you will be able to generate the Access Link and the ID for this Arcana. Click on “Access Link” to generate it.

Clicking Access link prompts the user to confirm the creation of an access link.

This page will update and either show directly the link or you may need to click on it again to get the link for your Arcana

Once the access link is created it show the link, which can be copied, as well as the Arcana ID and the Access Key.

You can click on the link or copy it into a new tab, which will open Chat AI with the Arcana ID preset into the advanced options

The Chat-AI window has Arcana ID and Key filled in.

Now you are able to share the link as well as the IDs and the Access Key for this Arcana.

Warning

You need to use the Meta Llama 3.1 8B RAG model for Arcana to work. It is in the bottom of the model list.

Updating files in an Arcana

There is an option to download and customize annotations for uploaded files, or modify previous manual or generated annotations. After the file has been uploaded and processed, you can update its annotation if necessary. Opening the details dialog of the file either shows this for a text file (Markdown files look similar)

The file details dialog for a txt file lists information about he file as well as the index and conversion status. Below the information, there is the option to download the Text/Markdown file and to upload an updated file with annotations.

The details dialog for a PDF has a few more options

The file details dialog for a PDF file lists information about the file as well as the index and conversion status. Below is the option to download the JSON output of the Docling process. At the bottom is the option to download the annotated Markdown file, upload an updated file and an option to reset the annotated Markdown file.

For all file types, there the is option to download the annotated Text/Markdown file. In case of a PDF, this annotated Markdown file - also called Markdown plus - is automatically generated by the Docling process. Also, both have the option to upload an manual annotated file (text or markdown). This is very useful to manually set the annotations or to updated them.

PDF files also have the options to reset the annotated Markdown file in case the uploaded file failed. Additionally, PDF have a JSON file from the Docling process that can be downloaded and viewed.

The avaiable annotations are explained in the Docling process under the heading Markdown Plus Annotations.

Info

Only fully annotated and not annotated files can be used. If you upload a Text or Markdown file, you can either use it without annotations or you need to fully annotate it.