bug(medcat-trainer): Uploading exports with html issue

[mart-r]

There's an issue with uploading certain trainer exports (or other things in that format) to trainer. Specifically, HTML tag-including ones. If these aren't present, there doesn't seem to be any issue.

There's 2 underlying issues:

1. The text gets sanitised (through api.data_utils.sanitise_input) to remove HTML tags
2. There's some pandas serialisation and deserialisation happening as well
   • The data is read from the json
   • It is then converted into pd.DataFrame and saved on disk
   • And the Dataset.save() call triggers the Document creation from the file
   • Which reads the file again

The combination of the above two means that the text that's getting queried no longer matches the text that's in the database. And just using sanitise_input would also not fix the issue alone.

For reference, the type of failure I'm talking about is this:
https://gist.github.com/mart-r/74b8b44e96f816068166bb772c8f4519
As we can see, the document_id for a specific entity is None. And that's because the search for it returned nothing.

So the solution I've come up with is to keep track of the (unique!) names of each document and use those for the lookup instead. That way they always match.

Along the way I've also refactored the api.data_utils.upload_projects_export function and split it into different parts.

PS:
The current plan for this PR is to have the first workflow (that only adds the test) fail, and have the subsequent workflow (with the fix) to succeed.

[tomolopolis]

not sure I understand what has happened here? do no uploads work anymore or only some fail if they have HTML in the text?

Looks like most of this change is refactor rather than fixing the import function right?

[tomolopolis]

typo on MetaTask

[mart-r]

I can fix it, but was there before :)

[mart-r]

not sure I understand what has happened here? do no uploads work anymore or only some fail if they have HTML in the text?

It was only an issue for text with HTML tags - sorry, should have made that clearer.

Looks like most of this change is refactor rather than fixing the import function right?

Yes - 98% of non-test code is just refactor. The only change was this:

diff --git a/medcat-trainer/webapp/api/api/data_utils.py b/medcat-trainer/webapp/api/api/data_utils.py
index 56faf420..a4220f69 100644
--- a/medcat-trainer/webapp/api/api/data_utils.py
+++ b/medcat-trainer/webapp/api/api/data_utils.py
@@ -119,7 +119,7 @@ def _init_proj_ann_ents(
     return p
 
 
-def _create_dataset(proj: dict, p: ProjectAnnotateEntities) -> Dataset:
+def _create_dataset(proj: dict, p: ProjectAnnotateEntities) -> tuple[Dataset, dict[str, str]]:
     # escape - filename
     ds_file_name = MEDIA_ROOT + '/' + re.sub('/|\.', '_', p.name + '_dataset') + '.csv'
     names = [doc['name'] for doc in proj['documents']]
@@ -132,7 +132,13 @@ def _create_dataset(proj: dict, p: ProjectAnnotateEntities) -> Dataset:
     ds_mod.original_file.name = ds_file_name
     ds_mod.save()
     p.dataset = ds_mod
-    return ds_mod
+    # creating text 2 name mapping so we can find the doucments based on the name
+    # even if the text has been processed through pandas conversion and/or sanitisation
+    text2name: dict[str, str] = {
+        doc["text"]: name
+        for doc, name in zip(proj['documents'], names)
+    }
+    return ds_mod, text2name
 
 
 def _upload_project(
@@ -152,7 +158,7 @@ def _upload_project(
 
     ent_labels, meta_tasks, rels, unavailable_users, available_users = _prepare_state(proj)
 
-    ds_mod = _create_dataset(proj, p)
+    ds_mod, text2name = _create_dataset(proj, p)
 
     p.save()
 
@@ -200,11 +206,21 @@ def _upload_project(
 
 
     for doc in proj['documents']:
-        _process_doc(doc, p, ds_mod, available_users)
+        _process_doc(doc, p, ds_mod, available_users, text2name)
 
 
-def _process_doc(doc: dict, p: ProjectAnnotateEntities, ds_mod: Dataset, available_users: dict):
-    doc_mod = Document.objects.filter(Q(dataset=ds_mod) & Q(text=doc['text'])).first()
+def _process_doc(
+    doc: dict,
+    p: ProjectAnnotateEntities,
+    ds_mod: Dataset,
+    available_users: dict,
+    text2name: dict[str, str],
+):
+    # NOTE: using text2name mapping here since the text in the database
+    #       can be sanitised and changed during pandas serialisation (and deserialisation),
+    #       however we've kpet track of per-text names (which are unique) so will use these
+    #       instead here
+    doc_mod = Document.objects.filter(Q(dataset=ds_mod) & Q(name=text2name[doc['text']])).first()
     annos = []
     for anno in doc['annotations']:
         a = AnnotatedEntity()

[tomolopolis]

awesome thank you!