LINDAT: Annotated dataset release

[K4TEL]

Two ways to solve the licensing problem:

1. modify old data shared by the old handle link
2. publish data on a new handle link + add old link redirection to the new link

In any case licensing data will be added for the list of document files, per file information to be added:

• author name and surname
• year
• organization
• doi
• document
• filename
• landing page link
• sha 512
• license

Which will be provided in CSV and JSON formats in addition to the already existing page image files, CSV of annotated categories and cross validation folds split of data samples

[motyc]

From my point of view, the first option is ok and probably safe for your case, but I have no strong preference. Also sharing the SHA of the files in the metadata may not be needed, it was more for your own reference if needed while pairing.

The important missing information is that no all the data used in training will be directly published. There will be a limited number of files (as discussed in emails) which does not currently allow open poblishing. Sharing their metadata and link to the AMCR Digital Archive/DOI is ok, however

[K4TEL]

annotated_documents_SMAPLE.json
not a full dataset
data_annotation_5_licensed_SAMPLE.csv
also, not a full dataset

these where made before the issue creation (experimental) - similar but bigger will be added to the updated dataset repository

the unkown license type documents will be removed from the shared zip archives of categories - their metadata in both json and csv (above) will remain untouched - as discussed

[motyc]

Agreed

[K4TEL]

The below output is from the read-to-publish directory:

venv) lutsai@stargate:/lnet/work/projects/atrium/transformers/pages/licensed_archives$ ls -lh
total 82G
-rw-rw-r-- 1 lutsai ufal_ext  809 čen 14 19:10 CITATION.cff
-rw-r--r-- 1 lutsai ufal_ext  12M čen 14 17:22 data_annotation_licensed.csv
-rw-r--r-- 1 lutsai ufal_ext  15K čen 14 17:22 data_annotation_not_included.csv
-rw-rw-r-- 1 lutsai ufal_ext 2,4M čen 14 18:57 dataset_crossval_folds.csv
-rw-rw-r-- 1 lutsai ufal_ext 5,0G čen 14 17:27 DRAW_L.zip
-rw-rw-r-- 1 lutsai ufal_ext 8,3G čen 14 17:26 DRAW.zip
-rw-rw-r-- 1 lutsai ufal_ext  19K čen 14 19:05 LICENSE
-rw-rw-r-- 1 lutsai ufal_ext  14M čen 14 19:01 licensed_crossval_folds.csv
-rw-rw-r-- 1 lutsai ufal_ext  13M čen 14 17:42 licensed_per_document.json
-rw-rw-r-- 1 lutsai ufal_ext 5,3G čen 14 17:27 LINE_HW.zip
-rw-rw-r-- 1 lutsai ufal_ext 1,7G čen 14 17:28 LINE_P.zip
-rw-rw-r-- 1 lutsai ufal_ext 9,2G čen 14 17:30 LINE_T_part001.zip
-rw-rw-r-- 1 lutsai ufal_ext 2,8G čen 14 17:30 LINE_T_part002.zip
-rw-rw-r-- 1 lutsai ufal_ext 8,1G čen 14 17:33 PHOTO_L.zip
-rw-rw-r-- 1 lutsai ufal_ext 7,3G čen 14 17:31 PHOTO.zip
-rw-rw-r-- 1 lutsai ufal_ext 4,4G čen 14 17:38 TEXT_HW.zip
-rw-rw-r-- 1 lutsai ufal_ext 9,2G čen 14 17:35 TEXT_part001.zip
-rw-rw-r-- 1 lutsai ufal_ext 9,2G čen 14 17:37 TEXT_part002.zip
-rw-rw-r-- 1 lutsai ufal_ext 7,2G čen 14 17:38 TEXT_part003.zip
-rw-rw-r-- 1 lutsai ufal_ext 2,4G čen 14 17:39 TEXT_P.zip
-rw-rw-r-- 1 lutsai ufal_ext 5,2G čen 14 17:40 TEXT_T.zip
-rw-rw-r-- 1 lutsai ufal_ext 3,0M čen 14 17:40 zip_archives_record.csv
(venv) lutsai@stargate:/lnet/work/projects/atrium/transformers/pages/licensed_archives$ cat CITATION.cff 
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: "ATRIUM's page classifier annotated dataset: historical page images manually annotated into content-specific categories for further NLP processing routing."
version: "1.1.0"
date-released: "2026-06-14"
authors:
  - family-names: "Lutsai"
    given-names: "Kateryna"
    orcid: https://orcid.org/0009-0002-4773-2797
  - family-names: "Straňák"
    given-names: "Pavel"
    orcid: https://orcid.org/0000-0002-6895-8536
  - family-names: "Novák"
    given-names: "David"
    orcid: https://orcid.org/0009-0005-8722-0245
  - family-names: "Křivánková"
    given-names: "Dana"
    orcid: https://orcid.org/0000-0001-5718-9447
repository-code: "https://github.com/ufal/atrium-page-classification"
license: "CC BY-NC 4.0"
(venv) lutsai@stargate:/lnet/work/projects/atrium/transformers/pages/licensed_archives$ head LICENSE 
Attribution-NonCommercial 4.0 International

=======================================================================

Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
(venv) lutsai@stargate:/lnet/work/projects/atrium/transformers/pages/licensed_archives$ head *.csv
==> data_annotation_licensed.csv <==
FILE,PAGE,CATEGORY,DOCUMENT,DOI,FIRST_AUTHOR,ORGANISATION,YEAR,LICENSE,LANDING_PAGE
CTX192000019,7,TEXT,C-TX-192000019,10.60585/C-TX-192000019,"Poutný, Lukáš","Institute of Archaeology of the Czech Academy of Sciences, Prague",1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000019
CTX192000047,1,LINE_HW,C-TX-192000047,10.60585/C-TX-192000047,"Jíra, Josef Antonín","Institute of Archaeology of the Czech Academy of Sciences, Prague",1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000047
CTX192000061,1,LINE_HW,C-TX-192000061,10.60585/C-TX-192000061,"Jíra, Josef Antonín","Institute of Archaeology of the Czech Academy of Sciences, Prague",1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000061
CTX192000072,1,LINE_HW,C-TX-192000072,10.60585/C-TX-192000072,"Šimák, Josef Vítězslav",[other organisation],1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000072
CTX192000152,1,TEXT_HW,C-TX-192000152,10.60585/C-TX-192000152,"Chadl, Jan Evangelista","Institute of Archaeology of the Czech Academy of Sciences, Prague",1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000152
CTX192000160,1,TEXT_HW,C-TX-192000160,10.60585/C-TX-192000160,"Šimák, Josef Vítězslav",[other organisation],1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000160
CTX192000163,1,TEXT_HW,C-TX-192000163,10.60585/C-TX-192000163,"Chadl, Jan Evangelista","Institute of Archaeology of the Czech Academy of Sciences, Prague",1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000163
CTX192000251,1,TEXT_HW,C-TX-192000251,10.60585/C-TX-192000251,anonym,[amateur archaeologist],1920,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192000251
CTX192100040,1,TEXT,C-TX-192100040,10.60585/C-TX-192100040,"Choc, Josef","Institute of Archaeology of the Czech Academy of Sciences, Prague",1921,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192100040

==> data_annotation_not_included.csv <==
FILE,PAGE,CATEGORY,LICENSE,DROP_REASON
CTX194004519,1,TEXT,unknown,disallowed_license
CTX194102101,1,TEXT_HW,unknown,disallowed_license
CTX194102101,2,TEXT_T,unknown,disallowed_license
CTX194102101,5,TEXT_T,unknown,disallowed_license
CTX194102101,6,TEXT_T,unknown,disallowed_license
CTX194102101,7,TEXT_T,unknown,disallowed_license
CTX194102101,8,TEXT_T,unknown,disallowed_license
CTX194102101,9,TEXT_T,unknown,disallowed_license
CTX194102101,10,TEXT_T,unknown,disallowed_license

==> dataset_crossval_folds.csv <==
file,page,categ,fold1,fold2,fold3,fold4,fold5
CTX192100263,1,DRAW,test,test,test,dev,train
CTX192500554,1,DRAW,train,dev,train,train,test
CTX192500814,2,DRAW,dev,train,dev,test,dev
CTX192600727,4,DRAW,train,train,train,train,train
CTX192700172,4,DRAW,train,train,train,train,train
CTX192700172,5,DRAW,train,train,train,train,train
CTX192700593,8,DRAW,train,train,train,train,train
CTX192700593,12,DRAW,train,train,train,train,train
CTX192700593,17,DRAW,train,test,test,test,train

==> licensed_crossval_folds.csv <==
file,page,categ,fold1,fold2,fold3,fold4,fold5,DOI,FIRST_AUTHOR,ORGANISATION,YEAR,LICENSE,LANDING_PAGE,DROP_REASON,ARCHIVE,PNG,BYTES
CTX192100263,1,DRAW,test,test,test,dev,train,10.60585/C-TX-192100263,"Dubský, Bedřich",[independent archaeologist],1921.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192100263,,DRAW.zip,CTX192100263-1.png,3457814.0
CTX192500554,1,DRAW,train,dev,train,train,test,10.60585/C-TX-192500554,anonym,"Institute of Archaeology of the Czech Academy of Sciences, Prague",1925.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192500554,,DRAW.zip,CTX192500554-1.png,2539214.0
CTX192500814,2,DRAW,dev,train,dev,test,dev,10.60585/C-TX-192500814,"Böhm, Jaroslav","Institute of Archaeology of the Czech Academy of Sciences, Prague",1925.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192500814,,DRAW.zip,CTX192500814-2.png,1807301.0
CTX192600727,4,DRAW,train,train,train,train,train,10.60585/C-TX-192600727,anonym,"Institute of Archaeology of the Czech Academy of Sciences, Prague",1926.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192600727,,DRAW.zip,CTX192600727-4.png,2722020.0
CTX192700172,4,DRAW,train,train,train,train,train,10.60585/C-TX-192700172,"Böhm, Jaroslav","Institute of Archaeology of the Czech Academy of Sciences, Prague",1927.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192700172,,DRAW.zip,CTX192700172-4.png,11034341.0
CTX192700172,5,DRAW,train,train,train,train,train,10.60585/C-TX-192700172,"Böhm, Jaroslav","Institute of Archaeology of the Czech Academy of Sciences, Prague",1927.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192700172,,DRAW.zip,CTX192700172-5.png,3978025.0
CTX192700593,8,DRAW,train,train,train,train,train,10.60585/C-TX-192700593,"Böhm, Jaroslav","Institute of Archaeology of the Czech Academy of Sciences, Prague",1927.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192700593,,DRAW.zip,CTX192700593-08.png,5209786.0
CTX192700593,12,DRAW,train,train,train,train,train,10.60585/C-TX-192700593,"Böhm, Jaroslav","Institute of Archaeology of the Czech Academy of Sciences, Prague",1927.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192700593,,DRAW.zip,CTX192700593-12.png,4082657.0
CTX192700593,17,DRAW,train,test,test,test,train,10.60585/C-TX-192700593,"Böhm, Jaroslav","Institute of Archaeology of the Czech Academy of Sciences, Prague",1927.0,Creative Commons Attribution-NonCommercial 4.0 International,https://digiarchiv.aiscr.cz/id/C-TX-192700593,,DRAW.zip,CTX192700593-17.png,3503670.0

==> zip_archives_record.csv <==
ARCHIVE,CATEGORY,DOCUMENT,PAGE,PNG,BYTES
DRAW.zip,DRAW,CTX192100263,1,CTX192100263-1.png,3457814
DRAW.zip,DRAW,CTX192500554,1,CTX192500554-1.png,2539214
DRAW.zip,DRAW,CTX192500814,2,CTX192500814-2.png,1807301
DRAW.zip,DRAW,CTX192600727,4,CTX192600727-4.png,2722020
DRAW.zip,DRAW,CTX192700172,4,CTX192700172-4.png,11034341
DRAW.zip,DRAW,CTX192700172,5,CTX192700172-5.png,3978025
DRAW.zip,DRAW,CTX192700593,8,CTX192700593-08.png,5209786
DRAW.zip,DRAW,CTX192700593,12,CTX192700593-12.png,4082657
DRAW.zip,DRAW,CTX192700593,17,CTX192700593-17.png,3503670
(venv) lutsai@stargate:/lnet/work/projects/atrium/transformers/pages/licensed_archives$ head *.json
[
  {
    "document": "C-TX-192000019",
    "doi": "10.60585/C-TX-192000019",
    "first_author": "Poutný, Lukáš",
    "organisation": "Institute of Archaeology of the Czech Academy of Sciences, Prague",
    "year": 1920,
    "license": "Creative Commons Attribution-NonCommercial 4.0 International",
    "landing_page": "https://digiarchiv.aiscr.cz/id/C-TX-192000019"
  },
(venv) lutsai@stargate:/lnet/work/projects/atrium/transformers/pages/licensed_archives$ 

It includes all of the applied changes with licensing and filtering of page samples

to be re-published