OpenEM Network Ingestor

Please refer to https://www.openem.ch/ for an overview.

This repository provides a data ingestion service for dataset transfer and metadata registration in a catalog. It targets Scicat as dataset catalog.

Data can be transferred via Globus or to an S3 compatible endpoint.

The main entrypoint is a headless service that provides a REST API.

Building the Service

Generating the REST API

Based on the OpenApi specs in openapi.yaml, the REST API for the server implementation (Gin can be built:

/Ingestor$ go generate ./...

this will update api.gen.go.

/Ingestor$ go build ./cmd/ingestor-web-service

Building the App

The app is based on Fyne and can be cross-compiled for Windows on Linux. See the setup pages and the cross-compilation for the required dependencies.

Linux Build

Prerequisites

sudo apt-get update && sudo apt-get install libgl1-mesa-dev xorg-dev libxkbcommon-dev -y

In order to avoid doubly defined macros, the CGO_CFLAGS need to be set

CGO_CFLAGS="-Wno-error -U_FORTIFY_SOURCE" fyne build  --src cmd/openem-ingestor-ap

Windows Build

Windows builds can be created by cross-compiling the application

Prerequisites

On Ubuntu, in addition to the previous prerequisites, MinGW needs to be installed

sudo apt-get update && sudo apt-get install mingw-w64 gcc-mingw-w64-x86-64 -y

To following environment variables need to be set

CC	x86_64-w64-mingw32-gcc
CGO_ENABLED	1
CGO_LDFLAGS	-static -lssp
GOOS	'amd64'
GOARCH	'windows'

and the same build command can be used

fyne build  --src cmd/openem-ingestor-ap

Debugging

launch.json and task.json are provided to define debug targets for VS Code. Running the Debug Service task will start the service on the configured port (default: 8888) and a Swagger UI documentation page can be accessed at

http://localhost:8888/docs/index.html

Configuration

Configuration options are described in configs/ReadMe.md

Both the desktop app and the service will use a configuration file named openem-ingestor-config.yaml expected to be located next to the executable or in os.UserConfigDir()/openem-ingestor where the first takes precedence. As documented in the Go documentation), the following config locations are considered:

• Unix: $XDG_CONFIG_HOME/openem-ingestor/openem-ingestor-config.yaml if non-empty, else $HOME/.config/openem-ingestor/openem-ingestor-config.yaml
• MacOS: $HOME/Library/Application Support/openem-ingestor/openem-ingestor-config.yaml
• Windows: %AppData%\openem-ingestor\openem-ingestor-config.yaml

See configs/openem-ingestor-config.yaml for an example configuration.

Core

The core package contains main functionality. It makes use of the scicat-cli tools for interactions with Scicat. Two APIs are provided; a REST API for it interact with it as a service, and a Go API to interact with it within the same application.

Further Documentation

• General Configuration: docs/configuration.md
• Metadata Extractors: docs/metadataextractors.md
• Transfer: docs/transfer.md
• WebServer: docs/authentication.md
• Keycloak Setup for development: docs/keycloak-setup.md

Linux Deployment

For deployment instruction and example setup see openem-deployment repository.

Windows Deployment

Executable

The executable can be download directly from the releases page and executed. Alternatively, a Chocolatey package can be downloaded which installs the ingestor for all users.

Choco Package From Github

Install the ingestor using chocolatey

choco install openem-ingestor --source=https://nuget.pkg.github.com/swissopenem/index.json -params="'/Environment:prod'"

Parameter	Description
--force	to (re-)install a specific version or apply a configuration change
-y	accept w/o prompting

Note: You will be prompted for a Github username and password as Github does not allow for unauthenticated downloads.

There should be a Desktop shortcut. After starting, verify the ingestor is up and running by entering http://localhost:8888/version in a browser. The installed version should appear

Note: This command is lacking the collection location. See below.

Choco Package From Gitlab at ETHZ

From within the ETHZ network, no authentication is necessary and the Ingestor can be installed from gitlab.ethz.ch:

choco install openem-ingestor --force --source=https://gitlab.ethz.ch/api/v4/projects/64653/packages/nuget/index.json -params="'/Environment:prod'"

Choco Package Parameters

• /Environment: - One of "dev", "qa", "prod", will set the corresponding Scicat and backend URLs`
• /CollectionLocations: List of collection location name and path - example --params="'/CollectionLocations:storage1:\\path\to\storage1;storage2:\\path\to\storage2'"