cluster_tools/README at master · mhulsman/cluster_tools · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
PREREQUISITES
=============
- add LFC_HOME to your environment,
   (not necessary if only low-level interface and/or executables are used)

   e.g. add to your script something like:
   export LFC_HOME='/grid/lsgrid/marc/'
   This directory will be used to store the data.

- startGridSession lsgrid

CLUSTER STORAGE
===============
Cluster storage performs all operations robustly
- it automatically retries on failure (with waiting periods)
- it replicates accross grid
- it uses hashing to check identity


Python objects
--------------
For sending large python objects, make use of submit / receive functions in cluster_storage.py:

#somewhere on grid, do:
id = submit(your_object)
#or, with fixed id:
id = submit(your_object,"your_id")

#somewhere else on grid, do:
object = receive(id)

#clean up
destroy(id)


Files
-----
Same as python objects, but use filenames:

#somewhere on grid, do:
id = submit_file(filename)

#or, with fixed id:
id = submit_file(filename,"your_id")

#somewhere else on grid, do:
filename = receive_file(id)

#clean up
destroy(id)


Low-level
---------
Use ClusterStorageEngine object, available as:  cluster_storage.cs

cs.is_file('/grid/lsgrid/marc/test.dat')
cs.delete_file('/grid/lsgrid/marc/test.dat')
cs.replicate_all('/grid/lsgrid/marc/test.dat')

cs.retrieve_file('/grid/lsgrid/marc/test.dat','local_name.dat')
cs.store_file('local_name.dat','/grid/lsgrid/marc/test.dat')

#do not redownload file to check if store really succeeded:
cs.store_file('local_name.dat','/grid/lsgrid/marc/test.dat', check_equal=False)

cs.list_dir('/grid/lsgrid/marc')

#list of storage servers
cs.get_other_storage_engines()


Executables
-----------
copy all files to bin directory, make them executable if necessary (use chmod), then:

#copy
gcp /grid/lsgrid/marc/test.dat local_name.dat
gcp local_name.dat /grid/lsgrid/marc/test.dat

#replicate
greplicate /grid/lsgrid/marc/test.dat

#delete
grm /grid/lsgrid/marc/test.dat

#directories
gmkdir /grid/lsgrid/marc/test
grmdir /grid/lsgrid/marc/test


#Note 1:
Grid paths can be indicated by prepending with grid:/
This can be useful with a command such as gcp, to distinguish local and grid paths
(Otherwise, paths starting with /grid/ are assumed to be on the grid). E.g:

gcp grid://grid/lsgrid/marc/test.dat local_name.dat

#Note 2:
If a grid path does not start with the / symbol, the command will prepend it with
the contents of LFC_HOME (i.e. your 'home' directory on the grid storage)

#create a dir test under the home dir:
gmkdir test

#copy a file to it
gcp local_name.dat grid:/test/test.dat