README.md 4.33 KB
Newer Older
Sascha Herzinger's avatar
Sascha Herzinger committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
### About

This page contains instructions on how data are loaded into Fractalis.

### General

First, it is important to understand that Fractalis, unlike other analytical 
platforms, does not have a persistent database in the traditional sense.
Data are "imported" on-demand into the analysis cache. Whether that happens via
REST API, some sort of data stream, or file import is entirely up to the
MicroETL.

### MicroETLs

MicroETLs in Fractalis are submittable jobs that are responsible for the data
(E)xtraction from the target service, the (T)ransformation into an internal
standard format, and the (L)oading into the analysis cache. MicroETLs can be
very simple or very complex. It highly depends on how easy it is, to extract
data into a workable format, but generally it should only take a few hours to
have some basic implementation. The [Ada Integer ETL](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/fractalis/data/etls/ada/etl_integer.py)
is a good example for a simple MicroETL.


### Implementation

There are very few restrictions on how a MicroETL should look like. It is
entirely up to you how to decide how to extract data from the service you want
to support. If your service offers a REST API, we recommend using the Python
requests module. Nothing stops you from directly accessing the database or some
files, though. Inspiration can be found [here](https://git-r3lab.uni.lu/Fractalis/fractalis/tree/master/fractalis/data/etls).

The only real requirement is, that your MicroETL must inherit the 
[ETL Class](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/fractalis/data/etl.py).
This class is responsible for making your MicroETL a submittable celery job and 
that your MicroETL produces the correct internal format, among other things.

You don't have to understand the ETL class in order to inherit from it. It is
designed in a way that you should always get a readable error if you do something
wrong. It won't hurt to have a look, though.

### Variables

They are all over the ETL code: 
`descriptor` (dict), `handler` (str), `server` (str), `auth` (dict).

It is up to the front-end to decide what they contain. They are used like this:
- Fractalis decides which MicroETL group/handler to use based on the `handler` 
(e.g. `ada`)
- The MicroETLs in that group decide whether they can handle the request based 
on the information in `descriptor` (e.g. `{'data-type': 'image', ...}`)
- The data are extracted from the `server` (e.g. `https://localhost`) 
- The ETL authenticates itself using authentication `auth` (e.g. `{'token': 123345}`)
- The ETL decides what to download from the server based on the `descriptor` (e.g. `{..., 'field': 'Age'}`)

### Internal Formats

Fractalis technically supports all formats. Yes, all. On a very basic level, Fractalis
is a distributed job framework with MicroETLs that executed Python/R scripts on
extracted data. Nothing stops you from loading brain image data, genomic data, or 
financial data into Fractalis and code a visualisation for it. It doesn't mean you 
should do that, though. There is two factors that should be taken into account:
1. **The data size.** It wouldn't be a good idea to move 50GB of genomic data into 
Fractalis on a regular base, albeit not impossible.
Instead you might want to consider connecting analyses or ETLs with other systems
like [Hail](https://github.com/hail-is/hail) to merge analyses results or data 
from different sources into a single visualisation.
2. **How much time you want to spend coding your own visualisation.** You can of
course import financial or wheather data into Fractalis, but you will likely not
profit much from the existing analysis scripts or visualisations. Fractalis focus
is explorative analysis in the field of translational research, so you should
consider this, when thinking about adding a new format.

TL;DR: To see which formats are currently used and how they are defined, please
look at the [integrity check modules](https://git-r3lab.uni.lu/Fractalis/fractalis/tree/master/fractalis/data/integrity).
If you want to add a new data type, this is the only place you have to touch.

### FAQ

> Why is there no `load` method in the MicroETLs?
- There is, but you don't have to add it yourself. That's because the format
returned by `transform` is a internal standard format (after passing integrity checks),
so the loading step is the same for all MicroETLs of that type.