Commit 5dbc8a1c authored by Sascha Herzinger's avatar Sascha Herzinger

Relative paths in READMEs

parent 421d2bc1
......@@ -21,18 +21,18 @@ If you do not want to use docker or want a higher level of control of the severa
- Run and expose the Fractalis web service with whatever tools you want. We recommend **gunicorn** and **nginx**, but others should work, too.
- Run the celery workers on any machine that you want within the same network. (For a simple setup this can be the very same machine that the web service runs on).
Note: The [docker-compose.yml](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/docker/docker-compose.yml) describes how the different services are started and how they connect with each other.
Note: The [docker-compose.yml](docker/docker-compose.yml) describes how the different services are started and how they connect with each other.
### Configuration
Use the environment variable `FRACTALIS_CONFIG` to define the configuration file path.
This variable must be a) a valid python file (.py) and b) be available on all instances that host a Fractalis web service or a Fractalis worker.
Tip: Use the [default settings](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/fractalis/config.py) as an example for your own configuration file.
Tip: Use the [default settings](fractalis/config.py) as an example for your own configuration file.
Please note, that all this files combines [Flask settings](http://flask.pocoo.org/docs/0.12/config/), [Celery settings](http://docs.celeryproject.org/en/latest/userguide/configuration.html), and Fractalis settings, which are all listed and documented within this file.
Please don't overwrite default settings if you don't know what you are doing. This might have severe implications for security or might cause Fractalis to not work correctly.
### Add support for new services
Support for other services is exclusively implemented within [this folder](https://git-r3lab.uni.lu/Fractalis/fractalis/tree/master/fractalis/data/etls). We recommend looking at the *ada* example implementation. Just make sure you use the class inheritance (ETL, ETLHandler) correctly, and you will get readable error messages if something goes wrong.
Please refer to [this document](fractalis/data).
### Citation
Manuscript is in preparation.
......@@ -7,11 +7,11 @@ This behavior can be changed by setting the environment variables `FRACTALIS_HTT
For more detailed information please look into the files. They are rather self-explanatory and good place to do own modifications.
### Configuration (Fractalis / Celery / Flask)
1. Modify [docker/fractalis/config.py](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/docker/config/fractalis/config.py) before `running docker-compose up`.
1. Modify [docker/fractalis/config.py](config/fractalis/config.py) before `running docker-compose up`.
2. Replace the [dummy certificates](https://git-r3lab.uni.lu/Fractalis/fractalis/tree/master/docker/config/nginx/certs) with your own. The included one are only for development purposes.
2. Replace the [dummy certificates](config/nginx/certs) with your own. The included one are only for development purposes.
Tip: Use the [default settings](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/fractalis/config.py) as an example for your own configuration file.
Tip: Use the [default settings](../fractalis/config.py) as an example for your own configuration file.
Please note, that all this files combines [Flask settings](http://flask.pocoo.org/docs/0.12/config/), [Celery settings](http://docs.celeryproject.org/en/latest/userguide/configuration.html), and Fractalis settings, which are all listed and documented within this file.
Please don't overwrite default settings if you don't know what you are doing. This might have severe implications for security or might cause Fractalis to not work correctly.
......
......@@ -17,7 +17,7 @@ MicroETLs in Fractalis are submittable jobs that are responsible for the data
standard format, and the (L)oading into the analysis cache. MicroETLs can be
very simple or very complex. It highly depends on how easy it is, to extract
data into a workable format, but generally it should only take a few hours to
have some basic implementation. The [Ada Integer ETL](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/fractalis/data/etls/ada/etl_integer.py)
have some basic implementation. The [Ada Integer ETL](etls/ada/etl_integer.py)
is a good example for a simple MicroETL.
......@@ -27,10 +27,10 @@ There are very few restrictions on how a MicroETL should look like. It is
entirely up to you how to decide how to extract data from the service you want
to support. If your service offers a REST API, we recommend using the Python
requests module. Nothing stops you from directly accessing the database or some
files, though. Inspiration can be found [here](https://git-r3lab.uni.lu/Fractalis/fractalis/tree/master/fractalis/data/etls).
files, though. Inspiration can be found [here](etls).
The only real requirement is, that your MicroETL must inherit the
[ETL Class](https://git-r3lab.uni.lu/Fractalis/fractalis/blob/master/fractalis/data/etl.py).
[ETL Class](etl.py).
This class is responsible for making your MicroETL a submittable celery job and
that your MicroETL produces the correct internal format, among other things.
......@@ -59,24 +59,25 @@ is a distributed job framework with MicroETLs that executed Python/R scripts on
extracted data. Nothing stops you from loading brain image data, genomic data, or
financial data into Fractalis and code a visualisation for it. It doesn't mean you
should do that, though. There is two factors that should be taken into account:
1. **The data size.** It wouldn't be a good idea to move 50GB of genomic data into
Fractalis on a regular base, albeit not impossible.
1. **The data size.** It wouldn't be a good idea to move half a terabyte of
genomic data into Fractalis via REST API.
Instead you might want to consider connecting analyses or ETLs with other systems
like [Hail](https://github.com/hail-is/hail) to merge analyses results or data
from different sources into a single visualisation.
from different sources into a single visualisation managed by Fractalis.
2. **How much time you want to spend coding your own visualisation.** You can of
course import financial or wheather data into Fractalis, but you will likely not
profit much from the existing analysis scripts or visualisations. Fractalis focus
is explorative analysis in the field of translational research, so you should
is **explorative** analysis in the field of **translational research**, so you should
consider this, when thinking about adding a new format.
TL;DR: To see which formats are currently used and how they are defined, please
look at the [integrity check modules](https://git-r3lab.uni.lu/Fractalis/fractalis/tree/master/fractalis/data/integrity).
look at the [integrity check modules](integrity).
If you want to add a new data type, this is the only place you have to touch.
### FAQ
> Why is there no `load` method in the MicroETLs?
- There is, but you don't have to add it yourself. That's because the format
returned by `transform` is a internal standard format (after passing integrity checks),
so the loading step is the same for all MicroETLs of that type.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment