Analytics Maestro

Analytics App is a module to analytics graph and create xml of Maestro Server, yours responsibility is:

  • Create grids
  • Create bussiness graph
  • Create network graph
  • Create infra graph
  • Drawing
  • SVGs

Maestro Server - Analytics maestro architecture

Analytics using Flask, and python >3.5, has api rest, and tasks.

Setup dev env

cd devtool/

docker-compose up -d

Will be setup rabbitmq and redis

Windows Env

If you use windows, celery havent support for windows, the last version is 3.1.25.

pip3 install celery==3.1.25

npm run powershell

Important topics

  • Controller used only graph to start all tasks:

  • The drawer process is compound by:

    • entry: First task, figure out all entry applications accordingly system endpoint parameters, our any direct application if avalaible.

    • graphlookup: Request for Data App a aggregate query using MongoDB $graphLookup.

    • network bussiness: Construct Grid Map, and send to enrichment and info bussines.

    • enrichment: Request for Data App all servers used on grid.

    • info bussiness: Calculate histogram, counts, density and connections.

    • network client: Request for Data App all clients used in grid.

    • draw bussiness: Create svgs based of grid.

    • notification: Send updates for Data App.

    • send front app: Send svgs to Analytics Front app.

      Each step have unique task.

  • Config is managed by env variables, need to be, because in production env like k8s is easier to manager the pods.

  • Repository has pymongo objects.


Flower - Debbug Celery

You can install a flower, it’s a control panel to centralize results throughout rabbitMQ, very useful to troubleshooting producer and consumers.

pip install flower

flower -A app.celery

npm run flower

Installation with python 3

  • Python >3.4
  • RabbitMQ

Download de repository

git clone https://github.com/maestro-server/discovery-api.git

Install dependences

pip install -r requeriments.txt

Install run api

python -m flask run.py

or

FLASK_APP=run.py FLASK_DEBUG=1 flask run

or

npm run server

Install run rabbit workers

celery -A app.celery worker -E -Q analytics --loglevel=info

or

npm run celery

Warning

For production environment, use something like gunicorn.

# gunicorn_config.py

import os

bind = "0.0.0.0:" + str(os.environ.get("MAESTRO_PORT", 5020))
workers = os.environ.get("MAESTRO_GWORKERS", 2)

Env variables

Env Variables Example Description
MAESTRO_PORT 5020 Port
MAESTRO_DATA_URI http://localhost:5010 Data Layer API URL
MAESTRO_ANALYTICS_FRONT_URI http://localhost:9999 Analytics Front URL
MAESTRO_WEBSOCKET_URI http://localhost:8000 Webosocket App - API URL
MAESTRO_SECRETJWT_PRIVATE XXX Secret Key - JWT private connections
MAESTRO_NOAUTH XXX Secret Pass to validate private connections
MAESTRO_WEBSOCKET_SECRET XXX Secret Key - JWT Websocket connections
MAESTRO_GWORKERS 2 Gunicorn multi process
CELERY_BROKER_URL amqp://rabbitmq:5672 RabbitMQ connection
CELERYD_TASK_TIME_LIMIT 10 Timeout workers