Installation Notes of Holmes-ProcessingDec 9, 2016
Holmes Processing was born out of the need to rapidly process and analyze large volumes data in the computer security community. At its core, Holmes Processing is a catalyst for extracting useful information and generate meaningful intelligence. Furthermore, the robust distributed architecture allows the system to scale while also providing the flexibility needed to evolve.
I quoted the description from its website. The architecture looks like this:
Publication can be downloaded from here.
Binaries for OSX is provided on their home page, docker is a great idea of virtualizing applications instead of the whole OS. It simply runs on all the platforms.
docker for OSX comes with docker engine, docker-compose, and also docker-machine.
It is a message queue help you process messages asynchronously. The easiest way of installing it is via homebrew:
This will install rabbitmq in /usr/local/Cellar/rabbitmq/VERSION/
Binaries can be downloaded from their home page. Unzip the tar-ball, and optionally add the $CASSANDRA_HOME/bin to your PATH.
- distributed on nodes
- peer-2-peer network
- fault-tolorence by replicate
- high availability
- key-value data store
Passed for the moment, it worths awhile to take a look at fake-s3.
Installed fake-s3, run the fake-s3 server by:
the fake-s3 server is up and running. You need then configure it in Holmes-Storage.
Go-lange can be downloaded and installed without any problem. After installation, you need to do one thing:
To define the environment variable GOPATH, run command above as you like. Any binaries by go get xxx will be pulled & compiled in the workspace.
It is basically a wrapper for coordinating different databases on clusters, and is responsible to interact with data. To install, run bash:
This will install binaries in your GOPATH. One problem occurred is the “magic.h” is missing, it turns out you need to install dev version of magicmime: libmagic, which can be easily installed by homebrew.
Default config file is shipped, you can copy it and change your data storage settings accordingly.
Totem provides a set of static file analysis services. This can be downloaded and built by:
Of course you need to install sbt first by brew install sbt.
Run Totem services
To run totem services, checkout the individual services folder in src folder. In each service, you need to copy service.conf.example to service.conf.
Compile and run service in docker:
This will bring up services in docker.
At this point, services are all running in docker. You can run Totem by:
You will see then:
- configure local storage
- configure service confs
- manual tasking totem
- install holmes-dynamic
- install holmes-gateway
- install holmes-toolbox
- install RiakCS or S3-like storage