Turbinia - Open Source Framework For Digital Forensics Tools

Turbinia- Open Source Framework For Digital Forensics Tools

Turbinia is an open-source framework for deploying, managing, and running distributed forensic workloads. It is intended to automate running of common forensic processing tools (i.e. Plaso, TSK, strings, etc) to help with processing evidence in the Cloud, scaling the processing of large amounts of evidence, and decreasing response time by parallelizing processing where possible.

How it works

Turbinia is composed of different components for the client, server and the workers. These components can be run in the Cloud, on local machines, or as a hybrid of both. The Turbinia client makes requests to process evidence to the Turbinia server.

The Turbinia server creates logical jobs from these incoming user requests, which creates and schedules forensic processing tasks to be run by the workers. The evidence to be processed will be split up by the jobs when possible, and many tasks can be created in order to process the evidence in parallel. One or more workers run continuously to process tasks from the server. Any new evidence created or discovered by the tasks will be fed back into Turbinia for further processing.

Communication from the client to the server is currently done with either Google Cloud PubSub or Kombu messaging. The worker implementation can use either PSQ (a Google Cloud PubSub Task Queue) or Celery for task scheduling.

More information on Turbinia and how it works can be found here.

Status

Turbinia is currently in Alpha release.

Installation

There is an rough installation guide here.

Usage

The basic steps to get things running after the initial installation and configuration are:

• Start Turbinia server component with turbiniactl server command
• Start one or more Turbinia workers with turbiniactl psqworker
• Send evidence to be processed from the turbinia client with turbiniactl ${evidencetype}
• Check status of running tasks with turbiniactl status

turbiniactl can be used to start the different components, and here is the basic usage:

$ turbiniactl --help
usage: turbiniactl [-h] [-q] [-v] [-d] [-a] [-f] [-o OUTPUT_DIR] [-L LOG_FILE]
                   [-r REQUEST_ID] [-R] [-S] [-C] [-V] [-D]
                   [-F FILTER_PATTERNS_FILE] [-j JOBS_WHITELIST]
                   [-J JOBS_BLACKLIST] [-p POLL_INTERVAL] [-t TASK] [-w]
                   <command> ...

Optional arguments:

 

 -h, --help            show this help message and exit
  -q, --quiet           Show minimal output
  -v, --verbose         Show verbose output
  -d, --debug           Show debug output
  -a, --all_fields      Show all task status fields in output
  -f, --force_evidence  Force evidence processing request in potentially
                        unsafe conditions
  -o OUTPUT_DIR, --output_dir OUTPUT_DIR
                        Directory path for output
  -L LOG_FILE, --log_file LOG_FILE
                        Log file
  -r REQUEST_ID, --request_id REQUEST_ID
                        Create new requests with this Request ID
  -R, --run_local       Run completely locally without any server or other
                        infrastructure. This can be used to run one-off Tasks
                        to process data locally.
  -S, --server          Run Turbinia Server indefinitely
  -C, --use_celery      Pass this flag when using Celery/Kombu for task
                        queuing and messaging (instead of Google PSQ/pubsub)
  -V, --version         Show the version
  -D, --dump_json       Dump JSON output of Turbinia Request instead of
                        sending it
  -F FILTER_PATTERNS_FILE, --filter_patterns_file FILTER_PATTERNS_FILE
                        A file containing newline separated string patterns to
                        filter text based evidence files with (in extended
                        grep regex format). This filtered output will be in
                        addition to the complete output
  -j JOBS_WHITELIST, --jobs_whitelist JOBS_WHITELIST
                        A whitelist for Jobs that we will allow to run (note
                        that it will not force them to run).
  -J JOBS_BLACKLIST, --jobs_blacklist JOBS_BLACKLIST
                        A blacklist for Jobs we will not allow to run
  -p POLL_INTERVAL, --poll_interval POLL_INTERVAL
                        Number of seconds to wait between polling for task
                        state info
  -t TASK, --task TASK  The name of a single Task to run locally (must be used
                        with --run_local.
  -w, --wait            Wait to exit until all tasks for the given request
                        have completed

Commands:

 

 <command>
    rawdisk             Process RawDisk as Evidence
    googleclouddisk     Process Google Cloud Persistent Disk as Evidence
    googleclouddiskembedded
                        Process Google Cloud Persistent Disk with an embedded
                        raw disk image as Evidence
    directory           Process a directory as Evidence
    listjobs            List all available jobs
    psqworker           Run PSQ worker
    celeryworker        Run Celery worker
    status              Get Turbinia Task status
    server              Run Turbinia Server

The commands for processing the evidence types of rawdisk and directory specify information about evidence that Turbinia should process.

By default, when adding new evidence to be processed, turbiniactl will act as a client and send a request to the configured Turbinia server, otherwise if --server is specified, it will start up its own Turbinia server process.

Here's the turbiniactl usage for adding a raw disk type of evidence to be processed by Turbinia:

$ ./turbiniactl rawdisk -h
usage: turbiniactl rawdisk [-h] -l LOCAL_PATH [-s SOURCE] [-n NAME]

optional arguments:

  -h, --help            show this help message and exit
  -l LOCAL_PATH, --local_path LOCAL_PATH
                        Local path to the evidence
  -s SOURCE, --source SOURCE
                        Description of the source of the evidence
  -n NAME, --name NAME  Descriptive name of the evidence

Take Course

Beagle - An Incident Response And Digital Forensics Tool

Beagle is an incident response and digital forensics tool which transforms security logs and data into graphs.

Beagle Supported data sources include FireEye HX Triages, Windows EVTX files, SysMon
logs and Raw Windows memory images. The resulting Graphs can be sent to graph databases such as Neo4J or DGraph, or they can be kept locally as Python NetworkX objects.

Beagle can be used directly as a python library, or through a provided web interface.

The library can be used either as a sequence of functional calls.

>>> from beagle.datasources import SysmonEVTX

>>> graph = SysmonEVTX("malicious.evtx").to_graph()
>>> graph
<networkx.classes.multidigraph.MultiDiGraph at 0x12700ee10>

Or by strictly calling each intermediate step of the data source to graph process.

>>> from beagle.backends import NetworkX
>>> from beagle.datasources import SysmonEVTX
>>> from beagle.transformers import SysmonTransformer

>>> datasource = SysmonEVTX("malicious.evtx")

# Transformers take a datasource, and transform each event
# into a tuple of one or more nodes.
>>> transformer = SysmonTransformer(datasource=datasource)
>>> nodes = transformer.run()

# Transformers output an array of nodes.
[
    (<SysMonProc> process_guid="{0ad3e319-0c16-59c8-0000-0010d47d0000}"),
    (<File> host="DESKTOP-2C3IQHO" full_path="C:\Windows\System32\services.exe"),
    ...
]

# Backends take the nodes, and transform them into graphs
>>> backend = NetworkX(nodes=nodes)
>>> G = backend.graph()
<networkx.classes.multidigraph.MultiDiGraph at 0x126b887f0>

Graphs are centered around the activity of individual processes, and are meant primary to help analysts investigate activity on hosts, not between them.

Installation

Docker

Beagle is available as a docker file:

docker pull yampelo/beagle
mkdir -p data/beagle
docker run -v "data/beagle":"/data/beagle" -p 8000:8000 beagle

Python Package

It is also available as library.

Full API Documentation is available on https://beagle-graphs.readthedocs.io

pip install pybeagle

Configuration

Complete overview of each configuration entry

Any entry in the configuration file can be modified using environment variables that follow the following format; BEAGLE__{SECTION}__{KEY}.

For example, in order to change the VirusTotal API Key used when using the docker image, you would use -e parameter and set the BEAGLE__VIRUSTOTAL__API_KEY variable:

docker run -v "data/beagle":"/data/beagle" -p :80008000 -e "BEAGLE__VIRUSTOTAL__API_KEY=$API_KEY" beagle

Environment variables and directories can be easily defined using docker compose

version: "3"

services:
    beagle:
        image: yampelo/beagle
        volumes:
            - /data/beagle:/data/beagle
        ports:
            - "8000:8000"
        environment:
            - BEAGLE__VIRUSTOTAL__API_KEY=$key$

Web Interface

Beagle's docker image comes with a web interface that wraps around the process of both transforming data into graphs, as well as using them to investigate data.

Uploading Data

The upload form wraps around the graph creation process, and automatically uses NetworkX as the backend. Depending on the parameters required by the data source, the form will either prompt for a file upload, or text input.

 For example:

• VT API Sandbox Report asks for the hash to graph.
• FireEye HX requires the HX triage.

Any graph created is stored locally in the folder defined under the dir key from the storage section in the configuration. This can be modified by setting the BEAGLE__STORAGE__DIRenvironment variable.

Optionally, a comment can be added to any graph to better help describe it.

Each data source will automatically extract metadata from the provided parameter. The metadata and comment are visible later on when viewing the existing graphs of the data source.

Browsing Existing Graphs

Clicking on a data source on the sidebar renders a table of all parsed graphs for that data source.

Graph Interface

Viewing a graph in Beagle provides a web interface that allows analysts to quickly pivot around an incident.

The interface is split into two main parts, the left part which contains various perspectives of the graph (Graph, Tree, Table, etc), and the right part which allows you to filter nodes and edges by type, search for nodes, and expand a nodes properties. It also allows you to undo and redo operations you perform on the graph.

Any element in the graph that has a divider above it is collapsible:

Inspecting Nodes and Edges

Nodes in the graph display the first 15 characters of their a specific field. For example, for a process node, this will be the process name.

Edges simply show the edge type.

A single click on a node or edge will focus that node and display its information in the "Node Info" panel on the right sidebar.

Focusing on a Node

Focusing on an Edge

Expanding Neighbours

A double click on a node will pull in any neighbouring nodes. A neighbouring node is any node connected to the clicked on node by an edge. If there are no neighbors to be pulled in, no change will be seen in the graph.

• This is regardless of direction. That means that a parent process or a child process could be pulled in when double clicking on a node.
• Beagle will only pull in 25 nodes at a time.

Hiding Nodes

A long single click on a node will hide it from the graph, as well as any edges that depend on it.

Running Mutators

Right clicking on a node exposes a context menu that allows you to run graph mutators. Mutators are functions which take the graph state, and return a new state.

Two extremely useful mutators are:

1. Backtracking a node: Find the sequence of nodes and edges that led to the creation of this node.

• Backtracking a process node will show its process tree.

2. Expanding all descendants: From the current node, show every node that has this node as an ancestor.

• Expanding a process node will show every child process node it spawned, any file it may have touched, and pretty much every activity that happened as a result of this node.

Backtracking a node

Backtracking a node is extremely useful, and is similar to doing a root cause infection in log files.

Expanding Node Descendants

Expanding a node's descendants allows you to immediately view everything that happened because of this node. This action reveals the subgraph rooted at the select node.

Toggling Node and Edge Types

Sometimes, a Node or Edge might not be relevant to the current incident, you can toggle edge and node types on and off. As soon as the type is toggled, the nodes or edges of that type are removed from the visible graph.

Toggling a node type off prevents that node type to be used when using mutators, or when pulling in neighbours.

Undo/Redo Action and Reset

Any action in the graph is immediately reversable! Using the undo/redo buttons you can revert any action you perform. The reset button sets the graph state to when it loaded, saving you a refresh.

Graph Perspectives

As you change the graphs current state using the above action, you might also want to view the current set of visible node and edges in a different perspective. The tabs at the top of the graph screen allow you to transform the data into a variety of views:

Graph (Default perspective)

• Tree
• Table
• Timeline
• Markdown

Each of the perspectives supports focusing on nodes by clicking on them.

Python Library

The graph generation process can be performed programatically using the python library. The graph generation process is made up of three steps:

1. DataSource classes parse and yield events one by one.
2. Transformer classes take those inputs, and transform them into various Node classes such as Process.
3. Backend classes take the array of nodes, place them into a graph structure, and send them to a desired location.

The Python package can be installed via pip:

pip install pybeagle

Creating a graph requires chaining these together. This can be done for you using the to_graph() function.

from beagle.datasources import HXTriage

# By default, using the to_graph() class uses NetworkX and the first transformer.
G = HXTriage('test.mans').to_graph()
<networkx.classes.multidigraph.MultiDiGraph at 0x12700ee10>

It can also be done explicitly at each step. Using the functional calls, you can also define which Backend you wish to use for example, to send data to DGraph

from beagle.datasources import HXTriage
from beagle.backends import DGraph
from beagle.transformers import FireEyeHXTransformer

# The data will be sent to the DGraph instance configured in the
# configuration file
backend = HXTriage('test.mans').to_graph(backend=DGraph)

# Can also specify the transformer
backend = HXTriage('test.mans').to_transformer(transformer=FireEyeHXTransformer).to_graph(backend=DGraph)

When calling the to_graph or to_transformer methods, you can pass in any arguments to those classes:

from beagle.datasources import HXTriage
from beagle.backends import Graphistry

# Send the graphistry, anonymize the data first, and return the URL
graphistry_url = HXTriage('test.mans').to_graph(backend=Graphistry, anonymize=True, render=False)

You can also manually invoke each step in the above process, accessing the intermediary outputs

>>> from beagle.backends import NetworkX
>>> from beagle.datasources import HXTriage
>>> from beagle.transformers import FireEyeHXTransformer

>>> datasource = HXTriage("test.mans")
>>> transformer = FireEyeHXTransformer(datasource=datasource)
>>> nodes = transformer.run()
>>> backend = NetworkX(nodes=nodes)
>>> G = backend.graph()

If you want to manually call each step, you will need to ensure that the Transformer class instance is compatible with the output of the provided DataSource class.

• All Backends are compatible with all Transformers.

Each data source defines the list of transformers it is compatible with, and this can be accessed via the .transformers attribute:

>>> from beagle.datasources import HXTriage
>>> HXTriage.transformers
[beagle.transformers.fireeye_hx_transformer.FireEyeHXTransformer]

Controlling Edge Generation
By default, edges are not condensed, that means that if a process node u writes to a file node v 5000 times, you will have 5000 edges between those nodes. Sometimes, especially when trying to visualize the data, this may overwhelm an analyst.

You can condense all 5000 edges into a single edge for that type of action (wrote in this case), by passing the backend class the consolidate_edges=True parameter,

For example:

SysmonEVTX("data/sysmon/autoruns-sysmon.evtx").to_graph(NetworkX, consolidate_edges=False)
# Graph contains 826 nodes and 2469 edges.

SysmonEVTX("data/sysmon/autoruns-sysmon.evtx").to_graph(NetworkX, consolidate_edges=True)
# Graph contains 826 nodes and 1396 edges

By default, the web interface will consolidate the edges.

Download Beagle

Mad-Metasploit For Custom Modules Plugins And Resource Script

Metasploit custom modules, plugins, resource script and.. awesome metasploit collection

Awesome

open https://github.com/hahwul/mad-metasploit/blob/master/awesome.md

Add mad-metasploit to metasploit framework

1. config your metasploit-framework directory

$ vim config/config.rb
$metasploit_path = '/opt/metasploit-framework/embedded/framework/'
#                    /usr/share/metasploit-framework

2 A. Interactive Mode

$ ./mad-metasploit

2-B. Commandline Mode(preset all)

$ ./mad-metasploit [-a/-y/--all/--yes]

Use custom modules

search auxiliary/exploits, other..

HAHWUL > search springboot

Matching Modules
================

   Name                                          Disclosure Date  Rank    Check  Description
   ----                                          ---------------  ----    -----  -----------
   auxiliary/mad_metasploit/springboot_actuator                   normal  No     Springboot actuator check

Use custom plugins

load mad-metasploit/{plugins} in msfconsole

HAHWUL > load mad-metasploit/db_autopwn
[*] Successfully loaded plugin: db_autopwn

HAHWUL > db_autopwn
[-] The db_autopwn command is DEPRECATED
[-] See http://r-7.co/xY65Zr instead
[*] Usage: db_autopwn [options]
-h          Display this help text
-t          Show all matching exploit modules
-x          Select modules based on vulnerability references
-p          Select modules based on open ports
-e          Launch exploits against all matched targets
-r          Use a reverse connect shell
-b          Use a bind shell on a random port (default)
-q          Disable exploit module output
-R  [rank]  Only run modules with a minimal rank
-I  [range] Only exploit hosts inside this range
-X  [range] Always exclude hosts inside this range
-PI [range] Only exploit hosts with these ports open
-PX [range] Always exclude hosts with these ports open
-m  [regex] Only run modules whose name matches the regex
-T  [secs]  Maximum runtime for any exploit in seconds

etc...

List of

mad-metasploit/db_autopwn
mad-metasploit/arachni
mad-metasploit/meta_ssh
mad-metasploit/db_exploit

Use Resource-scripts

 #> msfconsole

 MSF> load alias
 MSF> alias ahosts 'resource /mad-metasploit/resource-script/ahosts.rc' 
 MSF> ahosts
 [Custom command!]

List of rs

ahosts.rc
cache_bomb.rb
feed.rc
getdomains.rb
getsessions.rb
ie_hashgrab.rb
listdrives.rb
loggedon.rb
runon_netview.rb
search_hash_creds.rc
virusscan_bypass8_8.rb

Archive(Informal metasploit modules)

archive/
└── exploits
    ├── aix
    │   ├── dos
    │   │   ├── 16657.rb
    │   │   └── 16929.rb
    │   ├── local
    │   │   └── 16659.rb
    │   └── remote
    │       └── 16930.rb
    ├── android
    │   ├── local
    │   │   ├── 40504.rb
    │   │   ├── 40975.rb
    │   │   └── 41675.rb
    │   └── remote
    │       ├── 35282.rb
    │       ├── 39328.rb
    │       ├── 40436.rb
    │       └── 43376.rb
.....

Patch mad-metasploit-archive

 #> ln -s mad-metasploit-archive /usr/share/metasploit-framework/modules/exploit/mad-metasploit-arvhice
 #> msfconsole

 MSF> search [string!]
 ..
 exploit/multi/~~~
 exploit/mad-metasploit-arvhice/[custom-script!!]
 ..    

How to update?

mad-metasploit

$ ./mad-metasploit -u

mad-metasploit-archive

$ ruby auto_archive.rb

or 

$ ./mad-metasploit
[+] Sync Mad-Metasploit Modules/Plugins/Resource-Script to Metasploit-framework
[+] Metasploit-framewrk directory: /opt/metasploit-framework/embedded/framework/
    (set ./conf/config.rb)
[*] Update archive(Those that are not added as msf)? [y/N] y
[-] Download index data..

How to remove mad-metasploit?

$ ./mad-metasploit -r

 or

$ ./mad-metasploit --remove

Development

Hello world..!

 $ git clone https://githhub.com/hahwul/mad-metasploit

Add to Custom code

./mad-metasploit-modules
 + exploit
 + auxiliray 
 + etc..
./mad-metasploit-plugins
./mad-metasploit-resource-script

#hoc
Download Mad-Metasploit