Universal Storage Collector

First, I have created a VNX Collector. Then I have created a VPLEX Collector. This systems are very similar. My next task was periodically extracting performance data from NAR files. At that moment I decided to create an universal tool to collect performance data from different storage systems with output to different collectors. Later I decide to add gathering performance data from EMC VMAX.

And now after some month of work I present Universal Storage Collector — a modular and flexible tool.

What is it

Modular tool. This mean that it is easy to add an extractor for new types of storage systems, or add output to new data collector. This is modularity by source code. It does not use binary plugin at this time. But it is easy to add new code.

Flexible tool. This mean that we could work with different storage systems. Each storage system has its own extractor, output to its own collector, and has its own interval for gathering.

Extractors. At this time this tool could extract performance data from:

• EMC CLARiiON/VNX block using naviseccli
• EMC Celerra/ VNX file
• EMC CLARiiON/VNX block using NAR files
• EMC VPLEX
• EMC VMAX from Unisphere reports

Output. At this time this tool could output to Carbon (Graphite) and InfluxDB.

How it works

This tool has one configuration file. At this file we specify:

• List of extractors with its common parameters.
• List of outputs with its parameters.
• List of storage systems we would work with.

For each storage system we should specify:

• name
• class — name of storage system type (VNX, VMAX, VPLEX and so on)
• type — type (block or file) for the unified storage systems (optional)
• interval — time interval between polling
• extractor — one from the list with concrete parameters for this storage system
• output — one from the list

Combinations of name, class and type are unique. We could have (array1, vnx, block) and (array1, vnx, file), but we could not have two (array1, vnx, block)  with different extractors. Only one will survive. The first one from the list.

After start, this tool:

• check each extractor definition
• check each output definition
• check each storage system definition and create concrete extractor and output exemplar for each of them
• create the list of actors, each actor is a concrete storage system
• ‘ask’ each actor in its interval

When each actor receive ‘ask’ signal, its begin transmission to output, send ‘ask’ message to its extractor, and then stop transmission.

When each concrete extractor receive ‘ask’ signal it extract data from concrete storage system and send its directly to concrete output.

Setup

To start this tool, we should:

• create a home directory for it
• create a conf subdirectory and put configuration file collector.xml into it
• create a log subdirectory for error log file
• create a pool subdirectory, if you plan to use extractor from VNX NAR files
• specify USC_HOME environment variable with home directory

The source code of this tool is available here https://github.com/vzaigrin/UniversalStorageCollector