File structures, notations, wizards, response analysis, configuring HTTP requests, target servers, aggregation rules, logs and many more.

REST Commander is a parallel async http client as a service. It is scalable, fast, and able to conduct generic response aggregation and variable based replacement in requests. Commander can run commands via step by step wizard UI or powerful REST APIs Commander leverages the following libraries. If you are new to HTTP or RESTful/SOAP web service, here is a beginner's guide.

Library Description
Play Framework 1.2.4 A Java MVC Framework with high performance netty based web server.
AKKA (in Java) Actor Model Framework for simplifying thread pool management. Here we use it for sending parallel HTTP requests, aggregate and extract responses.
Async HTTP Client (NING) Asynchronus HTTP Client.
Bootstrap 2.3 Opensource CSS and frontend library.

1.1 Notations

Important!: Please read this notation table before proceeding. Note that you can edit these configs both offline and online (while Commander is running).

Click this buttons in the middle of the "Home" page of Commander to start editing.

Notation Table

Notation Description
Command A http request: have which port, URL, http header, if a post, what is the post content. A command can have the concrete payload or just a template with some parts as a variable. Then a replacement of the variable is possible when we need to send different actually http requests using the same "command template" when replacing the variable with different values. Commands are defined in a very simple format in agentcommand.conf
Node / Node Group A node is normally a target server. Commander normally sends requests to multiple nodes, which is a node group. When sending to the same server, the node becomes a string that replaces the defined variable in command. Pre-define Nodes are defined in a very simple format in nodegroup.conf
Aggregation or Aggregation Rule Normally means the regular expression. Why need this? When you get 1000 responses back, each response may be huge. But you are just interested in a specific value. Since each response is just a string, we use regular expression to generically extract the needed values out. Aggregations are defined in a very simple format in aggregation.conf. Just an id and then the regular expression.
HTTP Header Type Http header type is a unique key string associated with a set of key, value pairs of http header values. Each command is associated with some HTTP header type. It is defined in httpheader.conf

Commander source code are under the AgentMaster folder.

├── AgentMaster
│   ├── adhoc_components
│   ├── app
│   ├── app_logs
│   ├── app_logs_adhoc
│   ├── app_logs_none_standard
│   ├── conf
│   ├── dev_notes
│   ├── lib
│   ├── public
│   └── test
├── clean_application_pid_after_run_win.bat
├── eclipsify_application_win.bat
├── play-1.2.4  
└── start_application_win.bat

In Configuration directory(/conf), the folder content looks as below. As Commander does not use database, the key configuration files are stored in this folder. All these configs can be changed offline or online with the edit config buttons in the Commander front page.

├── conf
│   ├── actorconfig.conf
│   ├── agentcommand.conf
│   ├── aggregation.conf
│   ├── application.conf
│   ├── dependencies.yml
│   ├── httpheader.conf
│   ├──
│   ├── messages
│   ├── nodegroup.conf
│   ├── notes.conf
│   ├── routes
│   └── wisbvar.conf

Important!: Read this key config table about these key configuration files before moving on.

Table of the Key Config Files.

Config File Name Description
agentcommand.conf The list of pre-defined commands are defined here. Each command (HTTP request) will have a http header. Some variable can be defined in this conf too.
nodegroup.conf The list of pre-defined node groups (target http servers) are defined here. The format is pretty straight forward
aggregation.conf The list of pre-defined regular expressions for extracting strings. When mutliple repsonses come back, each response is basically a large string. We want to use the regular expression to extract the specific part that we are interested in.
httpheader.conf The list of pre-defined http header types are defined here. Each http header type is with a set of htttp headers. Each command must associate with an HTTP header type.
application.conf The key config in a play framework project. Also includes the parameters specific to REST Commander, such as the timeout and default max concurrency settings. Details please check here.
routes The dispatcher in the MVC framework by Play. Which API to hit which controller.
actorconfig.conf The config of AKKA actors settings.

The Commands Page Sample:

3.1 Command Structure

We always have to use exactly two lines to define a new command. The 1st line format: commandID httpmethod requestProtocol requestPort requestUrlPostfix maxConcurrency responseExtractIndexStart responseExtractIndexEnd

Parameter Name Description
Command ID The ID that uniquely identify this command in the system. Must be unique: use it as the key to save in the hashmap in memory.
Http Protocol http or https
Connection Port which port? normally it is 80 for comment. Internally can be any port, such as 8080, 9000, dependent on your target server.
URL Postfix the url part after hostname:port. e.g. : , the URL-Suffix is "/index.html". There is a special UrlPostfix of "$NULL_URL", where you just want to hit http://node:port (like the front page) .
Max Concurrency It indicates what is the maximum concurrent requests the Commander will perform. i.e., If this value is 100, that means Commander will not send the "n_th" request until it gets the "{n-100}_th" response back. This helps to protect the target server when Commander is sending to the same target server; and also protects the machine running Commander from too much IO and socket load. If this value is 0 or negative: will use the default value of 1500 (defined as MAX_CONCURRENT_SEND_SIZE in application.conf
Extract From When response is too large, may just extract some part of it. e.g. the first 2KB [from:0, to:2000). When extract from and to are both -1: will extract the whole response. For example the GET_FRONT_PAGE command, just save back the first 5000 Bytes from each response
Extract To When extract from and to are both -1: will extract the whole response.
HTTP Header Type The http header type (id) for this command. Will load the set of http headers defined in the httpheader.conf. Refer to the http header part.

Attention!: The maxConcurrency is a key parameter in the command. It indicates what is the maximum concurrent requests the Commander will perform. i.e., If this value is 100, that means Commander will not send the "n_th" request until it gets the "{n-100}_th" response back. e.g. The 500th request will not be sent until receiving the reponse from the 400th request. This helps to protect the target server when Commander is sending to the same target server; and also protects the machine running Commander from too much IO and socket load. If this value is 0 or negative: will use the default value of 1500 (defined as MAX_CONCURRENT_SEND_SIZE in application.conf

The 2nd line is the request body (data) when it is a POST or PUT. When you dont need the request body just put "$AM_FULL_CONTENT" (like a place holder). the ``` in the beginning of this line is just for formatting purpose; no real meaning.

Two sample commands are defined here. First command is a GET to /act/index.rss. The 2nd command is a POST call to a SOAP based Web service (a public WSDL on weather). Note that $APIVARREPLACE_ZIP_VAR is the variable to be replaced. To protect the public weather service server, the maxConcurrency is set as 20. Please be careful of setting this value and protect public service or you may be banned by the server.

Warning! The variables can be in either the 1st or tne 2nd line of the command (both url postfix and the POST body). All variables must follow this convention: Starts with the prefix of "$APIVARREPLACE_" . There is a special UrlPostfix of "$NULL_URL", where you just want to hit http://node:port .
GET_CRAIGSLIST_RSS GET http 80 /act/index.rss 0 0 5000 SUPERMAN_GLOBAL

%Call public WSDL web service; POST a SOAP (XML) object where the variable is the ZIP code
```<soapenv:Envelope xmlns:soapenv="" xmlns:weat=""><soapenv:Header/><soapenv:Body><weat:GetCityWeatherByZIP><weat:ZIP>$APIVARREPLACE_ZIP_VAR</weat:ZIP></weat:GetCityWeatherByZIP></soapenv:Body></soapenv:Envelope>

Attention!: There are two types of variable replacement in commands. In form of $VAR and ${VAR}. $VAR is more important and a must to send different requests for different servers. While ${VAR} is optional to reduce multiple definition.

Table of Comparing $VAR vs ${VAR}

Replacement Form When Motivation Function Location to Define the Replacement String to the Var
$VAR, with prefix: "$APIVARREPLACE_" ${VAR} will be replaced in the last minute before sending out the request for enabling sending different requests to different servers. This is a key feature! Used for run time replacement, a must for enabling sending different requests to different servers. used for run time replacement: support both uniform replacement or server specific replacement. (different values for different nodes) (1) in the REST API: replacement map; (2) in the single server wizard; (3) in the code or rest client to Commander.
${VAR} ${VAR} will be replaced at the time when this agentcommand.conf is loaded (normally during startup of application or a reload trigger) This is optional. Goal is to make the definition in the agentcommand.conf looks nicer to reduce potential multiple same changes (like to remove magic numbers in the code) . Say when a value e.g. some software version appear in multiple commands, use this ${var} to define it in a single place. uniform replacement only in agentcommand.conf

The Section in agentcommand.conf for defining ${VAR}: optional

%VARIABLES here are with ${VAR} will be first replaced here globally inside of this .conf file.

3.2 Edit/Add Http Headers

You may go to http://localhost:9000/config/editConfig/httpheader to edit the http header or just edit the httpheader.conf. In the example, the id is HEADER_PUBLIC_WEATHER, then the "```HTTP_HEADER_LIST_START" and "```HTTP_HEADER_LIST_END" tags; between these two are a list of http header key value pairs. Each key value pair uses two lines. And the 2nd line (which is the value) use the formatting string "```" (no meaning).


The Node Page Sample: You can click the ID to get the json view and click on the button to get the domain name (FQDN) text line by line.

4.1 Sample Pre-defined Node Group

In the example, the type is ADHOC. The id is ADHOC_NODE_LIST_CRAIGLIST, then the "```ADHOC_NODE_LIST_START" and "```ADHOC_NODE_LIST_END" tags; betwen these two are a list of nodes. (Advanced feature for extension: you may add new data source type and dynamic loading nodes from a centralized databases or an API.)


5.1 Aggregation Sample

In the example, two aggregation rules are defined. Each rule needs two lines. The first line is the ID, and the second line is the regular expression starts with the formatting string "```" (no meaning). The () indicates the part of string to be extracted. E.g. In the first rule is the part between the temperature.

Tip: The () indicates the part of string to be extracted.



You can check logs on the pre-defined node group is via http://commanderhost/logs; while for the ad hoc node group is via url http://commanderhost/logs/adhocLog. Here is a screenshot of the adhoc log page.

The Icon buttons have the following functions.

Icon Description
Download the raw json logs of the hashmap of the request and response pairs. (not using regular expression aggregation rules).
Show the aggregation tables: which nodes have which values. You may (1) easily change the aggregation rule; (2) click on the column of "Node List" to obtain the corresponding list of nodes that share the extracted string.
The text / json version of the aggregation after using the aggregation rule (regular expression) Default regular expression is :SUPERMAN_SPECIAL_STATUS_CODE. Note that this is URL addressable. So you can change the URL to change the regular expression.

1. Click give you the raw data.

2. Click give you the table view after the regular expression aggregation

Tip: You may change the aggregation rule (regular expression) by clicking on the "Change aggregation rule...". Like here we want to use the "Temperature" regular expression to aggregate:

Tip:You man click on the "Open" in the "Node List" Column in the table to get the nodes have this value. Note that this is URL addressable too. e.g.: http://localhost:9000/agents/aggregateMetaDataNodeList?nodeGroupType=ADHOCDATASTORE_NODEGROUP_20140115162653056-0800&agentCommandType=GET_PUBLIC_WEATHER_APIVARREPLACE&value=-3&timeStamp=20140115162653406-0800&rawDataSourceType=LOG_FILE&aggrRule=PATTERN_WEATHER_WSDL_TEMPERATURE

Tip: Add Back Pie Chart Visualization: We originally have a pie chart view besides the table view with the Highcharts library. We removed it due to potential license issues. If you use sueprman for non-commercial purpose, you may want to restore the pie chart visualization. The data for feeding the pie chart has been passed to the frontend. Please check detailed instruction on the source code at /views/Agents/aggregatePieChart.html

3. Click : for get a text view after the regular expression aggregation

Here are some sample log names and how to access them.

// Logs for predefined node group
Access: http://localhost:9000/logs/logContent/20140114134629472-0800~ADHOC_NODE_LIST_CRAIGLIST~GET_CRAIGSLIST_RSS.jsonlog.txt

// Log for ad hoc node group

The key part inside of Commander to run jobs on predefined node group is as follows:

String agentCommandType = VarUtils.AGENT_CMD_KEY_GET_FRONT_PAGE;
Setting Description URL
Explore Files Like a simple online file browser. Discover the archived log files. http://localhost:9000/logs/exploreFiles
System Health Commander run a system check on JVM memory, threads, and disk space every 5 minutes. You can check the result in the right bottom part in Home page or the url /monitor http://localhost:9000/monitor ; http://localhost:9000/monitor/threads ; http://localhost:9000/monitor/disk
Config Page Many config options including editing the .conf file online; clear all data in memory (all the request/response hashmaps), http://localhost:9000/config Many more parameters are set in this file under models.utils N/A