External Metric Configuration

Overview #

edgeCore instrumentation is available via Prometheus, an open source time-series monitoring tool. edgeCore uses Micrometer to record metrics that are then scraped by Prometheus based on polling intervals. Prometheus stores metrics in a time-series database with dimensions. All edgeCore metrics include the tags: ‘application’: ‘edgecore’ and ‘cluster’: ‘standalone’. The cluster tag value should be renamed when the metric is scraped during a polling interval to properly identify the server or the environment.  Additionally, tags are added to the Prometheus scrape configuration (also known as ‘labels’), which include job_name and possibly group name to be used to identify the system (prod, qa, or dev).

Metrics are split into three categories: Gauges, Counters, and Timers. Micrometer allows other Monitoring Systems to be dropped in as well.

Telemetry for external metrics can be viewed here.

Available Metrics #

Micrometer supports three different types of metric categories: Gauge, Counters, and Timers. Initial metrics center around logins, page access, and job information, plus a number of default bindings used to track Java and system performance.

edgeCore Application Metrics: #

  • Active Sessions Gauge (‘active_sessions’) tracking current users logged in (all sessions on the cluster).
  • Login Counts (‘sessions_login’) tracking count of logins with tags: ‘domain’, ‘user’, and ‘useragent’.
  • Page Access Counts (‘pageaccess’) tracking pages accessed via the browser with tags: ‘domain’, ‘user’, ‘path’, ‘pagerRef’.
  • Job Records Size Gauge (‘job_data_size_results’) with tags: jobType, producer, and params.
  • Job Memory Size Gauge (‘job_data_size_memory’) with tags: jobType, producer, and params.
  • Job Timer (‘job_timer’) with tags: jobType, producer, and params.

Micrometer has default bindings for the following metrics used to instrument Java Applications: #

  • JVM metrics report utilization of:

    • Various memory and buffer pools
    • Garbage collection statistics
    • Threads utilization
    • Number of classes loaded/unloaded
  • CPU metrics
  • File descriptor metrics
  • Log4j2 metrics: record the number of events logged to Log4j2 at each level
  • Uptime metrics: report a gauge for uptime and a fixed gauge representing the application’s absolute start time
  • Tomcat metrics
  • Spring integration metrics

Endpoint Security #

edgeCore expects access to the additional management endpoints from specific hosts. By default, the IP Address Filter is limited to ‘127.0.0.1’ localhost access. This can be adjusted by adding the following line to [Install_Home]/conf/custom.properties:

security.management.hasIpAddress=127.0.0.1/16

This setting should be the IP Address of the Prometheus server.

Prometheus

Architecture #

The illustration below represents the primary interactions between edgeCore and Prometheus, which will be detailed in the following sections.

Installation and Configuring Prometheus #

Setting up Prometheus is possible inside and outside of a docker container. Setting up Prometheus in both of these instances is a fairly simple process and requires a YAML File, which is a configuration file for Prometheus, and also a dockerfile if run inside a container.

Docker Installation #

Prometheus can also run inside a docker container together with an edgeCore docker container in the same network and on the same server. Setting up Prometheus inside a docker will require a dockerfile and YAML format configuration file, as there is already a preloaded Prometheus image for docker. The docker file should look like the following:

This will deploy an instance of the Prometheus image inside a docker container and also add the YAML configuration file. You will need a docker file for setting up the edgeCore instance inside of docker, which should look like the following:

You will need an edgeCore build which will run inside the docker container. Once the docker files are created, it will require that you build docker images which are done by navigating to the folder where your dockerfile and YAML File are located and running:

  • docker build -t prometheus .

Similar to building a docker image, navigate to the edgeSuite docker folder where your docker file and edgeSuite build is located and run:

  • docker build -t edgeCore .

These commands will build a local image for both docker containers on your machine.

YML Configuration File #

Prometheus will require a YAML File, which represents a configuration file where you will set up a location, interval, and job rules for scraping from the specific location. The following is an example of a YAML File:

This will scrape all metrics at the location 127.0.0.1/sysinfo/Prometheus (local instance).

Configuration #

Once Prometheus and edgeCore (or their docker containers) are started up, the first thing should be to verify if Prometheus is successfully scraping the required metric data:

Prometheus is by default set to localhost:9090. On the “Targets” page in Prometheus, verify if the edgeCore job has State UP which means it is successfully scraping from the location you specified in the .yml file.

Targets monitoring multiple edgeCore Instances #

It is also possible to setup Prometheus to scrape from multiple edgeCore instances by configuring the YAML File. This is achieved by adding additional targets inside a configuration file:

If you are running Prometheus inside containers, any additional edgeCore container will have to be set into the same network in order to properly scrape from it.

Running Prometheus #

If Prometheus is installed outside of docker, it will require you to download it from the following URL:

https://prometheus.io/download/

After unpacking Prometheus and setting up the YAML file you can start the instance by running the following command:

  • ./prometheus --config.file=prometheus.yml

 If you are running Prometheus through a docker container, after setting up the dockerfile and YAML file you can start the docker instance locally by running the following commands:

  • docker run -p 9090:9090 prometheus – Prometheus docker
  • docker run -p 8080:8080 edgeSuite – edgeCore docker

edgeCore Prometheus Connection and Feeds #

Once Prometheus is connected and successfully scraping data from edgeCore, the next step is to set up a Prometheus connection in edgeCore. That can be done manually, or by using a Prometheus adapter that contains a content bundle with preset Feeds and Transform examples that will help you familiarize yourself with it.

The following sections detail the steps for the manual setup. The final section will then detail the steps for installing the adapter with the sample bundle.

Connection #

First set up a web data connection inside the edgeSuite pipeline. The following is an example configuration:

edgeCore uses the “Generic Web Data” connection type to connect to Prometheus and there is no need for a SSO handler.

Custom Javascript Feed #

After the connection is set up, the next step is to set up the required data feed. Connecting Prometheus data is done via a Custom feed using JavaScript that converts the required attributes and data. The following is an example feed configuration:

JavaScript example:

function getAttributes(inputString, nodeVars, secVars) {

            // TODO: Create an array of attribute objects following the sample below

            // use inputString to derive attributes

            // Note: inputString can be anything: JSON, CSV, XML, etc.

            // The method of parsing the data is up to the user

 

            var attributes = [];

            var Application = {

                        name : ‘Application’, // Alphanumerics and underscores only; no leading, trailing, or adjacent underscores

                        type : ‘string’, // ‘string’|’int’|’long’|’number’|’boolean’|’date’

                        isId : false, // true|false

                        format : null // string consisting of valid Java SimpleDateFormat pattern

            };

                        var Instance = {

                        name : ‘Instance’, // Alphanumerics and underscores only; no leading, trailing, or adjacent underscores

                        type : ‘string’, // ‘string’|’int’|’long’|’number’|’boolean’|’date’

                        isId : false, // true|false

                        units : ‘millis’, // ‘millis’|’seconds’ (if attribute is timestamp)

                        format : null // string consisting of valid Java SimpleDateFormat pattern

            };

                        var Cluster = {

                        name : ‘Cluster’, // Alphanumerics and underscores only; no leading, trailing, or adjacent underscores

                        type : ‘string’, // ‘string’|’int’|’long’|’number’|’boolean’|’date’

                        isId : false, // true|false

                        units : ‘millis’, // ‘millis’|’seconds’ (if attribute is timestamp)

                        format : null // string consisting of valid Java SimpleDateFormat pattern

            };

            var Time = {

                        name : ‘Time’, // Alphanumerics and underscores only; no leading, trailing, or adjacent underscores

                        type : ‘number’, // ‘string’|’int’|’long’|’number’|’boolean’|’date’

                        isId : false, // true|false

                        format : null // string consisting of valid Java SimpleDateFormat pattern

            };

            var Value = {

                        name : ‘Value’, // Alphanumerics and underscores only; no leading, trailing, or adjacent underscores

                        type : ‘number’, // ‘string’|’int’|’long’|’number’|’boolean’|’date’

                        isId : false, // true|false

                        format : null // string consisting of valid Java SimpleDateFormat pattern

    };

                        attributes.push(Application);

                        attributes.push(Instance);

             attributes.push(Cluster);

                        attributes.push(Time);

                        attributes.push(Value);

 

            return jsAttributesSuccess(attributes, “Example Success”);

            // can optionally return jsAttributesFailure(failureMsg) to handle failures

}

 

function getRecords(inputString, nodeVars, secVars) {

            // TODO: Create an array of record objects

            // use inputString to derive records

            var records = [];

    var i = 0;

    var myJson = JSON.parse(inputString);

    myJson.data.result.forEach( function(metricObj) {

      metricObj.values.forEach( function(metricValues) {

      //for ( var i=0; i< metricObj.values.length; i++ ) {

        i++;

         var record = {

            ‘Application’ : metricObj.metric.application,

            ‘Instance’ : metricObj.metric.instance,

            ‘Cluster’ : metricObj.metric.cluster,          

            ‘Time’ : metricValues[0],

            ‘Value’ : metricValues[1],

         };

         records.push(record);

      });

    });

            return jsRecordsSuccess(records, “Example Success”);

            // can optionally return jsRecordsFailure(failureMsg) to handle failures

}

After the feed is created and metrics are pulled from Prometheus into edgeCore, a time conversion operation must be completed. This is because Prometheus registers the epoch time when scraping data and saving metric values. The following is an SQL transform query to successfully convert the time:

  • select Application,Instance,Cluster, dateadd( second, Time, CAST( '1970-01-01' as datetime ) ) AS Time,Value as Count from {src.ActiveSessions}

After the transform query has successfully completed, the metric data is ready to be ingested for creating further visualizations or transforms depending on your need. 

Prometheus API Calls #

Data from Prometheus is retrieved by running API calls with a query to the Prometheus page.

The format of start URI should be like /api/v1/query?query= and you add the query of the value you would like to retrieve. These are all the values that edgeCore is currently retrieving from Prometheus in the content bundle:

  • /api/v1/query?query=sessions_active[10m] – Number of active sessions
  • /api/v1/query?query=process_cpu_usage{job="edgeSuite"}[5m] – CPU Usage in last 5 minutes
  • /api/v1/query?query=log4j2_events_total – Total count of each type of event in edgeSuite
  • /api/v1/query?query= job_data_size_memory[1m] – Job data size over time
  • /api/v1/query?query=job_timer_seconds_count{job="edgeSuite"} – Job count for edgeSuite
  • /api/v1/query?query=job_timer_seconds_count{job="edgeSuite"}[5m] – Job count changes over time
  • /api/v1/query?query=system_load_average_1m – System load average in last minute
  • /api/v1/query?query=sessions_login_total{job="edgeSuite"}[5m] – Session logins over time
  • /api/v1/query?query=jvm_memory_used_bytes[10m] – Memory used over time
  • /api/v1/query?query=process_files_open_files – Opened files by edgeSuite instance
  • /api/v1/query?query=process_uptime_seconds – Current uptime by edgeSuite instance
  • /api/v1/query?query=pageaccess_total – Page accessed count inside edgeSuite instance
  • /api/v1/query?query=http_server_requests_seconds_count – Proxy Gauge for each edgeSuite instance

Prometheus Adapter #

The steps above show how to manually create connections in edgeCore to display Edge metrics from Prometheus. This section details how to install the Prometheus Adapter which has a bundle of pre-configured Feeds, Transforms, and Visualizations.

Perform the following steps to install the Prometheus Adapter:

  1. Save adapter into [Install_Home]/modules
  2. Go to Manage / Adapters / Available Adapters
  3. Activate adapter
  4. Deploy Content Bundle
  5. In the pipeline, edit the “Prometheus Test” Connection
  6. Update Destination Endpoint to your Prometheus URL
  7. Observe Pages with various metric Visualizations

Fetching Metrics Data via Edge CLI #

Once a connection, feed, and (optionally) a transform have been established in the edgeSuite pipeline as described above, metrics data can be retrieved using the Edge CLI fetch command. The data is returned in a comma-separated value format with the first row acting as a header row. The format for the command is fetch -p PRODUCER [-v VARS...] where PRODUCER is the name of the feed or transform, and VARS (which is optional) is a list of “key=value” pairs to set any node variables that have been defined for the producer (e.g. -v "foo=bar"). Using the above example, the command would look like

./bin/es-cli.sh fetch -p ActiveSessions

To run the command, user credentials need to accompany the arguments, or the edgeSuite server needs to be configured for pre-authenticated CLI access. For more information about fetch and other commands, see Edge CLI.

Quick Start via edgeCore Prometheus Adapter and Content Bundle #

The Prometheus adapter for edgeCore is currently available that utilizes a web data connection and has three available feed types (JSON, Custom Time Base script, Custom Gauge Base Script). Included with this adapter is a content bundle that contains an example Prometheus configuration with all the previously mentioned API calls and transforms together with pre-built visualizations and pages for monitoring scraped metrics from edgeCore.

<Include Directory>

Future Updates: Joining multiple metrics via SQL Transforms #

There are currently plans on further updating the Prometheus example configuration with joining multiple metrics together via SQL Transforms to provide more precise data analytics and expanded functionality.