Monitoring is a process of checking if the observed behavior of a component conforms to a behavioral specification. Behavioral specifications can be interface models and component models. In case of interface model the process is interface monitoring: a check if the behavior of a component that provides the interface conforms to the interface definition. In case of component model the process is component monitoring: a check if the behavior of a component conforms to the functional, time and data constraints in the model.
To perform monitoring with the CommaSuite framework, the following ingredients are required:
-
Behavioral model (.interface or .component file). The model in turn imports interface and signature definitions.
-
One or more traces with events (calls to commands and signals, notifications and replies to commands) observed during a component execution. Traces can be provided in the following ways:
-
as files with extension .events or files in JSON format (extension .jsonl). It is not recommended to create such files manually. They are usually obtained via network sniffing or from execution logs. Trace file names have to start with a letter and have to contain only letters, digits and underscore ("_")
-
as a stream of events transmitted over a socket. The format of the events must be JSON. This allows runtime monitoring during system run.
-
Monitoring Tasks
In order to perform monitoring, one or more tasks for monitor generation have to be included in the CommaSuite project file.
A monitoring task has to specify how the traces are obtained. Trace files can be listed, a folder with trace files can be given, or they can be produced by another generation task. Furthermore, the task has to indicate the behavioral model (interface or component). In the example below a single file with traces is used (TraceITest.events) for interface monitoring:
Generate Monitors { iTestMonitor for interface ITest { trace files "TraceITest.events" } }
The following example shows a task that uses a directory (named trace_files) with traces for component monitoring:
Generate Monitors { monitorTask for component Imaging { trace directories "trace_files" } }
Trace directories have to contain trace files with unique names. If two directories contain distinct files with the same name an error will be indicated.
If the trace files are produced by another task then refer to the task:
Generate Monitors { iTestMonitor for interface ITest { trace files task traceProducer } }
When the events are supplied via a socket, the monitoring task generates a monitor program that will listen for connections from a client that supplies the events. The events have to be transmitted one by one and have to be in JSON format (each event on a single line). The monitoring task lists the port at which the events will be supplied and a feedback port.
Generate Monitors { iTestMonitor for interface ITest { trace port 6666 feedback port 6667 } }
The monitor will transmit the results from the monitoring via the feedback port. The results are in JSON format and have the same structure as the results serialized to a file. Multiple results can be transmitted during a monitoring session, for example, violation of a time rule, followed by violation of a functional constraint and so on. When the monitoring is finished, 'stop' is sent.
Users can choose suitable trace and feedback ports. Connecting to the feedback port is optional. Such a connection can be done even after the monitoring is started. Furthermore, the connection to the feedback port can be closed and reestablished multiple times during the monitoring.
In case of component monitoring using sockets, the task has to provide a file that contains information about the component instances in the trace:
Generate Monitors { monitorTask for component Imaging { trace port 6666 feedback port 6667 instances "components.jsonl" } }
During monitoring all time and data constraints defined in the interface or component model are checked over the traces. In some cases users may need to skip the check of time and data constraints due to various reasons, for example, when the real hardware is not present yet thus making the execution environment different than the expected one. Constraint checks can be skipped by including one of the instructions skip time constraints, skip data constraints, skip constraints. The following example monitoring task skips both time and data constraints:
Generate Monitors { iTestMonitor for interface ITest { skip constraints trace files task traceProducer } }
There are certain situations when the order of the events in the trace files is different than the order they are observed at the client or the server side. This usually occurs when network sniffing is used to collect the traces. It may happen that notifications are observed between a call and its reply whereas at the client side, the notifications are expected after the reply. In these situations, an instruction for reordering of trace events can be given in the monitoring task:
Generate Monitors { iTestMonitor for interface ITest { reorder trace files task traceProducer } }
For a given pair of client and server, the 'reorder' instruction moves all notifications observed between a command and its reply after the reply and modifies the notifications' timestamps to be the same as the one of the reply.
In case of component monitoring, interface monitoring is performed as well for the interfaces on each component port. Every event used in a component instance is first checked by the corresponding interface monitor and after that passed to the component monitor. The component monitor checks if functional, data and time constraints are satisfied.
Generation and Execution of Monitoring Code
After a right mouse button click on the project file, select 'Run As'. Select: "Generate and execute monitoring". As a result, all generator tasks will be executed followed by execution of all the monitoring tasks. Results are located in the folder "comma-gen" (explained further on this page).
Generation of Monitoring Code and Namespaces
When monitoring code is generated for interfaces and components all the used constructs like types, signatures, interfaces, etc. have to be accessible from the project in which the monitoring is performed. Some of these constructs can be located in files contained in multiple Eclipse projects.
When explicit file imports are used, the transitive import mechanism ensures that all these elements are accessible and can be loaded. This may not be the case when namespace imports are used. If the models used for monitoring use namespace imports, the Eclipse project has to refer to all the projects that contain the models contributing to the used namespaces. If these dependencies are not properly set, the generated code may miss some classes. This will be manifested by a compilation error.
When monitoring based on namespace imports is performed from the CommaSuite command line tool, then the argument modelpath has to be used.
Inspecting the Results of Monitoring
Monitoring results unite the results from all traces in all monitoring tasks in the given project file.
For a given trace:
-
in case of interface monitoring, all connections in the trace over the given interface are monitored
-
in case of component monitoring, all instances of the given component model are monitored. Component monitoring process checks if all functional constraints hold for a set of traces. The process of checking a functional constraint is done in parallel with interface monitoring, that is, a check if the given trace satisfies all the interfaces at the component ports. A port associated to multiton interface is monitored by creating a monitor for each connection. A port of singleton interface uses a single monitor for all the connections
-
if trace error occurs (this is a syntax error in the events file) all monitoring stops and the results obtained so far are reported along with the trace error
For interface monitoring, we may observe:
-
interface error: a violation of the behavior given in interface state machines. Monitoring of the corresponding connection stops
-
interface warning: a violation of interface time/data/generic constraints. Monitoring of the connection continues
For component monitoring, we may observe:
-
component error: a violation of a functional component constraint. Monitoring of this constraint stops
-
component warning: a violation of a component time/data/generic constraint. Monitoring of the component instance continues
-
port error: interface error in a connection to a component port. Monitoring of functional and time/data constraints for the corresponding instance stops. Interface monitoring of the remaining port connections continues
-
port warning: interface warning in a connection to a component port. Monitoring of the component instance continues
In order to inspect the monitoring results:
-
Navigate to folder comma-gen. For every monitoring task, a folder with the name of the task is created. A summary of the results from all tasks is stored in JSON format in file tasksResults.json. Furthermore, a web page dashboard.html is automatically created. It contains an overview of the results per trace. If you want more detailed information about the results continue to step 2. Every task results folder has the following structure:
-
One folder for each trace file, named after the file name
-
Folder statistics that contains information about the checking of the time and data constraints (if any)
-
-
Inspect the results for a particular trace file by opening the corresponding folder. Results are structured differently for interface and component monitoring.
-
Interface monitoring. For every connection (usage of an interface by a unique pair of client and server) there will be a folder with name $ServerName$_$ClientName$_$Interface$. The folder has at least two files: $ServerName$_$ClientName$_$Interface$_trace.txt that contains the processed trace and $ServerName$_$ClientName$_$Interface$_results.txt that contains a summary of the monitoring results for the connection. If errors and warnings are detected they will result in additional files.
In case of detected issues, they are described by giving the context and their exact nature. Errors and warnings are also described in separate files that can be visualized in the PlantUML view.Visualize the content of .plantuml files (if any) in the PlantUML view to check the description of the error/warning. The issue is reported in a UML sequence diagram that gives the path to the error and a description.
-
Component monitoring. For every component instance in the given trace file there will be a folder named after the component instance. Each such folder further contains one folder per component port (named after the port), one folder per functional constraint (named after the constraint), and one folder named time_data that contains the results of checking the time and data constraints in the component model (if any). The folders derived from the ports contain the results of the interface monitoring on the given port. The results in all the folders are structured similarly to the case of interface monitoring: one text file with a summary of the results, one text file with the processed trace (up to the occurrence of an error if any), and possibly many files with error/warning descriptions.
-
Interface Monitoring and Non-Determinism
Interface specifications support non-determinism by allowing multiple ways to react to a given trigger (check the interface language and the "OR" clause in particular). The presence of non-determinism affects how monitoring is performed and how monitoring results are reported.
In general, when a non-determinism is detected during monitoring, the monitor will maintain and explore all execution paths. As more events from the trace are checked by the monitor, some execution paths may be identified as erroneous and therefore discarded. If all paths at a given moment do not allow the occurrence of an event then a monitoring error is reported and contextual information for all the paths will be shown.
We give an example of monitoring error in case of non-determinism and explain the structure of the reported results.
Imagine the interface of a simple vending machine that accepts coins and delivers products on user request. When a coin is inserted, the machine checks first if this is a valid coin and then reports the outcome to the user together with the value of possibly updated credit. If the machine is in error state then the coin is simply returned. The code snippet shows only the transition when a coin is inserted:
initial state NoCredit { transition trigger: InsertCoin(int val) do: credit := credit + 1 reply(credit,CoinResult::ACCEPTED) next state: CreditAvailable OR do: reply(credit,CoinResult::NOT_ACCEPTED) next state: NoCredit OR do: reply(0,CoinResult::NOT_OPERATIONAL) next state: NoCredit .... }
After inserting the coin, the transition gives three possible paths. This is an example of non-determinism. By reading the observed reply in the trace, the monitoring algorithm can determine which path to take or possibly an error is detected. Consider the following trace:
Command InsertCoin Reply to command InsertCoin. Parameters: -1 CoinResult::NOT_OPERATIONAL
The trace will cause an error because the reply parameters do not match any of the expected values given in the three possible transition bodies. All the three possible executions at the moment of observing the reply will be reported in the monitoring results.
The text file with the results will contain the following:
Interface monitoring error The event does not conform to the model Possible events in the current state of the model: Reply to command InsertCoin. Parameters: 1 CoinResult::ACCEPTED Reply to command InsertCoin. Parameters: 0 CoinResult::NOT_ACCEPTED Reply to command InsertCoin. Parameters: 0 CoinResult::NOT_OPERATIONAL Execution states of the model: Active state: NoCredit Values of global variables and current machine states: credit = 0 Machine userMachine in state NoCredit Active state: NoCredit Values of global variables and current machine states: credit = 0 Machine userMachine in state NoCredit Active state: NoCredit Values of global variables and current machine states: credit = 0 Machine userMachine in state NoCredit
As can be seen there is a description for each possible execution. It contains: the state in which the event is observed (active state), the values of interface variables, the current states of the interface state machines (recall that multiple machines are possible). In our example, it happens that all executions have the same information.
The visualization of the error as a sequence diagram will contain similar information:
The three sections in the note indicate the three possible executions at the moment of observing the reply. In this example they are identical.
Singleton and Multiton Interface Monitoring
The singleton/multiton distinction of an interface determines how the interface is monitored. Generally, a trace file contains information about servers that provide interfaces and clients that use them. A single server may have multiple clients of a given interface. A triplet of a client that uses an interface provided by a server is called connection.
A multiton interface is monitored by creating a separate monitor for every connection over that interface.
A singleton interface is monitored by creating a single monitor for all the connections that have the same server at their end. This reflects the fact that the clients of a singleton interface share one instance. In other words, the monitor of a singleton interface does not distinguish its clients, it treats them as a single client.