Monday 10 July 2017

SAP Hana 2.0 SP1 – Smart Data Streaming & DT Combined with Raspberry Pi Part-3

Build Smart Data Streaming project


The virtual scenario to construct my project is the following:
I have several workspace locations such as, store, Factory, Library … across different country that I would like to monitor the temperature, humidity, air quality and air density.

Among this location, I want to track only my IT Room location based in Mexico where 12 employees are working in this place with many computers.

To do so, I will combine my WORKSPACE table entries, which contain my location information with my sensor data received, filter only for Mexico location and store the output in specific table

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

My project is name “environment_sds”

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Once my project build is completed, I compile it to check for no error, to do so I click on the compile button

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

If no error, click on the “Run Streaming Project” to test it

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Before to run the test let’s have a quick view on the different windows stream to see what my incoming data will look like. I click on each component to look at them

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Now I will simulate the incoming data by choosing “Manual Input” and select the input stream

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

I insert random value for each of my field declared in my input stream and execute

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Once done I can see that 1 row has been recorded for my output adapter

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

My input stream is filled up with the value

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

My join stream has the data populated for all location

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

And my filtered stream with only Mexico location

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Finally, for the last validation, like mention earlier I have a table to store all the captured data name “MEXFL” where my output adapter push the data into, I check if I have my data recorded

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Build Streaming Lite project


The Streaming Lite project to build is very simple; it consists of having the same input stream as the SDS project and the output adapter.

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Streaming Lite support only “Streaming Web Output adapter, here the detail of what should be enter for parameter:

Protocol -> websocket or rest
Destination Cluster Auth. Data -> <user/password>
SWS Server Host -> the SDS Host “not Hana”
SWS Server Port -> the communication port
Dest. Workspace project -> the workspace where the project is created
Dest. Project name -> name of the target project
Dest. Stream name -> name of the stream input in SDS to fill

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Once the Streaming Lite project is done, I check the by compiling it if not error

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

And load it into my Raspberry PI, to do so I transfer the .ccx project file via winscp

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Once transferred, I test my SL project and try to send data from my Raspberry, to do so I need to invoke the streamingproject command by specifying the .ccx location and the port to use

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Before to try to inject data from my Raspberry I need to start the Stream Web Service on Smart Data Streaming node, to start it I go on the Hana cockpit and select the manage streaming node

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Click on SWS and start it

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Back to my Raspberry, I need to start the following string command to load my test data:

echo “IOTLAB,i,,15:04:54 PM,35,26,VERY GOOD,43” | /sap/install/bin/streamingconvert -d “,” -p iotlab:9093/environment_st | /sap/install/bin/streamingupload -p iotlab:9093/environment_st

Thus, we can see my insert successfully proceed and from a process standpoint that my connection has been initiated

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Now let’s check on the Hana side if my MEXFL table has received the information, and I can see all my input based on the insert command

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

The load test successfully done, I will achieve the last part of configuration in my Raspberry which consist of creating a custom python script which will retrieve the sensor data.

Retrieve sensor data and load them into Streaming Lite project


For my IoT project I use a GrovePI platform on top of my raspberry for sensor, thus I don’t have any physical sensor connect to my Raspberry.

GrovePI execute python script to use the sensor and retrieve the data, so how to load those data into Streaming Lite??

Once Streaming Lite is deployed on Raspberry, it basically uses XML-RPC internally to communicate which give several options:
  • Create a custom java adapter from my hana studio and load it into my Raspberry
  • Communicate directly with the XML-RCP internal server to load my sensor input
  • Create a custom python script which encapsulate input data and execution command

I have decided to use the third option, from a sequence stand point the first thing to do is to create a special command line at os level.
In the /usr/bin I create the command called “stream” and include the following line

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Now done I test my command with the syntax and check the result in Hana

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

So, my command is working, I will now create a test script to see if the proceed work to in python. I will voluntary put some data for input

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Execute my script and check the data in Hana again

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

It works, after this validation I can now create the real python script which will be use to capture the sensor input and send then into Hana.

Check live data streamed and received in Hana


Below, here is my final script to be used to ship the sensor data captured from GrovePI into Hana

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

In my above script the line 25, P1 capture the sensor data and ended by a pipe command ( | ) followed by P2, line 26,  which convert and upload the data to my smart data streaming server.

In order to control the cadence of data flowing to Hana, I have decided to only read every 10 second the captured data, line 28 time.sleep(10).

And finally, to be able to compare the data read by the script vs what is incoming in Hana I have declared a print, line 27, so I can see in realtime if the data matches.

Let’s run the script now and compare

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

As we can see the data are read and stored to Hana in my table “MEXFL” properly, I can follow up with Dynamic Tiering part.

Store the incoming data in Dynamic Tiering


Earlier in my documentation I have installed and configured dynamic tiering, the purpose of this is to avoid having a huge amount of data loaded in memory.

To store my MEXFL table in DT I will basically right click on it and mark as “extended”
Note : I have let me sensor for a moment in order to capture enough data before to store my table in the extended storage.

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Moving table in extend storage is a very simple move, however you will need to take some consideration such as the volume of data to move and the disk write speed performance.

For my lab I’m using SAS storage with 10k speed and also I’m controlling the frequency of the data sending (every 10 sec), for a real life case scenario consider to read the following document for guidance and best practice recommendation

I am done with the DT part now, so I will go to the next part of my documentation with the creation of my custom application.

Build a custom application for my project


To expose and visualize my data I will create a basic XS application, SAP HANA XS can be used to expose data from Tables, Views and Modeling Views to UI layer.
There are major 2 ways to expose data the data to clients – OData and XSJS, in my case I will use the OData option.

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

I will briefly explain my step but won’t go deep since it’s not the main goal of this documentation, how do I proceed? here are the step

I start to build the logic (calculation views), for my setup I have created one calc view per application which will be used for my xsodata services

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Once my calc view are all set, I create my xsapp and 3 folders such as:

“apps” – where I store my xswidget that will contain my dynamic application
“catalog” – where I store my xsappsite, which contain the catalog for application
“odata” – where i store my xsodata service used for each individual dynamic application

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

Once my build completed this is what my application look like

SAP Hana 2.0, SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Smart Data Streaming

My setup is completed for this part, in my next blog will explain how to configure HSR for this particular setup with Dynamic Tiering and Smart Data Streaming.

No comments:

Post a Comment