Application Specific Development of REST Provider Service Based Features

Adding information displays for users in Manage 2000 from REST providers is all well and good, but how can we use the Manage 2000 REST.SERVICES connections to do other more interesting and specific things?

What if the RESTful service lets us POST information back to the provider system?  

What if we want to develop integrations with FEDEX and UPS RESTful service APIs which supply not only tracking services, but Rates, Time in Transit, Dangerous Goods, Global Trade Documents and more?

Back in March I wrote about manually coding REST HTTP Post method calls in UniBasic. But it is not necessary to drop down to this level. REST.MSO and REST.NEWS rely on a new Tools subroutine named  REST.SERVICE to carry out the actual conversation with the REST provider system. 

Application code can declare the parameters for the REST exchange and then delegate the implementation to REST.SERVICE:

     $INCLUDE COM.RS FROM COPY.TOOLS.BP
     $INCLUDE TM_328 FROM TOOLS.LAYOUTS
     ...
     READ TM_328.REC FROM TM, '328*':SERVICE.ID
     ...

     RS.Endpoint               = TM_328.Endpoint
     RS.Protocol               = TM_328.Protocol
     RS.Security_Version       = TM_328.Security_Ver
     RS.HTTP_Version           = TM_328.Http_Version
     RS.Auth_Token             = TM_328.Auth_Token
     RS.Method                 = TM_328.Method
     RS.Content_Type           = TM_328.Content_Type
     RS.Parameter_Names        = TM_328.Parameter_Names
     RS.Parameter_Values       = TM_328.Param_Values
     RS.Parameter_Types        = TM_328.Param_Types
     RS.Body                   = TM_328.Body
     RS.Timeout                = TM_328.Timeout
 *
     CALL REST.SERVICE
     IF RS.Status # "" THEN CALL SCREEN.MSG("RS.Status - ":RS.Status<1,1>:" ] ":RS.Status<1,2>:";VT;#M260618")
     IF RS.Headers # "" THEN CALL SCREEN.MSG("RS.Headers - ":RS.Headers:";VT;#M260619")
     IF RS.Data # "" THEN
        SWAP @AM WITH @VM IN RS.Data        
        RS.Data = CONVERT(FILTER.CHARS, "", RS.Data)
        CALL SCREEN.MSG("RS.Data - ":RS.Data:";VT;#M260620")
     END 
     IF RS.Error # "" THEN  CALL SCREEN.MSG("RS.Error - ":RS.Error:";VT;#M260621")
     RETURN

Parameters may be loaded from the REST.SERVICES table as in this example or created on the fly, or some combination of the two. You might, for instance, want to post a bunch of FORM variables based on a work order or sales order into some RESTful provider endpoint defined in REST.SERVICES, and perhaps receive back some confirmation number or message.

REST services may accept inputs on the querystring or as part of the URL route path when using an HTTP GET method, or the service may accept inputs as form variables or as JSON or XML strings in the HTML Body element when using an HTTP POST method.  The REST.SERVICE subroutine supports all of these approaches to REST parameter passing.

Parameters are placed in the appropriate property in a named common block, then your program executes CALL REST.SERVICE, and then interrogates the result values which are also stored as properties in the common block. If the call is successful then RS.Data will contain the resulting JSON string.  All of the UniBasic Extensions' HTTP plumbing is taken care of by REST.SERVICE.

As always tools subroutine documentation is available from HELP:

     HELP REST.SERVICE

Templated Rendering of REST Provider's JSON Responses in Manage 2000

REST.MSO CustomFormatter subroutines are great for Unibasic programmers who don't mind familiarizing themselves with UDO JSON parsing and building text and HTML displays.

But what can you do without getting into programming?

There is another REST JSON formatter named REST.NEWS. Its purpose is to support writing NEWS.ARTICLES with content from REST providers.  It works similarly to the MCD_DATA.NEWS AR/SO dashboard code behind subroutine in that it lets you layout a template in the news article body with &var& type replacement variables. 

Use the new News Source REST in place of CUSTOM to save yourself some typing.

REST.NEWS will run the REST service specified and then look for expressions in the body like &propertyname& and replace them with the corresponding value from the web service JSON response. It will also check for expressions matching variables sent to the web service like CITY in this example.

So we can hit the same REST provider as our previous MSO and see the results laid out nicely, all without doing any actual programming:

There is an important distinction between MSOs and NEWS.ARTICLES in their scope.

At release 8.1 sp5 MSOs can be configured to be available in any PWS or web function throughout Manage 2000. 

The scope of news articles is a little different:

In terms of Manage 2000 functions, you only see them in NewsReader and the News panels of the 16 or so web functions in the ROIPortals directory. 

However, news articles are NOT limited to viewing in Manage 2000 functions. Manage 2000 news feeds can be viewed from any RSS client. This includes Outlook, Sharepoint, Lotus, the built-in IE newsreader, browser add-ons like Sage for FireFox and SlickRSS for Chrome, standalone readers like RSSOwl, and others.

There is also an interesting reciprocity between the two: MSOs can be used as the source for NEWS.ARTICLES, and since Newsreader is a Manage 2000 function an MSO can wrapper a display of articles in Newsreader.

In this sense the news article scope is actually wider than the MSO scope.  One could even use the auto-start feature of MSOs to fire off a Newsreader display based upon the user entering any particular PWS function or prompt.

Custom Formatter Subroutines for Rendering RESTful Service JSON Results as Text or HTML

 There is an intriguing property in MSO.BUILD on the REST.MSO parameter screen named CustomFormatter:

When this property is blank the rendering of the JSON results from the REST endpoint is handled by a subroutine named FORMAT.JSON which, at release 8.1sp5, you will find in SUB.MFG.RPT.BP.

FORMAT.JSON does a blind format, not knowing anything about the JSON string it is formatting. It just goes through the string displaying property names and property values somewhat like a property inspector in an IDE might do.  You can use the IncludeFilter or the ExcludeFilter to eliminate unwanted properties.  But you do not have any control over the text or html rendering.

By writing your own custom formatter subroutine you get complete control over the rendering process.

In this example I have copied FORMAT.JSON to FORMAT.JSON.101 and modified it to be smarter about the JSON string that REST.SERVICES 101 returns.

I know that REST.SERVICES 101 returns a top level object with 4 properties, 3 of which are simple values and a 4th property named 'forecast' is of type array.  Forecast is, in fact, an array of objects each with 3 simple properties about a future day.  I can store the forecast properties instead of outputting them down the page, and then when the current conditions rendering is complete and the forecast properties have been parsed,  I can output the daily forecast properties across the page with a new row for each day.

REST.MSO pulls JSON data from any RESTful provider endpoint defined in REST.SERVICES. CustomFormatter subroutines let you present this data to users in a more readable format than the autoformatter.

 

Manage 2000 REST Consumer API

My latest development adventure has been expanding on text messaging REST services to create a generalized API for Manage 2000 to act as a REST API consumer.

To this end 8.1 sp5 will have a new function named REST.SERVICES which allows naming and configuring access to  REST providers.

The REST.MSO subroutine provides MSO access to REST.SERVICES endpoints.

These services can then be used in MSO.BUILD to display information from REST API provider endpoints in any Manage 2000 function.

This MSO can now be called up in PWS from the CUSTOMERS function:

Or, alternatively, from the CustomerPortal web function:

Perhaps you have an internal system that can be configured to supply a REST provider.  Let's say an MES system which lets you publish real time work order activity. You could display that in the work order status portal by configuring a REST.SERVICES item to access the provider and then publish internally as an MSO or news article.

One can envision REST API providers and consumers becoming a critical architecture for accelerating information up and down the supply chain: 

"... APIs can make certain that carriers, shippers and 3PLs have access to the same real-time data through the entire lifecycle of a shipment, providing true visibility across the supply chain...One of the greatest benefits of using API technology in a 3PL business is that APIs are capable of transmitting data back and forth across the supply chain in milliseconds, making real-time supply chain management a reality."

- Rempel, Eric | REDWOOD LOGISTICS. "Supply Chain Integration: How API-Led Connectivity Is Transforming the Logistics Industry" 3PL Magazine, 7 Jan 2019  

Whatever the source, REST.SERVICES, MSO.BUILD, and NEWS.ARTICLES allow you to build a loosely coupled integration of information from disparate sources into displays within Manage 2000.

Related topics coming soon: 

• Custom formatter subroutines for rendering JSON responses
• NEWS.ARTICLES templated support for REST providers
• Tools subroutine REST.SERVICE for application specific development of REST provider service based features

                                                  

Parsing SOAP XML Data with Default Namespaces Using UniBasic Extensions XPath

 This is another technical tidbit about the U2 UniBasic Extensions.

Most of the examples I could find about U2 XPath did not include namespace processing.  In my real-world application (integrating tax caclulations with Vertex SOAP based services) handling namespacing correctly was necessary.

XDOMLocate allows you to pass in a namespace map and this works fine for explicit namespace prefixes. It is a different story when the XML uses a default namespace and there is no prefix in the xml.

There is a secret (unexpected) trick to convincing U2 XPath that nodes with no prefix should be considered to belong to the identified default namespace.

Add an explicit prefix to the namespace map and use it in your XDOMLocate statements. You can make one up, I used 'd':

     nsMAP = 'xmlns=':VertexNamespace:' '
 * XDOM does not handle default namespace as expected, so create a dummy d: prefix to use in XPATH     
     nsMAP := 'xmlns:d=':VertexNamespace:' '

     nsMAP := 'xmlns:S=http://schemas.xmlsoap.org/soap/envelope/ '
     nsMAP := 'xmlns:nsf=http://schemas.xmlsoap.org/soap/envelope/ '
     nsMAP := 'xmlns:ns=urn:vertexinc:oseries:exception:1:0 '
     nsMAP := 'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance'

And then, when you are looking for defaulted nodes with no prefix, format the XPath expression with the 'd:' prefix:

 * Look for a VertexEnvelope, normal geodata returns will contain one, error packets won't
     node = ''           
     RESPONSE.DATA.PATH = "//d:VertexEnvelope"
     LocateStatus = XDOMLocate(domHandle, RESPONSE.DATA.PATH, nsMAP, node) 
     IF (LocateStatus = XML.SUCCESS) THEN  
 * Valid results returned from Vertex so return SOAP.MSG document and no ERR.MSG.
        DMY = XDOMClose(domHandle) 
        RETURN 
     END

Not intuitive, but it gets the job done.

Javascript Injection in Manage 2000 Web Functions Q&A

What is it?

WEB.CONSTANTS and FN.BUILD have prompts for Custom Javascript.

The scripts stored in WEB.CONSTANTS and FN.BUILD are downloaded to any Manage 2000 web site accessing that Manage 2000 account, during IIS application initialization. 

The scripts in WEB.CONSTANTS are injected into every web page. 

FN.BUILD script entries are only injected into web pages of that particular function.

In order to ensure that all other resources are loaded, scripts should be enclosed in a jQuery document ready call:

<script>
$j(document).ready(function(){...}
</script>

What is its upgrade path expense profile?

Since the scripts are stored in Manage 2000 files on the Unidata server, any custom scripts automatically migrate to new service packs and survive web server replacements.  It is possible some adjustment to script logic may be required due to changes in the mark-up or code-behind of web functions, but for the most part the scripts float along with other configuration items stored in any Manage 2000 User type account like ACME.MAIN.

In what release did it become available?

Custom Javascript Injection was first introduced in Manage 2000 release 8.1 sp1.

How can it be used?

Manage 2000 web functions include access to jQuery.js and Prototype.js.  In desktop view jQueryUI is also available, and in mobile view jQueryMobile is available.  There are also many Manage 2000 javascript functions available from the roiGlobal.js library.  Therefor brief scripts can accomplish significant adjustments to the Manage 2000 web functions.

For instance, 8.1sp4 introduced many collapsible panels which improve navigation on mobile devices. However, they quickly become annoying on desktop devices with large screens. A standard solution is coming in 8.1sp5 which will add the overlooked state memory to the panels so they remain in the state last set by the user. For those on 8.1sp4 a simple solution to this annoyance can be made through Custom Javascript Injection:

<script>
$j(document).ready(function(){
$j('.accordion-expand-all a').trigger( "click" );});
</script>

Add this to WEB.CONSTANTS and restart the IIS application accessing the account and EVERY page with collapsible panels will begin with all of the panels expanded.

This becomes a very powerful way to make adjustments to Manage 2000 web functions without introducing custom components on the web server. 

ARCHIVER UNINSTALL and REPLACE

Archiver packs together components of a function or a group of functions into a single file for storage or transport. It is also used to explode the package into its constituent items and put them away in an account. A package may be built in a development account and extracted in a live account or vice-versa.

For custom code development purposes this is like a Multi-Value version of WinZip transporting a collection of software in one go. It makes progressive development easy to redeploy no matter how large or complex the collection of software components grows.

There has always been an UNINSTALL method for removing the existing functionality before EXTRACTING and INSTALLING upgraded functionality, but it was not very accessible or possibly well known.  It also takes multiple steps to cleanly backup, uninstall, extract and install new versions.

New to ARCHIVER in Manage 2000 8.1sp5 is the REPLACE option on the ARCHIVER Extraction screen.

REPLACE automates the steps necessary to backup and replace existing functionality with replacement functionality stored in the ARCHVIER package:

1. creates a new PROJECT_LIBRARY record with Item ID {FILE.KEY}.bak. 
2. REBUILD {FILE.KEY}.bak package filling its contents with the current file items from the current account. 
3. UNINSTALL {FILE.KEY} removing all the identified file items from the current account. 
4. EXTRACT {FILE.KEY} copying all the package contents to the current account. 
5. The user is then prompted to INSTALL the package in the current account resulting in compile and screen building operations 

This feature was suggested by Mike Daniels and our thanks go out to him for helping to make Manage 2000 a better product.

Events, Conditions, Actions and Text Message Notifications

Text Messaging is the real-time mobile communication method of choice for most people. It is more effective for brief high priority communications and alerts than email notifications.

Now that Manage 2000 8.1 sp5's SUB.TEXT.OUTPUT utility understands text messaging and works with carrier email-to-text-message conversion gateways and third party texting services like Twilio and Nexmo, ECA Notifications can be directed to any contact's cell phone as a text message, providing an extremely flexible system for instantly alerting individuals inside or outside the company, wherever they are, to real-time ERP system events.

To enable control of this feature at a contact level the TextMsg and TextMsgSubj methods have been added to ECA Default Method and ECA Event Override Method prompts in CONTACTS.

With ECA 105 Notification delivery overridden from the default 'Normal' to 'TextMsg'

I can have a short ECA notification sent to my phone. Or I can use TextMsgSubj and just send the ECA subject line as a text message to my phone. I could even use TextMsgSubj in combination with Normal and receive the Subject as a text message and the detail as an email.

Just enter a new quote or whatever the triggering transaction is 

and the notification is delivered right to my hip pocket!

REST service programming using the UniBasic Extensions

 Previously I posted about the simple case of executing an HTTP GET against a URL with querystring parameters and then catching and parsing the JSON return from a REST service.  However recently I needed to dig a little deeper into UniBasic based REST service programming. My goal was to interact with Text Messaging services from Manage 2000 UniBasic code.

Nexmo and Twilio, at least, provide extensive examples for interfacing with their service from various environments, but not Multi-Value. In each case they provide handy toolkit wrappers that hide all the actual service details and expose the interface as some sort of object with methods and properties, sometimes coded as a JSON request string.

Unfortunately to implement an interface from within UniData I needed to understand some of the things that their nice little wrappers were hiding. One could imagine the JSON request string posting to the service URL and returning another JSON request string. That is, in fact, NOT what happens.

These services accept HTTP POST requests with form variables and then return JSON responses. So simply creating a GET request with a querystring, like my previous post, will not get the job done. Neither will changing the GET to POST and assuming the querystring variables will be magically transformed into form variables in the fashion of the jQuery AJAX command.

Now the UniBasic Extensions submitRequest command does includes a spot for post data:

 * submitRequest(request_handle, time_out, post_data, response_headers, response_data, http_status)

And one is tempted to try to put some type of querystring name=value expression into post_data, but I  could not get that to work. If these sites did accept JSON post data, then it would obviously fit into this argument.

What does work for x_www_form_urlencoded requests is adding in form variables to an HTTP POST request using addRequestParameter:

Ret = addRequestParameter(HANDLE, NAME, VALUE, "text/plain")

Using addRequestParameter works for both GET querystring handling and POST form variable handling so you get the nice symmetry that jQuery AJAX command exhibits.

This has the added benefit of encoding the text you are stuffing into parameters so you don't inadvertently corrupt the request.

The UniBasic Extensions HTTP POST pattern involves:

1. preparing the context for the HTTP Request using a number of setHTTP... commands
2. creating the request
3. adding the form variables with addRequestParameter
4. submitting the request

So a whole Request transaction might go something like:

 * Prepare HTTPS Request

     SECURITY.VERSION = "TLSv1.1"

     HTTP.METHOD = "POST"

     QS = ""

     CONTEXT = ""

     HTTP.REQUEST.HANDLE = ""

     HTTP.RESPONSE.HEADERS = ""

* Set HTTP version

    Ret = setHTTPDefault("VERSION", "1.1")

 * Add basic authentication credentials

     Ret = setHTTPDefault("AUTHENTICATE", AUTH.TOKEN)

 * Add content-type header

     HTTP.HEADERS = "content-type":@VM:"x-www-form-urlencoded"

     HTTP.HEADERS<2> = "user-agent":@VM:"UniData 6.0" 

     Ret = setHTTPDefault("HEADERS", HTTP.HEADERS) 

 * Create a security context.

     Ret = createSecurityContext(CONTEXT,SECURITY.VERSION)

 * Set the authentication to 'generous' while testing, otherwise UniData will require a certificate

     Ret = addAuthenticationRule(CONTEXT ,'2', 'VerificationStrength' , 'generous')

* Create Request

     Ret = createSecureRequest(URL:"?":QS,HTTP.METHOD,HTTP.REQUEST.HANDLE, CONTEXT)

 * Add POST DATA using addRequestParameter

     PARAM.CNT = DCOUNT(PARAM.NAMES,@AM)

     FOR PARAM.IDX = 1 TO PARAM.CNT

        Ret = addRequestParameter(HTTP.REQUEST.HANDLE, PARAM.NAMES<PARAM.IDX>, PARAM.VALUES<PARAM.IDX>, "text/plain")

        IF Ret <> 0 THEN 

           HTTP.RESPONSE.ERROR = "Error in addRequestParameter ":PARAM.NAMES<PARAM.IDX>:" ":Ret 

           RETURN

        END        

     NEXT     

* Submit Request

    Ret = submitRequest(HTTP.REQUEST.HANDLE,'','',HTTP.RESPONSE.HEADERS,HTTP.RESPONSE.DATA,HTTP.RESPONSE.STATUS)

 

Of course you want to check 'Ret' after each operation and expose error messages for trouble shooting.

This pattern will allow you to POST requests out of UniBasic to REST service endpoints and receive their responses. You may want to look into the UDO (U2 Dynamic Objects) commands for parsing the returned JSON.

All of this 'plumbing' code has been wrapped up in a new Tools subroutine named REST.SERVICE. More on that later.

Text Messaging from Manage 2000?

Recently, while thinking about system administrator notifications, I was chagrined to realize that SUB.TEXT.OUTPUT did NOT have a Text Message method. So I started to do a little research into Text Messaging and some more about using Unibasic Extensions to interact with REST services.

Now, there are a couple of different approaches to launching text messages programmatically:

1. Send an email to the phoneNbr@carrierGateway where the carrierGateway is a destination which converts emails to text messages for that particular carrier.

2. Send REST requests to third party services like Twilio and Nexmo which by-pass the carrier gateway. 

So if it is no bother to 'know' what carrier the phone number you are text messaging belongs to then option 1 is fine.  If on the other hand you want to text message phones which may change carriers at anytime without letting you know then it might be worthwhile to go with option 2.

To begin with I added the PHONE.CARRIERS table to 8.1 sp5 which can hold the gateway suffix for each carrier.  This can be used directly in an application to build the email destination and then the EMAIL method can be used to 'email' a text message to a phone. There is a new prompt in CONTACTS to hold the carrier.

I also added a property for Carrier to the EMAIL method. So, alternatively you can call SUB.TEXT.OUTPUT with Method=Email, To=phoneNbr and Carrier=carrierCode to send 'text messages' via carrier gateways.

Option 2 is complicated by the fact that each Text Messaging service provider uses a different interface. Yes, they are all REST, but some use Basic Authentication tokens while others pass credentials in the form variables.  They each use their own nomenclature for various variables in the request.

So I have attempted to make a pluggable interface with a new table named TEXTING.SERVICES. This table lets you configure the URL, Basic Authentication token, and map form variables from literals or properties of SUB.TEXT.OUTPUT.  We'll see how flexible it turns out to be.

At any rate, there is a new method for SUB.TEXT.OUTPUT in 8.1 sp5 named TEXTMSG which requires a property 'Service' to select the interface table from TEXTING.SERVICES. 

Next level up will be how to incorporate Text Messaging as a destination in the ECA system!

(More details of REST service programming using the UniBasic Extensions next time.)