Automating REST Authorization Calls

REST services provided by 3PL partners generally require a two-step process to use their service.  First a call must be made with secret credentials to obtain an authorization token.  Then the auth token is submitted with subsequent requests.

In Manage 2000 REST.SERVICES we can create one REST.SERVICES record to obtain the auth token and then reference it in another REST.SERVICES record:

GetAuthToken accepts JPATH expression parameters to identify the token and the expiration span. It then caches the token for the expiration span and uses the cached version until it expires before going back to the REST service to get a fresh token.

Services like the FEDEX Rates service accept a great many parameters.  These can be specified in the REST.SERVICES record:

A SCREEN.BUILD screen built with matching X_DATA_1 names makes it easy to map the inputs into the RS.COM area for calling the 3PL REST API for information.

The program needs to:

• Call SUB.MT500 to run the screen
• Overlay the variables from the screen to the REST common
• Call REST.SERVICE
• Parse the result with JPATH
• Present the results to user, perhaps in the PWS Viewer using SUB.TEXT.OUTPUT.

And the result is a Manage 2000 look and feel function that gets live information from the 3PL REST service:

Manage 2000 release 8.1 Service Pack 6 - sneak peek

The development cut is scheduled for Friday March 3rd.  Release Control processing will follow through the remainder of March.  And so expect GA sometime around the traditional April 15 timeframe.

Service pack 6 includes a number of important application features, more LINUX, and more Tools features.

AP.1099.AUDIT.ADDR

- a new function for 1099 year-end processing.

Retail Delivery Fee

- a new Sales Order feature to provide support for complying with new state RDF regulations.

Surcharge

- a new percent-based base extra charge code feature used in order and quote entry when an additional surcharge is needed. 

CountEntry

- a web version of cycle count COUNT.ENTRY. This is a major rewrite of the CountEntry web function first released with service pack 5.

ApproveCredit

- a web version of the CREDIT function.

Grand Totals Added to 30 Web Functions

CustArItems, CustOpenSalesOrders, VendorOpenInvoices, and many others now display grand totals for quick access to the customer's or vendor's overall position.

Fax Option on Red Hat Enterprise Linux

Conditional Support for running Manage 2000 on RHE Linux was announced with Manage 2000 release 8.1 service pack 5. However VSI-Fax features in Manage 2000 are NOT supported in that environment.

With service pack 6 the Manage 2000, faxing features like FXJ and QUICK.FAX are supported on RHE Linux via a Windows Server 2019 proxy.

Web Pseudo Functions

Wrapping web functions like LOOK.AR and LOOK.SO now accept type ahead parameters, so expressions like ‘LOOK.AR 1024’ can be used in the place of ‘CustomerPortal 1024’.

MyPrintJobs 

Now displays most recent print jobs first, and offers an option to delete the print job.

User Prompt Stack

Extended on primary key and validated prompts using the Most Recently Used list from the USER_CONTEXT file. This provides up-arrow-key access to recently used keys on many PWS screen prompts which matches the web function behavior.

REPORT.BUILD

Grand totals are now supported in the web interpreter page for REPORT.BUILD reports and in the roiHyperQueryControl, making them available on many user created web inquiries.

REST Provider API

Manage 2000 now provides 50 REST services covering:

• Authentication and Authorization
• Application Server Storage
• Primary Key Selection
• Data Validation and Formatting
• Data Retrieval
• Shopping Cart Operations

Manage 2000 REST Services have been tested and documented through Postman. Use the Manage 2000 function M2K.REST.SERVICES to access the API documentation and test cases in Postman. 

Testing and Documenting Manage 2000 REST Services using Postman

Manage 2000 release 8.0 provided 15 poorly documented REST services.

Release 8.1 sp5, the current field release, provides 18 poorly document REST services.

Release 8.1 sp6 (projected release Q2 2023) will provide 50 well documented REST services.

Postman has become an invaluable development tool for both testing and documenting the Manage 2000  REST Services.

The results of Manage 2000 development using Postman can be found at: Manage 2000 REST Services

The Manage 2000 REST Services collection may be imported into Postman and used for your own testing and debugging.

Multiple environments may be setup for running examples against MANAGE-2000, xxx.MAIN, xxx.TRAIN, xxx.DEV...

Service root URL and Auth Tokens may be defined in one place in the environment and then used on all services.

The resulting documentation provides not only parameter specifications but also examples with results, so developers can clearly see the content and format of the data being passed in and the results being returned.  

The results of using Postman speak for themselves.  It is clearly a huge contributor to the usability and quality of today's and tomorrow's REST APIs.

Bearer Token Authorizations and JPATH Expressions

 Yeah, apples and oranges.  And I am not going to relate the two either.  They are just the flotsam and jetsam floating about in my mind from recent development.

Bearer token authorization comes out of the OAuth 2.0 standard, but has taken on a life of its own and become a commonly used authorization mechanism for REST services.

In the Rocket UniData UniBasic Extensions Version 8.2.2 you can find details on the Authorization setting required in setRequestHeader.  There is a whole intro page about OAuth 2.0 and U2.  

Short version:

setRequestHeader(HTTPRequest.Handle,"Authorization", "Bearer ":BEARER.TOKEN)

setRequestHeader(HTTPRequest.Handle,"Authorization", "DIRECT ":HMAC.TOKEN)

And these have to be done after the Request object is created.  There is no option in setHTTPDefault to specify the authorization type ahead of time. 

So in the REST.SERVICES table you just need to start the Auth Token prompt with "Bearer " or "DIRECT " and then the token, and REST.SERVICE will setRequestHeader and your REST.SERVICE request will be off to the races.

JSONPath or JPATH is for JSON what XPATH is for XML; a standard for declaratively specifying pieces of a JSON string to be returned.  I stumbled into the need for such a thing while working on the REST provider for NEWS.ARTICLES.  If you want to provide the capability to an article author to layout an HTML div and specify replacements from a REST service then you need some standard declarative expression to say "put THAT piece from the REST service results HERE!". And it would be nice to make the replacement expressions familiar and consistent with some larger standard.

The REST.NEWS news provider subroutine simply calls out to JPATH to extract out the pertinent data.

 *

2110* Replace dictionary references with values

 *     

     VAR.CNT = COUNT(LINE, "&")/2

     FOR VAR.IDX = 1 TO VAR.CNT

        VAR.IDX = VAR.IDX ;* Audits!

        VAR.NAME = FIELD(LINE, "&", 2, 1)

        TARGET.NAME = VAR.NAME

        TARGET.VALUE =  JPATH(udoRESTData, TARGET.NAME )

JPATH is not yet a W3C standard.  There is an IETF working group developing a standards-track JSONPath specification based on work by Stefan Goessnerwhich.  So while not set in stone there is general consensus on what a JPATH expression ought to look like and how it ought to function.

There are no JPATH features in the Rocket UDO functionality currently. However, the primitives are there for building a JPATH function.  And that is what I have started. It is available as a patch to Manage 2000 8.1 sp5.  And I would lead with "not complete yet".  But it provides the basic functionality which allows expressions in NEWS.ARTICLES like "&$.Customers[1].OutputEmailAddresses.SaleseQuoteEmail&" and since "$" is optional and array slice filter defaults we can write more concisely &Customers.OutputEmailAddresses.SalesQuoteEmail&.  

Why "not complete yet"?  Well it will do basic array slicing expressions, but a lot of work remains to implement all the possible filter expression and array slices within array slices and so forth.  

Manage 2000 Release 8.1 Service Pack 5 Now Available

Just in case you missed the official announcement:

 "We are pleased to announce that Release 8.1 Service Pack 5 was released today, March 23^th, 2022 and is now available for upgrades."

The surprise in the official announcement is the limited support for LINUX.  We still have the vsi-fax caveat, faxing out of Manage 2000 on LINUX is NOT there yet. But if you want to run Manage 2000 8.1 sp5 on rhe LINUX 8 and do not need faxing then we are ready to support you in that. 

Bud and I continue digging through the options trying to find the best path to get faxing support on LINUX, without ESKER directly supporting vfx... on rhe LINUX 8.  You'll probably hear us shout when we finally find the right path.

Customer enthusiasm for moving forward on Manage 2000 has been gratifying to see with four 8.1 sp5 upgrade orders already in the first week of GA.

So what will we imagine for service pack 6?  

My journey this last year has included the opportunity to explore what other software teams are doing with REST API development.  While 8.1 sp5 includes a nice REST API platform, it is mostly 'toolsey' stuff that you can use to build your own, but does not provide the full manufacturing application provider API that I am seeing other software teams working on.  So one of my research and development projects is imagining how we can take the REST API platform and build out an API to specific sales and manufacturing processes.

The current leg of my continuing journey has me exploring POSTMAN and using it to test and document existing REST services:

Manage 2000 JSON Services

A little quicker and easier than running JSONServices, viewing source, and reverse engineering :-)

Application Specific Development of REST Provider Service Based Features

Adding information displays for users in Manage 2000 from REST providers is all well and good, but how can we use the Manage 2000 REST.SERVICES connections to do other more interesting and specific things?

What if the RESTful service lets us POST information back to the provider system?  

What if we want to develop integrations with FEDEX and UPS RESTful service APIs which supply not only tracking services, but Rates, Time in Transit, Dangerous Goods, Global Trade Documents and more?

Back in March I wrote about manually coding REST HTTP Post method calls in UniBasic. But it is not necessary to drop down to this level. REST.MSO and REST.NEWS rely on a new Tools subroutine named  REST.SERVICE to carry out the actual conversation with the REST provider system. 

Application code can declare the parameters for the REST exchange and then delegate the implementation to REST.SERVICE:

     $INCLUDE COM.RS FROM COPY.TOOLS.BP
     $INCLUDE TM_328 FROM TOOLS.LAYOUTS
     ...
     READ TM_328.REC FROM TM, '328*':SERVICE.ID
     ...

     RS.Endpoint               = TM_328.Endpoint
     RS.Protocol               = TM_328.Protocol
     RS.Security_Version       = TM_328.Security_Ver
     RS.HTTP_Version           = TM_328.Http_Version
     RS.Auth_Token             = TM_328.Auth_Token
     RS.Method                 = TM_328.Method
     RS.Content_Type           = TM_328.Content_Type
     RS.Parameter_Names        = TM_328.Parameter_Names
     RS.Parameter_Values       = TM_328.Param_Values
     RS.Parameter_Types        = TM_328.Param_Types
     RS.Body                   = TM_328.Body
     RS.Timeout                = TM_328.Timeout
 *
     CALL REST.SERVICE
     IF RS.Status # "" THEN CALL SCREEN.MSG("RS.Status - ":RS.Status<1,1>:" ] ":RS.Status<1,2>:";VT;#M260618")
     IF RS.Headers # "" THEN CALL SCREEN.MSG("RS.Headers - ":RS.Headers:";VT;#M260619")
     IF RS.Data # "" THEN
        SWAP @AM WITH @VM IN RS.Data        
        RS.Data = CONVERT(FILTER.CHARS, "", RS.Data)
        CALL SCREEN.MSG("RS.Data - ":RS.Data:";VT;#M260620")
     END 
     IF RS.Error # "" THEN  CALL SCREEN.MSG("RS.Error - ":RS.Error:";VT;#M260621")
     RETURN

Parameters may be loaded from the REST.SERVICES table as in this example or created on the fly, or some combination of the two. You might, for instance, want to post a bunch of FORM variables based on a work order or sales order into some RESTful provider endpoint defined in REST.SERVICES, and perhaps receive back some confirmation number or message.

REST services may accept inputs on the querystring or as part of the URL route path when using an HTTP GET method, or the service may accept inputs as form variables or as JSON or XML strings in the HTML Body element when using an HTTP POST method.  The REST.SERVICE subroutine supports all of these approaches to REST parameter passing.

Parameters are placed in the appropriate property in a named common block, then your program executes CALL REST.SERVICE, and then interrogates the result values which are also stored as properties in the common block. If the call is successful then RS.Data will contain the resulting JSON string.  All of the UniBasic Extensions' HTTP plumbing is taken care of by REST.SERVICE.

As always tools subroutine documentation is available from HELP:

     HELP REST.SERVICE

Templated Rendering of REST Provider's JSON Responses in Manage 2000

REST.MSO CustomFormatter subroutines are great for Unibasic programmers who don't mind familiarizing themselves with UDO JSON parsing and building text and HTML displays.

But what can you do without getting into programming?

There is another REST JSON formatter named REST.NEWS. Its purpose is to support writing NEWS.ARTICLES with content from REST providers.  It works similarly to the MCD_DATA.NEWS AR/SO dashboard code behind subroutine in that it lets you layout a template in the news article body with &var& type replacement variables. 

Use the new News Source REST in place of CUSTOM to save yourself some typing.

REST.NEWS will run the REST service specified and then look for expressions in the body like &propertyname& and replace them with the corresponding value from the web service JSON response. It will also check for expressions matching variables sent to the web service like CITY in this example.

So we can hit the same REST provider as our previous MSO and see the results laid out nicely, all without doing any actual programming:

There is an important distinction between MSOs and NEWS.ARTICLES in their scope.

At release 8.1 sp5 MSOs can be configured to be available in any PWS or web function throughout Manage 2000. 

The scope of news articles is a little different:

In terms of Manage 2000 functions, you only see them in NewsReader and the News panels of the 16 or so web functions in the ROIPortals directory. 

However, news articles are NOT limited to viewing in Manage 2000 functions. Manage 2000 news feeds can be viewed from any RSS client. This includes Outlook, Sharepoint, Lotus, the built-in IE newsreader, browser add-ons like Sage for FireFox and SlickRSS for Chrome, standalone readers like RSSOwl, and others.

There is also an interesting reciprocity between the two: MSOs can be used as the source for NEWS.ARTICLES, and since Newsreader is a Manage 2000 function an MSO can wrapper a display of articles in Newsreader.

In this sense the news article scope is actually wider than the MSO scope.  One could even use the auto-start feature of MSOs to fire off a Newsreader display based upon the user entering any particular PWS function or prompt.

Custom Formatter Subroutines for Rendering RESTful Service JSON Results as Text or HTML

 There is an intriguing property in MSO.BUILD on the REST.MSO parameter screen named CustomFormatter:

When this property is blank the rendering of the JSON results from the REST endpoint is handled by a subroutine named FORMAT.JSON which, at release 8.1sp5, you will find in SUB.MFG.RPT.BP.

FORMAT.JSON does a blind format, not knowing anything about the JSON string it is formatting. It just goes through the string displaying property names and property values somewhat like a property inspector in an IDE might do.  You can use the IncludeFilter or the ExcludeFilter to eliminate unwanted properties.  But you do not have any control over the text or html rendering.

By writing your own custom formatter subroutine you get complete control over the rendering process.

In this example I have copied FORMAT.JSON to FORMAT.JSON.101 and modified it to be smarter about the JSON string that REST.SERVICES 101 returns.

I know that REST.SERVICES 101 returns a top level object with 4 properties, 3 of which are simple values and a 4th property named 'forecast' is of type array.  Forecast is, in fact, an array of objects each with 3 simple properties about a future day.  I can store the forecast properties instead of outputting them down the page, and then when the current conditions rendering is complete and the forecast properties have been parsed,  I can output the daily forecast properties across the page with a new row for each day.

REST.MSO pulls JSON data from any RESTful provider endpoint defined in REST.SERVICES. CustomFormatter subroutines let you present this data to users in a more readable format than the autoformatter.

 

Manage 2000 REST Consumer API

My latest development adventure has been expanding on text messaging REST services to create a generalized API for Manage 2000 to act as a REST API consumer.

To this end 8.1 sp5 will have a new function named REST.SERVICES which allows naming and configuring access to  REST providers.

The REST.MSO subroutine provides MSO access to REST.SERVICES endpoints.

These services can then be used in MSO.BUILD to display information from REST API provider endpoints in any Manage 2000 function.

This MSO can now be called up in PWS from the CUSTOMERS function:

Or, alternatively, from the CustomerPortal web function:

Perhaps you have an internal system that can be configured to supply a REST provider.  Let's say an MES system which lets you publish real time work order activity. You could display that in the work order status portal by configuring a REST.SERVICES item to access the provider and then publish internally as an MSO or news article.

One can envision REST API providers and consumers becoming a critical architecture for accelerating information up and down the supply chain: 

"... APIs can make certain that carriers, shippers and 3PLs have access to the same real-time data through the entire lifecycle of a shipment, providing true visibility across the supply chain...One of the greatest benefits of using API technology in a 3PL business is that APIs are capable of transmitting data back and forth across the supply chain in milliseconds, making real-time supply chain management a reality."

- Rempel, Eric | REDWOOD LOGISTICS. "Supply Chain Integration: How API-Led Connectivity Is Transforming the Logistics Industry" 3PL Magazine, 7 Jan 2019  

Whatever the source, REST.SERVICES, MSO.BUILD, and NEWS.ARTICLES allow you to build a loosely coupled integration of information from disparate sources into displays within Manage 2000.

Related topics coming soon: 

• Custom formatter subroutines for rendering JSON responses
• NEWS.ARTICLES templated support for REST providers
• Tools subroutine REST.SERVICE for application specific development of REST provider service based features

                                                  

REST service programming using the UniBasic Extensions

 Previously I posted about the simple case of executing an HTTP GET against a URL with querystring parameters and then catching and parsing the JSON return from a REST service.  However recently I needed to dig a little deeper into UniBasic based REST service programming. My goal was to interact with Text Messaging services from Manage 2000 UniBasic code.

Nexmo and Twilio, at least, provide extensive examples for interfacing with their service from various environments, but not Multi-Value. In each case they provide handy toolkit wrappers that hide all the actual service details and expose the interface as some sort of object with methods and properties, sometimes coded as a JSON request string.

Unfortunately to implement an interface from within UniData I needed to understand some of the things that their nice little wrappers were hiding. One could imagine the JSON request string posting to the service URL and returning another JSON request string. That is, in fact, NOT what happens.

These services accept HTTP POST requests with form variables and then return JSON responses. So simply creating a GET request with a querystring, like my previous post, will not get the job done. Neither will changing the GET to POST and assuming the querystring variables will be magically transformed into form variables in the fashion of the jQuery AJAX command.

Now the UniBasic Extensions submitRequest command does includes a spot for post data:

 * submitRequest(request_handle, time_out, post_data, response_headers, response_data, http_status)

And one is tempted to try to put some type of querystring name=value expression into post_data, but I  could not get that to work. If these sites did accept JSON post data, then it would obviously fit into this argument.

What does work for x_www_form_urlencoded requests is adding in form variables to an HTTP POST request using addRequestParameter:

Ret = addRequestParameter(HANDLE, NAME, VALUE, "text/plain")

Using addRequestParameter works for both GET querystring handling and POST form variable handling so you get the nice symmetry that jQuery AJAX command exhibits.

This has the added benefit of encoding the text you are stuffing into parameters so you don't inadvertently corrupt the request.

The UniBasic Extensions HTTP POST pattern involves:

1. preparing the context for the HTTP Request using a number of setHTTP... commands
2. creating the request
3. adding the form variables with addRequestParameter
4. submitting the request

So a whole Request transaction might go something like:

 * Prepare HTTPS Request

     SECURITY.VERSION = "TLSv1.1"

     HTTP.METHOD = "POST"

     QS = ""

     CONTEXT = ""

     HTTP.REQUEST.HANDLE = ""

     HTTP.RESPONSE.HEADERS = ""

* Set HTTP version

    Ret = setHTTPDefault("VERSION", "1.1")

 * Add basic authentication credentials

     Ret = setHTTPDefault("AUTHENTICATE", AUTH.TOKEN)

 * Add content-type header

     HTTP.HEADERS = "content-type":@VM:"x-www-form-urlencoded"

     HTTP.HEADERS<2> = "user-agent":@VM:"UniData 6.0" 

     Ret = setHTTPDefault("HEADERS", HTTP.HEADERS) 

 * Create a security context.

     Ret = createSecurityContext(CONTEXT,SECURITY.VERSION)

 * Set the authentication to 'generous' while testing, otherwise UniData will require a certificate

     Ret = addAuthenticationRule(CONTEXT ,'2', 'VerificationStrength' , 'generous')

* Create Request

     Ret = createSecureRequest(URL:"?":QS,HTTP.METHOD,HTTP.REQUEST.HANDLE, CONTEXT)

 * Add POST DATA using addRequestParameter

     PARAM.CNT = DCOUNT(PARAM.NAMES,@AM)

     FOR PARAM.IDX = 1 TO PARAM.CNT

        Ret = addRequestParameter(HTTP.REQUEST.HANDLE, PARAM.NAMES<PARAM.IDX>, PARAM.VALUES<PARAM.IDX>, "text/plain")

        IF Ret <> 0 THEN 

           HTTP.RESPONSE.ERROR = "Error in addRequestParameter ":PARAM.NAMES<PARAM.IDX>:" ":Ret 

           RETURN

        END        

     NEXT     

* Submit Request

    Ret = submitRequest(HTTP.REQUEST.HANDLE,'','',HTTP.RESPONSE.HEADERS,HTTP.RESPONSE.DATA,HTTP.RESPONSE.STATUS)

 

Of course you want to check 'Ret' after each operation and expose error messages for trouble shooting.

This pattern will allow you to POST requests out of UniBasic to REST service endpoints and receive their responses. You may want to look into the UDO (U2 Dynamic Objects) commands for parsing the returned JSON.

All of this 'plumbing' code has been wrapped up in a new Tools subroutine named REST.SERVICE. More on that later.

REST Web Services from UniBasic

In the midst of working up some labs for Perspectives I started thinking about REST web service access from UniBasic. The lab I happened to be working-up is on SOAP based RPC web service usage, for which we have a number of working examples and implemented projects.

But in creating services for Manage 2000 internal consumption I have all but abandoned SOAP RPC in favor of the far more elegant JSON REST model. I am usually working in ASP.NET and IE DOM client land, thankfully with the prototype.js library. So the question of the day was what would REST JSON access look like from UniBasic and what are the central issues in using it?

Well in addition to all the SOAPRequest support added with the UniBasic Extensions are a couple of simple little commands for retrieving the content from a URL. Now URLs often return HTML intended for human consumption and are not very easy to parse. The JSON REST model however returns very nice orderly string serializations.

With something as simple as:

CREATE.REQUEST.RTN.CODE = createRequest(URL:"?":QS,HTTP.METHOD,HTTP.REQUEST.HANDLE)

SUBMIT.REQUEST.RTN.CODE = submitRequest(HTTP.REQUEST.HANDLE,'','',HTTP.RESPONSE.HEADERS,HTTP.RESPONSE.DATA,HTTP.RESPONSE.STATUS)

you can get orderly responses like:

HTTP.RESPONSE.DATA = {'oValItem':{'FileName':'CM', 'TableNbr':'', 'ItemId':'1024

', 'NewItemId':'1024', 'Valid':'True', 'Display':'Sears Systems, Incorporated',

'ErrorFlag':'False', 'ErrorMsg':''}}

UniData does not yet include a JSON DOM to match the XML DOM of the UniBasic Extensions, but the parsing difficulties of JSON serializations are relatively minor. And if you need to directly access JSON REST services in the middle of UniBasic code this seems like nice direct approach.

The real stumbling blocks aren't coding difficulties, but security of your application server if you open up the HTTP or HTTPS ports for UniBasic to "see" services on the Internet. One answer to this is to go through a proxy server. This also points out one of the benefits of the architecture of Manage 2000 with its ASP.NET arm which may be used in a similar manner to delegate interaction with the messy world away from your closely guarded business database into a DMZ area.