Upload files to FTP using the WINSCP .Net library

elevator-2203695_960_720

Introduction

Applications like Filezilla and WinSCP are great simple to use FTP/SFTP clients. However what about if you want to automate uploading files. Imagine you have a brilliant SSIS package, that extracts, manipulates and formats just the right data into the correct format, using SQL Server agent to run  your package early in the morning to take advantage of the gap in transactions hitting the database server. You want these extracted data files to be sent to a third party without you needing to log in open WINSCP  or Filezilla and manually upload them.

That is what we are going to look at today, a few simple methods for zipping some files and uploading them in an application that can then be set to run on a schedule with something like Windows Task Scheduler.

 

WINSCP .Net library

WINSCP proveds  a .Net library that is a wrapper around the scripting interface available with WINSCP. This library can be used with a SSIS script task or with PowerShell among other things. However we are going to interact with it using C#, mainly because that is what I am used to.

The library has a whole range of classes up uploading and downloading files, helper classes for manipulating directory paths and getting information about the transfer.

How to Zip files

Before we upload the files we need to zip them to redue the size of the transfer. We could do with from SSIS using a Script Task. However it is simple to do in C# as part of our FTP upload application.

Lets start of with an interface

The 2 setter methods are for setting properties from outside the class, passing in the ZipFile name, which could be generated dynamically or set from a config file. The list of files could be from file system lookup of a directory looking for particular file types, or named files.

The CreateArchive method is where the file paths are looped through and added to a Zip Archive, using the name supplied. Once added to the archive each file is deleted.

How to upload the file

Now that we have the files zipped and ready to go we need to add in the WINSCP.Net library.

As we might want to upload different files to different places as part of our automation, lets add the Upload configuration to a simple class.

We can then pass this object which will take data from an ini or config file to our Upload Class.

In the Upload class we have one public class SendFile. This creates the Session Options objects and calls a WINSCP put files operation.

Summary

We have used the WINSCP .Net library to create a simple application to automate zipping and uploading of files to an FTP site.

AX Integration: Royal Mail Shipping API Part 5 – Security

RoyalMail               Dyn-AX12_v_rgb

We are building an integration between the Royal Mail SOAP Shipping API and Microsoft Dynamics AX 2012 R2. In parts 1-3 we have built a C# component to add some helper methods for interacting with the Royal Mail client proxy. In part 4 we called this client proxy in X++ and used our helper methods to extend the headers we send with the request message.

Now that we have the service client, we need to add WS Security headers to the request message to pass the Royal Mail authentication.

CLR Error Messages

When use CLR objects in X++, error checking can be an issue. However there is a handy static method available AifUtil::getClrErrorMessage().

try{
//Something with .Net objects.
catch(Exception::CLRError)
{
errorMessage = AifUtil::getClrErrorMessage();
if(errorMessage)
{
throw error (errorMessage);
}
}

WS Security

Web Service Security is an extension of the SOAP specification, the standard is published by the OASIS commitee that oversees various web standards.

The specification that Royal Mail uses is the Web Services Security
Username Token Profile version 1.0 a . This mandates a SOAP security header (added to the header of the message, rather than the headers of http headers of the request), including a username, a password digest made up of a SHA-1 hash of the password, created datetime and the nonce; a nonce and the created datetime.

In my opinion for the type of transaction that this security is being used for, it is a bit overkill. Other carrier APIs (Parcelforce, DHL, Yodel, Asendia) just use a username/password combination. After all this is not a financial transaction, and the most vulnerable data – the customers name and address is being sent unencrypted in plain text.

SHA-1 Hash

A SHA-1 Hash is a type of cryptographic message digest that is designed to require several pieces of information to be able to generate the message. It was designed by the NSA for various security protocols and was originally used for the protection of unclassified sensitive data. It has now been superseded by SHA-2, especially after SHA-1 was shown to be vulnerable to  collision attacks in 2015.

We can construct a simple function to have a password as follows:

Here we get the default encoding of the operating system, then encode the string in to a sequence of bytes. We pass this to the SHA1 hashing method available in the System.Security.Cryptography namespace, to get our message digest.

Nonce

A nonce is a randomly generated number, that can only be used once, it is designed to protect against replay attacks using old communication data.

I had a little trouble with using the random function to produce a unique number, as I kept getting a similar number as had been produced before. So I decided to use a unique number that is available in AX tables the RecId. A RecId is unique for each row in a table.

I created an AX table that would hold details of each request, useful information like the SalesId, the request log file path, datetime created and whether the request received a response or not. Each time a message request is going to be sent I add a row to the table and retrieve the RecId, then use this to populate the nonce. I use the random function as a backup in case for some reason this table logging has failed and there is no RecId returned.

Then I update the table to add the nonce that was used for this request, which is useful to check what nonce is being used.

Password Digest

The password digest is a SHA1 hash of the nonce + created datetime + hashed password. This is then converted to Base 64 for transport.

Security Header XML

Then we need to create the XML that we will add to the request message with the authentication details in.

Main Function

Putting everything together, we create the password digest, create the WSSE Security headers that are placed in the SOAP Envelope header, then store this in the client proxy’s SecurityHeaderType object.

Summary

We have created a password digest and added the WS security headers to our request message.

Everything else that you need to do to send a request message (like add shipment data, receive the response and handle error messages) is relatively straight forward once you have this set up.

AX Integration: Royal Mail Shipping API Part 4

RoyalMail               Dyn-AX12_v_rgb

We are building an integration between the Royal Mail SOAP Shipping API and Microsoft Dynamics AX 2012 R2. In parts 1 , 2 and 3 we have built using Windows Communication Foundation (WCF) some components to handle SOAP elements we need to add to the request message and security headers.

In this part we are going to create a proxy client in X++ and call some of the code we have previously written. We are going to take advantage of X++ ability to use .Net assemblies, including the code we have previously created and deployed to the AOT.

Note: I am going to show the declaration of variables alongside the assignation, getting and setting of the objects, for ease of explanation. However in X++ all variables are declared at the top of the method. The summary of the method at the end of the article will show the proper structure.

The Client

We use a CLRObject to call .Net assemblies that are not accessible at compile time, and are accessed at runtime. Our client proxy is one such object.

CLRInterop is a class that provides type marshalling, so that we can use the reference of our proxy RoyalMail.shippingAPIPortTypeClient, get the x++ type and assign this to the CLRObject.

CLRObject clientType;

clientType = CLRInterop::getType("RoyalMail.shippingAPIPortTypeClient");

We use the AifUtil class to create the service client and assign this to the service client object. The createServiceClient method looks up the type from the VSAssemblies folder and uses the correct configuration file to get the client proxy configuration.

RoyalMail.shippingAPIPortTypeClient serviceClient;

serviceClient = AifUtil::createServiceClient(clientType);

Now we have a client proxy that works very similar to it does in C#, however there are some differences to overcome, the first is the lack of generics in X++. So we end up with a very verbose way of setting collections. There is none of the property get and setting shorthand available for .Net objects in X++. So where you would be used to assigning to a public Object.Property , in X++ you use the underlying set_Property() and get_Property() methods that used in the complied IL.

ServiceEndPoint

As we want the API end point to have a dynamically supplied URL – in case Royal Mail change it for example, or because hard code parameters is a horrible thing to see in code, we need to set the endpoint address.

First of all we are going to create objects in X++ that reference .Net assemblies from the System namespace, because there are not similar methods available in X++.

System.ServiceModel.EndpointAddress endpointAddress;

System.ServiceModel.Description.ServiceEndpoint endPoint;

System.Uri uri;

Then we set a URL, I have used the X++ Parameter pattern here, where a method acts like a C# property, being able to both get and set a variable available publicly. It is usually good practice to use X++ ExtensionTypes, In this case I have created a parameter called RMTTUrl and extended the inbuilt X++ URI type.

This would look something like

public RMTTUrl parmRMTTUrl(RMTTUrl _url = url)
{
    url = _url;

    return url;
}

We can then pass a X++ type into a constructor for a .Net object.

uri = new System.Uri(this.parmRMTTUrl());

We then construct an EndPointAddress object. passing the URL we have just created, a DNS Identity with the string “api.royalmail.com” and an AddressHeaderCollection object.

endpointAddress = new System.ServiceModel.EndpointAddress(
  uri,
 System.ServiceModel.EndpointIdentity::CreateDnsIdentity(this.parmDNSIdentity()),
  new System.ServiceModel.Channels.AddressHeaderCollection());

Now that we have the EndPointAddress set we can add this to the service client’s endPoint. Unfortunately we have to do this in longhand, assigning to a variable before being able to access its properties.

endPoint = serviceClient.get_Endpoint();

endPoint.set_Address(endpointAddress);

EndPointBehaviours

In Part 3 we created a class called EndPointBehaviour which implemented IEndPointBehaviour. This had 2 public methods SetCredentials and ApplyClientBehaviour. The only method we need to call here is SetCredentials to pass in the ClientID and Secret key.

First we reference our EndPointBehaviour class, notice that as we have deployed the c# project to the AOT we have access to its classes and methods.

RoyalMail.EndPointBehaviour endPointBehaviour;

Then we need to create a reference to an EndPointBehaviourCollection, as generics are not available in X++ we have to use a ClrObject and pass in the strange-looking string as below, to create the collection.

If you compile the c# project and look at the Compiled IL code, you will see that The EndPointBehaviourCollection is complied to “System.Collections.Generic.KeyedByTypeCollection`1[System.ServiceModel.Description.IEndpointBehavior]”.

ClrObject endPointBehavioursCollection =
new ClrObject("System.Collections.Generic.KeyedByTypeCollection`1[System.ServiceModel.Description.IEndpointBehavior]");

Again we need to assign to a variable to be able to access the methods and accessors of the object. This time the EndpointBehavioursCollection from the Service Client’s endpoint object to an EndpointBehavioursCollection variable.

We pass across the client ID and Secret key we are given when we set up the Royal Mail account in Part 1.

endPointBehavioursCollection = endPoint.get_EndpointBehaviors();

endPointBehaviour = new RoyalMail.EndPointBehaviour();

endPointBehaviour.SetCredentials(

this.parmRMTTClientID(),

this.parmRMTTSecret());

endPointBehavioursCollection.Add(endPointBehaviour);

OperationBehaviours

Then we need add to the OperationBehaviourCollection. This is slightly different. The OperationBehaviourCollection is several layers down in the Service Clients structure. We need to get a reference to the service endpoint’s contract object and then to its Operations collection.

System.ServiceModel.Description.ContractDescription contract;

System.ServiceModel.Description.OperationDescriptionCollection operations;

contract = endpoint.get_Contract();

operations = contract.get_Operations();

To access a collection in X++ we need to make use of the GetEnumerator method that is available to every object that implements .Net’s IEnumerable. This returns an enumerator that we can use to get the OperationDescription

System.Collections.IEnumerator enumerator;

enumerator = operations.GetEnumerator();

Then we move through the collection using the enumerator adding our RoyalMailOperationBehaviour object that we created in part 3, to the collection, so that this will be used by the service client.

while(enumerator.MoveNext()) {

  operationDescription = enumerator.get_Current();

   operationBehaviourCollection = operationDescription.get_Behaviors();

   operationBehaviourCollection.Add(new                 RoyalMail.RoyalMailOperationBehavior());
}

The Full Method

Summary

We have created our service client that connects the Royal Mail endpoint and added our custom WCF classes created in Parts 1, 2 and 3, to the client.

In the next Post

We will add a the WS Security that Royal Mail require as part of the API request message.

AX Integration: Royal Mail Shipping API Part 3

RoyalMail               Dyn-AX12_v_rgb

To send the SOAP headers required by the Royal Mail Shipping API we will use the functionality available in WCF to set headers and intercept and alter messages as they are being sent and received by the client.

EndPointBehaviour

We have already built a ClientMessageInspector class (see Part 1 and 2) implementing the IClientMessageInspector interface. Now we need to add this message inspector to the client’s message inspector collection.

Create a new class implementing IEndpointBehavior, add the method stubs need to implement the interface. In the ApplyClientBehaviour method we will hook up the message inspector, so that this will be used for each message that is sent or received.

We also want to pass credentials to the ClientMessageInspector, so create a public method that will set a credential variable (taking the client id and secret as string). Then call the SetCredentials method on the Message Inspector before adding it to the Message Inspector collection.

Message

The Shipping API also requires a set of SOAP and WSS namespaces to be added to the message sent from the client.

This is very straight forward to do creating a custom Message class and overriding the various OnWrite methods.

 

Message Formatter

To use our custom message with outgoing requests we implement the IClientMessageFormatter interface and add our Message class in the SerializeRequest method.

This means that our custom message will be used every time a message needs to be serialized before being sent to the remote endpoint.

 

Operation Behaviour

In turn our Message Formatter needs to be set as the proxy message formatter, we do this by utilising the IOperationBehaviour interface that has a set of methods like IEndpointBehaviour. The two interfaces differ in the scope that they effect.

We use the ApplyClientBehaviour method to add our message formatter.

Summary

We now have everything set up in the WCF part of our integration. We are modifiying the HTTP headers to send security information with our request. We have modified the namespaces and added SOAP elements to the message.

In the next Post

We will look at how to call our client proxy from X++ code.

AX Integration: Royal Mail Shipping API Part 2

RoyalMail               Dyn-AX12_v_rgb

In Part 1 we covered setting up the Royal Mail API on their developer portal and getting the credentials needed to use the API. In Part 2 we will create the request and response logging messages that will be useful for debugging in test and production.

Request and Response

For the request message we want to log the headers as well as the body of the message, this is useful for debugging, especially with Royal Mail IT support who will want to know what headers were sent with the request.

I was not interested in the headers coming via the response message, so I haven’t logged them, although you could add this. For the response message I wanted to add something that references back to the request. I have used a guid, but you could use any object.

System.ServiceModel.Channels.Message Class

The Message class holds the entire communication between sender and receiver. So a message is either a Request from the client to the external endpoint, or a response.

Each message can only be consumed or written to once, so we need to copy the message to a MessageBuffer to be able to read it. We therefore leave the referenced message unaffected allowing us to consume the copy.

The Message.ToString() method calls a XmlDictionaryWriter which read the message and write it out as xml string to our log file.

Request Message Log

Using the copy of the message, extract the HTTPRequestMessageProperty headers that we added in the ClientMessageInspector and write them out to the log file with the body of the message.

Response Message Log

Add the guid to the top of the file to reference back to the request message log. Then get the StatusCode from the HttpResponseMessageProperty and write this along with the message body out to the log file.

In the next Post

We will finish off the .Net part of the solution adding classes that will implement IEndpointBehaviour, IClientMessageFormatter and IOperationBehaviour.

AX Integration: Royal Mail Shipping API Part 1

RoyalMail               Dyn-AX12_v_rgb

In this series we are going to look at how to set up the Royal Mail SOAP Shipping API with AX 2012 R2. I will primarily use X++ code to connect to the Royal Mail client proxy, however some C# WCF code will be needed to set the correct SOAP security headers and log incoming and outgoing messages.

Setting up the Royal Mail API Account

Your company will need a contract with Royal Mail and you will need to go through their onboarding process, as part of these you will need to create a new account on their developer website

RM-CreateAC

Create a New Account

Then register your application that will use the API. In the long run you will want at least 2 applications one for live and one for test/development. For the moment just set up a test one.

You will need to have the 10 character Royal Mail account number they gave you when you registered with them.

RM-RegisterApp

Register an application

You will receive a client ID and Secret when you register the application you will need these so keep them somewhere safe but retrievable.

Testing the API in the Portal

Go to the API Portal and choose Shipping API (SOAP). Here you can see all the operations you can use and examples in different programming languages of how to call the API and what the resulting request and response messages look like.

rm api portal

Shipping API Portal

Here test that your credentials work properly, by finding the Try this Operation section on the right hand side console like window and entering your Client Secret.

This is also a good place to return to when you want to quickly check that the requests you are sending from your application are valid and will work with the API.

You will also find the technical and user documentation for this API, which are well worth reading to make sure you are sending valid data.

Set up the WCF Project

Thank you to Mohamed Mahmoud El-Geish getting me started on this solution

Create a new C# Library Project and add a service reference to https://api.royalmail.net/shipping/v2 calling it RoyalMail

In the app.config make sure the basicHttpBinding  uses Transport security mode and that the maxReceivedMessageSize, maxBufferSize and maxBufferPoolSize are something large (I used 20000000).

We want to do 2 things with this project

  1. Add some HTTP Request Headers
  2. Add logging of the Request and Response messages

ClientMessageInspector

Lets start with the adding a class that implements IClientMessageInspector

This Message Inspector will be added to a MessageInspector collection and is used to inspect or modify messages passing through the WCF proxy.

The interface has 2 methods

  • BeforeSendRequest
  • AfterReceiveReply

BeforeSendRequest

We will use BeforeSendRequest to

  1. Set the HTTPMessagePropertyHeaders

We will pass the credentials to the ClientMessageInspector object from AX.

Then we check if there is a HttpRequestMessageProperty which holds the message headers. If this object does not exist we can create it and added it to the Request properties.

In the SetHeaders function we need to set the following:

  • add The Accept Header, setting this to application/soap+xml
  • add the Royal Mail API Client Id
  • add the Royal Mail API Secret

2. Log the Request Message

For debugging it is going to be useful in production to be able to view what messages have been sent and received. To avoid the logging code causing exceptions I have wrapped it in a try/catch and set it to silently fail.

For testing we can use something like Fiddler to check the request and response messages.

I also set a GUID, which we can use to identify the relationship between the Request and Response message.

AfterReceiveReply

Finally in this method we will just log the response message adding the CorrelationState which is the Guid we set in BeforeSendRequest.

This will be added to the response message log allowing a way to link back to the original request message.

In the next Post

We will add classes to write the request and response messages out to file for logging.