Sunday, May 10, 2020

Recursive calls in Oracle Integration Flows (Scenario: Paginated API calls for large Data Sets)

A number of use-cases can be implemented cleanly using a recursive approach. This post is not to debate the pros and cons of recursion versus looping but provides a simple approach to achieve this.
For scenarios such as the ones listed below, and possibly more, this approach is quite efficient, concise, maintainable, and most importantly, it is highly scalable. It also leaves a smaller runtime footprint with a smaller execution time per instance than a looping flow instance. This also makes error handling easier as I will describe later. 

  • Polling (continuously monitoring an FTP location, a database, or an API output)
  • Paginated API's (when the target system exposes an API with a paginated* interface such as the eBay findProducts operation)
  • Retryable flows

Thursday, July 18, 2019

Fault tolerance in integration flows - handling target system availability problems

An important non-functional property of any software system is "Availability". In the ISO/IEC 25010:2011 product quality model, this is grouped under an overall category of "Reliability". 
Fault tolerance is a closely associated property also grouped under "Reliability". 

System downtimes could be either due to scheduled maintenance

Wednesday, October 18, 2017

Selective persistence of Oracle Diagnostic Logging (ODL) output

Background and Goal

In any application, logging is widely used for diagnostics and debugging. 

Logging at various "checkpoints" (such as entering with request, exiting with response, error handler) in the application can provide a fairly reliable way to trace the execution path of the application - which a subsequent sweep or count can be used to report on. When the logs are regularly analysed and reported on, anomalies can get flagged up proactively and investigated further. Some examples

Tuesday, October 17, 2017

Geographical clusters with the biggest concentration of web services

From a data set of approximately 145 million IP addresses running at least one publicly accessible web service (such as a website), I was able to determine these 20 geographic "clusters".



Saturday, October 07, 2017

Raw results - countries list with total IP (IPv4) addresses


Background: 
http://weblog.singhpora.com/2017/10/how-many-programmers-does-it-take-to.html


Presented below is a list of countries (country codes) and the total count of live IPv4 addresses where a public facing service (such as a website) might be hosted as counted from the scan data of 1st October 2017

The reason these don't quite add up to anywhere in the ballpark of 4 billion (the total IPv4 address space) is because the data set I used might only be scanning for hosts that run some public service exposed over a TCP port (e.g. a website running on port 80 or 443)

The numbers definitely look incorrect and total up to only 145,430,195 - I will continue to investigate why, but they seem to be in proportion. 
It is likely that scans.io are only able to gather data about live IP addresses at the time of the scan as opposed to total allocated ones)


+-------+--------+
|country|ip_count|
+-------+--------+
|     LT|  120718|
|     DZ|  362827|
|     MM|    3494|
|     CI|   18954|

Friday, October 06, 2017

How many programmers does it take to update a Wikipedia page?

......or what it took to count the number of IPv4 addresses in every country (as of 1st October 2017). 

This Sunday, I found that the Wikipedia page on List of countries by IPv4 address allocation was using data from 2012. I wondered what it might take to add more up to date information on that page. During a recent course I attended, I got to know about scans.io - a fascinating project that involves periodically scanning ALL of the IPv4 address space and storing as much of publicly visible metadata about the active addresses as possible (location, ISP, open ports, services running, operating system, vulnerable services running if any). Each daily dump of the IPv4 address space is close to a terabyte.
An individual IP address record is represented as a JSon object - part of one of the records is shown here:


Saturday, September 30, 2017

Test driven SOA: Tool kit for comprehensive automated test coverage

In this post I am going to share some tools I find useful when developing components for the Oracle service bus - same principles should apply to the integration cloud service as well. 

If we are not test first (or at least test alongside) programming, we are essentially debug later programming (See "Physics of Test Driven Development").

If the enterprise service bus sits in the middle of an organisation's messaging and integration landscape, there are some key architectural principles that help in getting the best out of any service bus solution:
  • It is not the place for business logic but for integration logic i.e. heavy on message transformations and often enrichment
  • Any operations, message flows or pipelines that the service bus exposes should be stateless and without side effects (ideally). To achieve this, a lot would depend on backend services too - they would ideally need to be idempotent. 
  • Exposed interfaces must be designed to be canonical while invoked endpoints abstracted away so that calling systems are decoupled from calling systems (and then there are non-functional elements of decoupling that the Service Bus can help achieve too such as by messaging - but this post is not about the value addition of Service buses)
  • Like any other software, it must have comprehensive unit test coverage (no, not the platform but what we have developed on it) and I might be stating the obvious here but I often find test coverage inadequate at many FMW customer sites. 

Sunday, May 14, 2017

Enterprise World Wide Cloud - Notes from the Oracle #PaaSForum 2017



This must be one of the last posts on #PaaSForum 2017 that anyone has written. Too late but I hope not too little, as it gives us an opportunity to reflect upon the material presented there. I heartily thank Jürgen Kress for organising this event on a grand scale, at a beautiful location (Split, Croatia this year), for his invitation, warm welcome and hospitality. Also thanks to his colleagues from Oracle for their help in organising this event and in sharing their deep knowledge.  Further, we gained a wealth of information from leading Oracle professionals from all over the world - I felt privileged to meet so many of them in person!  We started this event on a positive note (Jürgen literally asked us to take our jackets off and get hands-on with some cutting edge Oracle technology!) and ended it full of enthusiasm watching the sunset over Adriatic sea over drinks and conversation.



Although I learnt a lot in the various presentations, please don't assume every idea or interpretation below is endorsed by Oracle or other partner presenters who spoke on the topics I have chosen to write about.
Review and comments are very welcome (especially if you find inaccuracies in my writing).
Notation guide: Numbers in [square brackets] refer to items in the References section. 

Tuesday, December 06, 2016

Progress with the Oracle Integration Cloud Adapter SDK

In the past few days, I have been making some progress with using the ICS Cloud Adapter SDK. 
Today, I created my first shell adapter - the design time views can be seen below!

The journey so far: 
 * Installation of all the offline material [Check]
       Gotcha's to note here: the step to install SDK patches wasn't required for 12.2.1 (the version I was on). 
 * Reading through the documentation [Ongoing]
 * Developing the empty adapter and deploying it for design time and runtime [Check]

There are a number of integration use-cases that we have identified. If all goes well, these will be available for a wider rollout, helping customers implement some complex integration use-cases with some important cloud services in "hours, not months" in keeping with the Oracle ICS philosophy (and in line with DRY software engineering)!

More coming soon...

Tuesday, October 25, 2016

WS Security - enabling passwordDigest authentication in an Oracle FMW environment

Objective:
To have a basic level of authentication on web services (especially where there's no transport layer security) without having to pass clear text passwords in the WS Security headers. 

Background:
The concepts are fairly generic but this post is highly Oracle Fusion middleware/SOA Suite specific. There can be complex decision tree (see [1]) involved when selecting the 'appropriate' level of security for any system. As security involves trade-offs between cost, performance, usability and other variables, the 'appropriate' level of security could be highly specific to the environment, usecase, system and people. But as developers, we can still perform some due diligence based on the tools and knowledge available to us.  

My rule of thumb when developing a traditional web service or microservice is: If it's reading from a secure database or some system that is accessible only via authentication, it must only expose a secure endpoint. 

Now sites can differ considerably and so does the definition of what "secure" is. 
When exposing ah http endpoint (SOAP or REST) hosted on cloud or accessible over the Internet, one would as a minimum ensure that it's over TLS and has authentication enabled. 

In an on-premise hosted solution, traditionally https has not been widespread within organisations and web service endpoints meant for internal consumption have most commonly only been exposed over http - hopefully accompanied by infrastructure level setup (firewalls, DMZs etc.) that ensures that the data or service is only accessible inside a 'trusted' network. 

Even in a trusted network without TLS, it is probably best if passwords weren't floating around in clear text (which is what the default UsernameToken with passwordText policies do)
With a few steps, one can enable the passwordDigest authentication that not only protects the password in-transit, but also provides protection against replay attacks (if nonce and creation time properties are set in the SOAP header as well)

Steps:
  • Basic steps are listed here:
  • For step 9.3.3, what I do is create a new policy pair based on oracle/wss_username_token_service_policy
This is done via /em -> WeblogicDomain -> Web Services -> WSM Policies

Search for oracle/wss_username_token_service_policy and copy to create a new one with the settings for passwordDigest applied (as per step 9.3.3 of the Oracle guide)

The one I created is singhpora/wss_UsernameToken_PasswordDigest_service_policy and 
singhpora/wss_UsernameToken_PasswordDigest_client_policy 
(based on oracle/wss_username_token_client_policy) and I keep these source controlled. (An additional benefit of putting them in source control is so developers can import these into their local JDeveloper policy store for design time and also to promote their initial versions or changes across environments from development through to production,much like any other artefact)
  • Another associated step is to have the oracle.wsm.security map and basic.credentials key to be present on the server that will use the client policy (you can use custom map and key names if required). This needs to contain the username(s) and passwords of the users who are allowed to invoke the web services that use the username token policies. (You can follow the principle of least privilege when assigning a group to these users.)
Impact:

This can clearly be seen when invoking the service via SOAPui. 
If your service uses a UsernameToken with PasswordDigest policy (like the one I shared above), SOAPui can be used to test it as it can automatically set the required security headers. 
If you look at the SOAPui logs, before applying the passwordDigest policy (e.g. when your services uses a username token with password text based authentication policy like in the default setup), this is how the password component of username token is created:

Unless your service is accessible only over SSL, this means you have passwords flowing around the network in clear text. Most corporate IT policies would I believe specifically forbid letting passwords float around like this and yet, this can often go unnoticed and unaddressed. 

After applying the username token with password digest policy, is how the WS Security headers get created:


Only the client and the server now know what the password is and no one in the middle can see this. 

Tradeoffs
* To enable digest authentication, the server has to store passwords in clear text as per the documentation (for the default authenticator to work - If you have more stringent requirements, it is possible to write your own authenticator that can read passwords from some encrypted credential store). The reason is that with digests, the client (such as SOAPui in the above example) creates a hash of the actual password, creation time and nonce -* the server on its side has to create the same hash for successful authentication and this requires the server to know the clear text password. 
But this is still okay as this can be contained behind strict administrator control. Way better than having clear text passwords travelling over the network. 

Limitations
Digest authentication on its own can help protect your password and with the nonce, it can help prevent replay attacks [3][4]. 
But, it is still vulnerable to man (sic) in the middle (MITM) attacks [3] - which means it is on the whole better to also enable TLS for web service endpoints* when possible. 
(Although, if you suspect MITM attacks from inside an organisation's network, you might have other serious issues!) 

References:
[1] Decisions and choices involved when selecting the appropriate security policy: https://docs.oracle.com/middleware/1221/owsm/security/choose-owsm-policy.htm#OWSMS3988

[2] Setup steps for enabling digest authentication: https://docs.oracle.com/middleware/1212/owsm/OWSMS/configure-owsm-authentication.htm#OWSMS5450

[2] Sections 3.2 & 3.3 : https://www.w3.org/Protocols/rfc2069/rfc2069

[4] https://www.oasis-open.org/committees/download.php/13392/wss-v1.1-spec-pr-UsernameTokenProfile-01.htm

Updates:
16-May-2017: Note about possibility of custom authenticator in the Tradeoffs section (prompted by Jason Scarfe's comment)
1-Jul-2017: Added limitations section