Tuesday, October 25, 2016

Managing shared metadata (MDS) in a CI environment

Goals and Summary:
* Package shared metadata in a SOA environment and make it widely distributable (SOA MDS [2], Servicebus, maven artifact repository) 
* Associated sample: https://github.com/jvsingh/SOATestingWithCitrus/tree/develop/shared-metadata  
* Key command (if you use the associated pom file) 
mvn deploy com.oracle.soa.plugin:oracle-soa-plugin:12.2.1-0-0:deploy -Dpassword=*****

       
Background:
Having worked on a wide range of projects, I came to the realisation that SOA can mean vastly different things in different places.
It can be about implementing the foundational service oriented architectural principles or it can be simply about using a tool or technology with SOA in its name- just like any other programming language.
In a mature SOA environment, the shared metadata contains valuable artefacts that provide the foundation – subject to design, it contains the canonical information model of the enterprise (in the form of business/entity objects) and the various organisational API interfaces (service interfaces and messages).
In the Fusion middleware world, this pattern is easily implemented via the MDS – a set of services that allows storage and retrieval of many types of shared resources such as WSDLs, XSDs, DVMs, reusable XQuery or XSL transformations etc. Within SOA composites, these are then accessible via the oramds:/ prefix
To take this one step further, we also can deploy the same copy of shared artefacts into the Oracle Service bus as a servicebus project so even the OSB services can access these without requiring local copies scattered everywhere.  A great benefit of deploying this content to the OSB is that you get some basic sanity checking of these artefacts for free (e.g. the OSB is a bit strict about unresolvable XSD imports in WSDLs – this kind of thing is highlighted at design-time only if you use a professional XML editor and not regular JDeveloper[1], which is what most FMW developers might commonly use)
There are some key principles here:
  • ·         Within an organisation, are service callers and called services able to access the same copies of schemas and WSDLs? Or are there copies floating all over in every project? This kind of thing invariably leads to ‘slightly’ different copies of the same schema and basically is a recipe for mess.

(Of course, when consuming ‘external’ services, we probably do want to save a specific version of their interface locally as that forms our ‘contract’)
  • ·         Are projects neat self contained units that interact with the ‘external world’ via well defined interfaces, or is there a complex web of cross-dependencies, deeply nested dependencies and even circular-dependencies with projects referencing each other? Shared metadata helps avoid these situations by providing both callers and implementers the same reference data model and interfaces.
  • ·         Is there any form of assurance or validation of the shared artefacts? Are the WSDLs and XSDs well-formed and valid? To be specific, are any schema errors flagged up regularly as part of a continuous build (rather than being detected much later when multiple such errors have accumulated?)
  • ·         Is the MDS being built and deployed as a single, versioned unit or do individuals simply zip up groups of “files” and promoting them across environments?

On the last point, I think it is important to consider the shared metadata as a single deployable unit that can be version controlled, tagged, built with identifiable versions,  validated, deployed, promoted, in the same way as a SOA composite or a ServiceBus project is a single deployable unit. (yes, I know you can create an *.sbar archive with only the  ‘files’ you changed within a project, but this kind of approach is completely contrary to practices that promote continuous integration and delivery. You essentially end up tracking individual files rather than treating a ‘project’ as a unit of deployment. )

Now, coming to the build and deployment of MDS, we use the approach of zipping these up (note the build section and packaging in my MDS pom.xml and then deploying the artefact using using maven the oracle-soa-plugin (specifying the sarFile property as apps.jar)

mvn deploy com.oracle.soa.plugin:oracle-soa-plugin:12.2.1-0-0:deploy -Dpassword=

Outcome:

·         As seen above, the MDS bundle is deployed to the SOA runtime.
·         It is also deployed to the maven repository configured in the distributionManagement section (this could be any repository such as nexus)
Note that since I call the oracle-soa-plugin directly in the maven command, I don’t need to explicitly configure it in the pom (I would have to do that only if I was piggybacking the soa deploy on top of one of the maven phases but here I specifically want “mvn deploy” to validate and then deploy the artefact to my maven repo. I specifically want my MDS deployment to the runtime MDS to happen separately).I have only configured some of the properties required by oracle-soa-plugin in the pom to keep my deploy command concise.
I further make it a point to make sure that the artefact produced by this last step is also deployed to the local and internal maven repositories (such as nexus). For this example, I have used a simple distributionManagement section in my MDS pom that installs the shared-metadata bundle into my local maven repository. This simple step ensures that ANY other consumer in the organisation is able to consume this metadata (e.g. a standalone Java web service or application that needs to call an internal web service)
In subsequent posts, I will add a Java consumer that can simply use the shared metadata as a dependency and consume the common repository of
In the brave new polyglot world of Oracle Application container cloud, this can in theory be ANY consumer – even PHP or python!

Coming up:
* Adding more validation for shared-metadata in CI



References and footnotes


[1] Some times unresolved types only come to light in JDeveloper if you ctrl+click on it. I think this flexibility might be by design to keep things simple for beginners perhaps but this is only an opinion. 
[2] Teams might use various approaches for this. Here is one of the earlier posts that also partly address MDS deployment via maven with a conceptually similar approach (create a zip then deploy using oracle-soa-plugin): http://weblog.redrock-it.nl/?p=740
My approach, though, avoids the need for the assembly plugin and its associated XML assembly descriptor to create the zip beforehand. The benefit is that the primary artifact produced by the main build is what maven also automatically pushes to the distribution repo (such as nexus) in the 'deploy' phase.

No comments: