Here are the setup steps:

Step 1: Create two WAS CE instances – App1Node1Server and App1Node2Server; and ensure both servers are shutdown. with assumption that the base port I start with for node1 is 21371 and for node2 21471; I am using a secure port so my script configures the server to listen at 21374 and 21474 respectively. The rmi ports are 21372 and 21472 respectively.

Step 2: Update config-substitutions.properties with clusterNodeName on both  server; under <server>/var/config folder and set the names to node1 and node2 respectively. I set the RemoteDeployHostname to localhost in my test. But this would probably change when I need to test across multiple VMs/Machines.

Step 3: Update config.xml for App1Node1Server as below, by adding the wadi modules (tomcat6-clustering-wadi, wadi-clustering, farming) and the gbean configuration for farming; it is important to put the NodeInfo and ClusterInfo gbeans as well. In the farming gbean you will need to put the username/password, and host/port, urlPath information for the other node (node2 in this case) that will be part of the cluster

 <module name="org.apache.geronimo.configs/tomcat6-clustering-wadi/2.1.4/car"
   load="true"/>
 <module name="org.apache.geronimo.configs/wadi-clustering/2.1.4/car" load="true"/>
 <module name="org.apache.geronimo.configs/farming/2.1.4/car" load="true">
   <gbean name="NodeInfo">
     <attribute name="name">${clusterNodeName}</attribute> 
   </gbean> 
   <gbean name="ClusterInfo">    
     <attribute name="name">${clusterName}</attribute> 
   </gbean> 
   <gbean name="org.apache.geronimo.configs/farming/2.1.4/car?ServiceModule=
       org.apache.geronimo.configs/farming/2.1.4/car,
       j2eeType=NodeInfo,name=NodeInfo2"
     gbeanInfo="org.apache.geronimo.farm.config.BasicNodeInfo">
     <attribute name="name">node2</attribute>
     <attribute
       propertyEditor="org.apache.geronimo.farm.config.BasicExtendedJMXConnectorInfoEditor"
       name="extendedJMXConnectorInfo">          
       <ns:javabean
          xmlns=""
          xmlns:ns4="http://geronimo.apache.org/xml/ns/attributes-1.2"
          xmlns:ns="http://geronimo.apache.org/xml/ns/deployment/javabean-1.0">            
            <ns:property name="username">node2User</ns:property>
            <ns:property name="password">node2Passwd</ns:property>
            <ns:property name="protocol">rmi</ns:property>
            <ns:property name="host">localhost</ns:property>
            <ns:property name="port">21474</ns:property>
            <ns:property name="urlPath">/jndi/rmi://localhost:21472</ns:property>
            <ns:property name="local">true</ns:property>
       </ns:javabean>    
     </attribute> 
   </gbean>
 </module>

Step 4: Update config.xml for App1Node2Server as below, by adding the wadi modules and the gbean for farming, we the the same as above but change the username/password, and host/port, urlPath information for the other node (node1) that will be part of the cluster

 <module name="org.apache.geronimo.configs/tomcat6-clustering-wadi/2.1.4/car"
   load="true"/>
 <module name="org.apache.geronimo.configs/wadi-clustering/2.1.4/car" load="true"/>
 <module name="org.apache.geronimo.configs/farming/2.1.4/car" load="true">
   <gbean name="NodeInfo">
     <attribute name="name">${clusterNodeName}</attribute> 
   </gbean> 
   <gbean name="ClusterInfo">    
     <attribute name="name">${clusterName}</attribute> 
   </gbean> 
   <gbean name="org.apache.geronimo.configs/farming/2.1.4/car?ServiceModule=
     org.apache.geronimo.configs/farming/2.1.4/car,j2eeType=NodeInfo,name=NodeInfo1"
     gbeanInfo="org.apache.geronimo.farm.config.BasicNodeInfo">
     <attribute name="name">node1</attribute>
     <attribute
       propertyEditor="org.apache.geronimo.farm.config.BasicExtendedJMXConnectorInfoEditor"
       name="extendedJMXConnectorInfo">          
       <ns:javabean
          xmlns=""
          xmlns:ns4="http://geronimo.apache.org/xml/ns/attributes-1.2"
          xmlns:ns="http://geronimo.apache.org/xml/ns/deployment/javabean-1.0">            
            <ns:property name="username">node1User</ns:property>
            <ns:property name="password">node1Passwd</ns:property>
            <ns:property name="protocol">rmi</ns:property>
            <ns:property name="host">localhost</ns:property>
            <ns:property name="port">21374</ns:property>
            <ns:property name="urlPath">/jndi/rmi://localhost:21372</ns:property>
            <ns:property name="local">true</ns:property>
       </ns:javabean>    
     </attribute> 
   </gbean>
 </module>

Step 5: If you restart the servers now – you will notice that the servers are looking for some specific folders that do not exist; Create the <server>/master-repository & <server>/cluster-repository folders under the App1Node1Server and App1Node2Server folders (these folder must be created at the same level as the var folder which holds the config, logs etc.., ); (Since I was setting this up as root and the actual instances where running with another id, I had to ensure the ownership and permissions on the new folders was setup properly to be accessible by the other id with read/write permissions for the user)

Step 6: Now restart both the server instances; and you will notice that one node is added as cluster member on the other node.

The complexity of managing the agents and monitoring the agents them selves is bound to shift some focus of the Operations teams away from the core Operations functions.

There seems to be a need for a consortium that will govern and standardize the agent specifications to perform the plethora of functions in the system which will enable

• Management of the agent footprint
• Self Management of Agents
• SLA governed intrusion
• Efficiencies in maintenance and communications
• security

Hyperic seems like a viable Open Source Option (but requires commercial license to get the enterprise features) and the overarching answer in building a unified solution that is able to touch all aspects of enterprise monitoring and analytic’s that will build efficiencies of scale into the Enterprise IT operations. The capabilities to discover, organize, monitor, alert, control, and visualize the enterprise IT resources – offers a low cost management & monitoring solution that is almost complete.

One thing that may come in the way for this product getting into enterprises is its use of JBoss server. I hope the commercial solution from SpringSource / VMWare (Now) is more generic in nature and deploys to application server of choice or at least a choice of open source app server middleware like tomcat/geronimo.

I am not sure if Hyperic can deliver on all aspects of what I mentioned above but it does seem like it covers a larger surface areas in the open source arena.

• The OS is with the mighty giants : IBM, Oracle, Microsoft, HP; obviously they will never lock out VMWare, but they will not be able to provide the best of the breed enterprise solutions either – which is the only selling point for the OS in the low margin business they are in. The native virtualization solutions in the OS layer are constantly playing catchup or leap froging the VMWare technology. It is only a matter to time before VMWare loses this game or gets acquired.
• VMWare has been a market leader in x86 virtualization.

• The platform of choice for most Java Enterprise Server side applications : Unix/Linux.
  • Spring has definite advantage here
  • VMWare on the other hand is limited to Linux on x86 platforms (at least from what is being used in the context of an enterprise today)
  • Spring.net may on the other hand will probably be in a better state – although I am not too sure if Microsoft Platform needs another Framework. The .Net frame work has come a long way and its tooling – IDEs, Supported Applications, etc is impressive.

• Spring is the most widely used Java Framework.
  • But is easily miss configured -with too many change management layer ills to its credit in an enterprise.
  • Complexity of finding vulnerabilities in application is far greater. Developer who is not careful can potentially leave a hole in the system which can be exploited.
  • Spring Lacks enterprise class security enforcement mechanisms: SLA driven approach with VMWare base, may actually trim the cost for the enterprise (see How do VMware and SpringSource intersect?); VMWare may eventually be leveraged to create black box application snapshots and the scale for the application driven by the SLAs to avoid SA/human intervention where possible – this should address concerns I listed below – but I think data access will be the next challenge to address for such application deployments. Over all the solution still needs to bake well before I woudl suggest it for a business critical application – but I think its not too early to start though.
    • Although Spring has the features via Spring Security (Acegi), it is almost certainly possible that developers will skip this;
    • Most enterprises rely on Application Server infrastructure for security. This can be easily broken into in case of spring if someone has access to the file system – even a casual user can potentially snoop in.
    • Spring provided mechanism’s to mask the passwords stored on disk via PropertyPlaceholderConfigurer – but it has the same issues where a developer will skip the use of such features.
    • Here is a good resource to learn to build better security into applications using Spring; but again the configuration is in a file and is spread across various descriptors that are part of various components in an application. Tracking changes to these descriptors at a system level is a challenge and will most likely be missed in enterprise scenarios of the volume of applications in the data center.
• TCO with VMWare in enterprise increases for the server side implementations. (TCO is relatively lower for desktop application use cases)
  • License Cost variations and complexities mask the ROI; License cost of various technology stacks (software/application) can be very confusing and may actually lower the ROI in some scenarios
  • Putting all your eggs in one basket is not a good idea: Virtualizing a smaller set of machines to improve Utilization, and then host critical applications can cost the business.
  • Performance Cost :
    • Performance test at University of Cambridge
    • Performance test at Clarkson University: This test goes further to showcase the comparison of native, vs xen, vs z/Linux overheads.
  • Cost of Scale:
    • Operationally managing massive number of spring applications in a central / segregated fashion is expensive and manually intensive unlike the commodity enterprise application server platforms
    • The offering of the dm Server from Spring is a change in paradigm for enterprises
    • Running application with a smaller footprint can be a winner at the end of the day – if the complexity of managing the configuration is solved.

Conclusion:

I like the direction dm Server concept was taking at SpringSource – I hope it stays on that vision; and VMWare can play a vital role here to enable the true scale in enterprise with the SpringSource solutions. I will post again if I can get my hands around the solution to see how we can implement in the real world.

I am going to leave the topic open for either SpringSource/VMWare and the rest of the world to comment on and see what everyone else has got to say.

Related Blogs:

Savio’s blog entry presents a pretty interesting perspective on this topic as well; although I am very eager to see IBM come out with a better alternative here as they have all the necessary ingredients for a similar solution Virtualization(z/Linux/AIX ), Open Source Modular Application Server container (IBM WebSphere CE – which is based on Apache Geronimo, Tomcat – was originally GlueCode), IBM WebSphere Virtual Enterprise (SLA driven work load management).

Sam Dean & The Var Guy make note of the competition that Redhat is going to face with this new entrant in the application server space “tc Server” / “dm Server” with a proven virtualization story from VMWare.  But could we say VMWare is tending to follow the path of Redhat and that actually may be good for Redhat as they have the lead into the solution already? May be/May be not. Interestingly, I could not find a single post from Redhat on this topic! I guess that is expected – more you talk about your competition the more publicity they get!

Security – is as expensive as the system complexity, and a factor of compliance measures and the risk tolerance threshold.

It takes a bit of policy and a bunch of processes there after to create complexity – and then comes the control and risk implications of the complexity – which seem to sow the seeds for more policy and then more process; and the viscous cycle continues leading the enterprise into a web of unknown loops and holes in security.

One approaches to security would be to entrust a trusted person(s) the role(s) to manage access points in the systems. If the trusted person(s) breaches the ground rule of security – ‘not to share or delegate the role’ – the system is at risk. Delegation is key to scale – so it becomes imperative to ensure there is sufficient audit trail of who grants access to a specific resource and who gets the access, and who uses the access and then the requirement for someone else to reconcile the access trail to ensure only those authorized were able to access the system and a system to ensure no one tampered with the audit trail. An additional layer of process would be to have access breaches monitored at every level and reported to a reconciliation and approval channel which evaluates the breaches to ensure all the access points are still secure.

If I have managed to successfully confuse the audience here – it is the nature of the problem.

Is there a better option? Break the system into modules (silo’s if you want to call them!) and create asynchronous interfaces (with non binding dynamic contracts), and build bridges between the silo’s (allow trusted communication specific to that module). Each module ( or the silo) should be self contained and reasonably small that a breach into the system would limit the risk, and the bridge will restrict access to authorized interfaces.

Simplifying Systems Architecture(s) is key to addressing complex business requirements and keeping the cost of security in IT solutions low – this is hard to achieve, but not out of reach.  System architecture should consider the options to swap in/out security modules – system security is vulnerable and needs constant up keep, tight coupling can increase the TCO.

Here is a relevant post on security architecture and design, from a series of notes from Gregory Guglielmett; Gregory makes very valid points which seem to miss the radar of many architects when assembling complex business designs & solutions and translating them into efficient IT systems.

Other Relevant posts & News:

• Contractor Returns Money to Pentagon

• Theft at Goldman Sachs and Opinion in New York Times

The user keys in the web site uri. The web server responds by checking if the user request has an established session and finds no associated session. The web server redirects the user to the login screen of a web application on a secure port. The User keys in the userid and password in the login screen and hits the login button and sends the credentials to the web server(uses one way ssl ). The web server uses a Single Sing-on module to pass the credentials to a policy/authentication server. The policy/authentication server responds with a positive result and a  session with user identity is created and tagged with the users request. The web server then forwards/proxies the request over to a application server via an application server plug-in. The application server passes on the request and session info to the application which then reads the session to retrive the user identity and proceeds to authorize the user. The session generated in the interaction with the policy/authentication server can now be trusted by multiple applications in a trusted domain.

Flaw:

The session was not signed/encrypted appropriately.

Result:

An attacker can potentially re-fabricate the session and change the user identity and spoof alternate identities and gain access / authorization.

Resolution:

Ensure the the session is encrypted with an alternate key that is only known to the applications in the domain.

Solution with CA Siteminder:

Install the Web Server Agent for SSO and configure the web server to use the Siteminder agent, also configure the web server to redirect to the login page whcih will direct the login request to the policy server (you will need to configure SSL certificates for the web server to connect to the policy server); then install the Application Server Agent on the host where the application server is running and configure the  application server to use the Identity Asserter from Siteminder ASA package to point to the Siteminder Policy server (you will need to configure SSL certificates for the application server to connect to the policy server).  I will post exact configuration steps in a later post – if you do not see one – ask me !

Why do we need the Web Server and why the two tier authentication?

Web Servers offer capabilities to serve static content – it may be better to serve static content fromt he web server layer to scale better.Web Servers also offer load balancing and fail over capabilities across the application servers. Web Servers are recommended entry points in a DMZ.

If an attacker gains access to such administrative interfaces they will have full control of the business and potentially can gain access to critical data, services & other integrated systems.

Java EE middle-ware addresses the problem spaces of SOA, EAI and forms the basis for many evolving BPM and ESB infrastructures. This level of dependency in the enterprise on the Java EE technology stacks in general and on the application server containers is more so a reason we need to carefully and completely secure the infrastructure.

To mitigate this risk is a challenge – a few  simple measures from operational teams can effectively contain the risk and ensure controlled access where necessary.

The first line of defense in security is always the Network Perimeter – access to administrative consoles should be confined to a predefined network segment.  Strict monitoring must be enforced for the access traffic in this network segment – and any anomalies should be tracked and reported for immediate action.

This can be achieved by deploying the administrative servers or deployment managers into a administrative network segment – which will be a partition of the production network. Access to the administrative server console via the browsers must be permitted only via known management desktops in a terminal server/VM farm.

Once the network is secured, the OS domain level security can be leveraged to access the terminal server / VM farm that has administrative console access.

Lastly the Administrative Consoles themselves need to be secured with in the products with middleware infrastructure  level security using a Federated / Standalone LDAP user registry.

There must be  regular reporting & review, and user management policy enforcement to ensure the access rights for specific users granted the entitlements to access the 3 layers of the security is constantly monitored for correctness and breaches.

It would be interesting to learn about other experiences with securing enterprise middle-ware infrastructure. So please send me your comments.

In a recent post to his Alert Box Dr Nielson suggests unmasking the password fields in the UI design.

Usability suffers when users type in passwords and the only feedback they get is a row of bullets. Typically, masking passwords doesn’t even increase security, but it does cost you business due to login failures.

It’s time to show most passwords in clear text as users type them. Providing feedback and visualizing the system’s status have always been among the most basic usability principles. Showing undifferentiated bullets while users enter complex codes definitely fails to comply.

With all due respect,  I have to disagree here unless all/most devices allow an alternate mechanism of registrations and authentication. It may be feasible if all security standards mandate biometric identification – but are we there yet! May be this is a random thought that would lead us into the next generation ID which is fool proof and safe.

We have weaved ourselves into protecting the passwords to an extreme – and invented ways to secure the passwords, authentication, two factor authentication, identity assertions, password vaults, HSM etc and built security mechanism’s so tightly integrated into the applications today that many security policies are written based on the assumption that password needs to protected from Social Engineering and other forms of hacking . Undoing this would be an interesting balancing act.

My experience has been that we can deliver increased if not similar server utilization numbers, on a non-virtualized hardware and hosts application server run time. A JVM boundary in general is the maximum heap allocated to it. The CPU utilization of many applications deployed to an application server (80:20 rule) tends to be on the low side but applications directly benefit from the built-in integrations for transaction management, clustering/fail-over mechanics, asynchronous & event services, security model, and many more other features are built into the container. For those applications that need a lower baggage the containers are now embracing the Modular architecture from OSGi, which can considerably reduce the footprint and feature set and also make the features/modules more plug n play in nature.

Given the memory that we can accommodate on the blades/the boards has been consistently growing – it seems fairly obvious that we can fit in more and more low CPU utilization applications into a piece of hardware. Performance in virtulaized infrastcture is a long debated topic and may be a moot point in such shared infrastructure model for application servers.

Here is an article from Adam Messinger & Mike Piech (WebLogic) that substantiates the idea

Why an Application Grid?
— Application servers, those dependable workhorses that run most enterprise Java applications, are rarely a hot topic of conversation these days. As a technology category, the application server appears to be fairly “established” and that the focus has moved elsewhere in the stack, but appearances can be deceiving.

As far as I remember, IBM defined the original Application Virtualization / Grid concept in late 2005 /early 2006 time frame when they introduces the concept of WebSphere Extended Deployment. But this idea really transformed into defining the strategy for Application Infrastructure virtualization with the release of IBM WebSphere Virtual Enterprise product. IBM WebSphere VE is far ahead in the game on identifying the key areas of concern in this space and delivers considerable amount of functionality. IBM definitely has a leg up on the Work load management architecture in the WebSphere VE product which leverages a lot of expertise form the Mainframe architectures. The key areas that are enterprise worthy with IBM WebSphere VE include its ability to for policy based workload management, Application health management, dynamic ability to scale (need Tivoli Provisioning and Orchestration product stacks to support this use case in a realistic sense), automated operations management. IBM cliams the ability to do application version/edition management in VE but I see a lot of gaps with it where it cannot address some of the application architecture complexities - I think there is lot more use cases and gaps that need to be filled with application editioning.

The story around WebLogic virtualization got very weak after the recent Oracle acquisition. Oracle Weblogic (as against BEA WebLogic) seemed to have fallen behind on the roadmap, although I believe the WebLogic product itself is well positioned to sustain/support this use case of application virtualization from the usability point (developer & operator friendly), it sure can use a face-lift in its enterprise readiness (address control & compliance risk areas).

Isolating applications and mitigating risk in a shared infrastructure model is a complex topic that I will discuss in a future blog. But I wanted to note that all said and done, there will be some downside of using/setting up application grids. I would like to know if anyone else out there has any recommendations.

jt859v4gz6