An Internal Cloud

Merrill Lynch and parent firm Bank of America share insights on launching an internal cloud. By Rob Daly

To reap the cost-saving benefits of cloud computing, Merrill Lynch and parent company Bank of America are in the process of developing an ambitious enterprise file system (EFS) that will power the bank's internal cloud computing initiative.

Although third-party vendors, such as Internet mainstays Amazon and Google, offer cloud computing environments, they have shortcomings, says Jeffrey Birnbaum, chief technology architect and global head of architecture and engineering with Merrill Lynch, who spoke at the Waters Power conference hosted by Waters in April. Amazon's environment requires users to put their applications into virtual machines (VMs) to run, while Google requires that applications running in its cloud be developed using its Google App tools. "The question is: Can you get the same economic benefits within your own enterprise?" he says.

For Merrill Lynch, the idea is to completely externalize its software portfolio from its server environment by placing it into the EFS. Then when the cloud needs computing resources, it can run the applications on any server in the bank's cloud by mounting the EFS onto an individual node.

An important part of this architecture is the "name space," which allows the cloud to locate the right version of the application that needs to be run, says Birnbaum. The bank's EFS directory structure is broken down by vendor, vendor's product, product version, a common directory, and the application's actual code. The caveat with this structure, however, is that all application and operating system (OS) dependencies also need to be contained within the EFS.

Once the EFS is completed, it needs to be cached globally so that it can be used efficiently in local environments. "Pushing something out to the EFS could take a couple of minutes to accomplish depending on bandwidth, but the task will not return until it is synchronized globally," says Birnbaum. The individual EFS caches can vary in size depending on what is being requested locally. "Let's say the entire EFS is 2 terabytes. A cache might just be .5 terabytes since it only consists of what is requested from the cache," he adds.

To create the compute fabric, the bank created "compute pods" that consist of between 500 and 1,000 compute nodes, each running a hypervisor and connected via a Level 2 networking connection using 10 Gbps Ethernet.

"Level 3 networking connections are expensive to deploy and expensive in terms of latency," says Birnbaum. "The Level 2 forwarding latency is usually less than 1 microsecond, while Level 3 forwarding could have up to 5 microseconds worth of latency," he adds.

Each compute pod connects to other compute pods via Level 3 network connections as well as the network-attached storage (NAS) the bank deploys in its "storage pods."

Merrill uses a combination of workload managers and placement engines that determine where to run individual jobs within the cloud.

"You write some description of what the application needs from a processing, bandwidth or storage perspective, and then the workload manager and placement agents' jobs are to place that out on the cloud to their best ability," says Birnbaum. Two open-source tools that can help with this process are the eXtreme Cluster Administration Toolkit (XCAT), which can be used for booting and configuring the individual nodes, and the Moab grid and cloud suite, which can be used as workload managers and placement engines, he adds.

The immediate benefit in adopting this overall architecture will be a dramatic reduction in cost per node, since it will eliminate the current costs associated with supporting redundant architecture. "This is where Google got it right-the idea is to go to much cheaper hardware and assume that it will eventually fail. But if I have 1,000 nodes and one node fails, I still have 999 nodes," says Birnbaum. He expects firms will pay only a third to half of what they are spending now on redundantly designed nodes in the new environment. -->