Posts Tagged ‘Apache Spark’

z System-Power-Storage Still Live at IBM

January 5, 2017

A mid-December briefing by Tom Rosamilia, SVP, IBM Systems, reassured some that IBM wasn’t putting its systems and platforms on the backburner after racking up financial quarterly losses for years. Expect new IBM systems in 2017. A few days later IBM announced that Japan-based APLUS Co., Ltd., which operates credit card and settlement service businesses, selected IBM LinuxONE as its mission-critical system for credit card payment processing. Hooray!

linuxone-emperor-2

LinuxONE’s security and industry-leading performance will ensure APLUS achieves its operational objectives as online commerce heats up and companies rely on cloud applications to draw and retain customers. Especially in Japan, where online and mobile shopping has become increasingly popular, the use of credit cards has grown, with more than 66 percent of consumers choosing that method for conducting online transactions. And with 80 percent enterprise hybrid cloud adoption predicted by 2017, APLUS is well positioned to connect cloud transactions leveraging LinuxONE. Throw in IBM’s expansion of blockchain capabilities and the APLUS move looks even smarter.

With the growth of international visitors spending money, IBM notes, and the emergence of FinTech firms in Japan have led to a diversification of payment methods the local financial industry struggles to respond. APLUS, which issues well-known credit cards such as T Card Plus, plans to offer leading-edge financial services by merging groups to achieve lean operations and improved productivity and efficiency. Choosing to update its credit card payment system with LinuxONE infrastructure, APLUS will benefit from an advanced IT environment to support its business growth by helping provide near-constant uptime. In addition to updating its server architecture, APLUS has deployed IBM storage to manage mission-critical data, the IBM DS8880 mainframe-attached storage that delivers integration with IBM z Systems and LinuxONE environments.

LinuxONE, however, was one part of the IBM Systems story Rosamilia set out to tell.  There also is the z13s, for encrypted hybrid clouds and the z/OS platform for Apache Spark data analytics and even more secure cloud services via blockchain on LinuxONE, by way of Bluemix or on premises.

z/OS will get attention in 2017 too. “z/OS is the best damn OLTP system in the world,” declared Rosamilia. He went on to imply that enhancements and upgrades to key z systems were coming in 2017, especially CICS, IMS, and a new release of DB2. Watch for new announcements coming soon as IBM tries to push z platform performance and capacity for z/OS and OLTP.

Rosamilia also talked up the POWER story. Specifically, Google and Rackspace have been developing OpenPOWER systems for the Open Compute Project.  New POWER LC servers running POWER8 and the NVIDIA NVLink accelerator, more innovations through the OpenCAPI Consortium, and the team of IBM and Nvidia to deliver PowerAI, part of IBM’s cognitive efforts.

As much as Rosamilia may have wanted to talk about platforms and systems IBM continues to avoid using terms like systems and platforms. So Rosamilia’s real intent was to discuss z and Power in conjunction with IBM’s strategic initiatives.  Remember these: cloud, big data, mobile, analytics. Lately, it seems, those initiatives have been culled down to cloud, hybrid cloud, and cognitive systems.

IBM’s current message is that IT innovation no longer comes from just the processor. Instead, it comes through scaling performance by workload and sustaining leadership through ecosystem partnerships.  We’ve already seen some of the fruits of that innovation through the Power community. Would be nice to see some of that coming to the z too, maybe through the open mainframe project. But that isn’t about z/0S. Any boost in CICS, DB2, and IMS will have to come from the core z team. The open mainframe project is about Linux on z.

The first glimpse we had of this came last spring in a system dubbed Minsky, which was described back then by commentator Timothy Prickett Morgan. With the Minsky machine, IBM is using NVLink ports on the updated Power8 CPU, which was shown in April at the OpenPower Summit and is making its debut in systems actually manufactured by ODM Wistron and rebadged, sold, and supported by IBM. The NVLink ports are bundled up in a quad to deliver 80 GB/sec bandwidth between a pair of GPUs and between each GPU and the updated Power8 CPU.

The IBM version, Morgan describes, aims to create a very brawny node with very tight coupling of GPUs and CPUs so they can better share memory, have fewer overall GPUs, and more bandwidth between the compute elements. IBM is aiming Minsky at HPC workloads, according to Morgan, but there is no reason it cannot be used for deep learning or even accelerated databases.

Is this where today’s z data center managers want to go?  No one is likely to spurn more performance, especially if it is accompanied with a price/performance improvement.  Whether rank-and-file z data centers are queueing up for AI or cognitive workloads will have to be seen. The sheer volume and scale of expected activity, however, will require some form of automated intelligent assist.

DancingDinosaur is Alan Radding, a veteran information technology analyst, writer, and ghost-writer. Please follow DancingDinosaur on Twitter, @mainframeblog. See more of his IT writing at technologywriter.com and here

IBM LinuxONE Can Uberize x86-Based IT

November 13, 2015

Uberization—industry disruption caused by an unlikely competitor—emerged as a dominant concern of C-suite executives in a recently announced IBM-Institute of Business Value study. According to the study, the percentage of C-suite leaders who expect to contend with competition from outside their industry increased from 43% in 2013 to 54% today.

IBM Csuite Study_Tiles_10_30_2 competition data

These competitors, future Ubers, aren’t just resulting from new permutations of old industries; they also are coming from digital invaders with totally different business models. Consider IBM LinuxONE, a powerful open source Linux z13 mainframe supported by two open communities, the Open Mainframe Project and the Linux Foundation. For the typical mass market Linux shop, usually an x86-based data center, LinuxONE can deliver a standard Linux distribution with both KVM and Ubuntu as part of a new pricing model that offers a pay-per-use option in the form of a fixed monthly payment with costs scaling up or down based on usage. It also offers per-core pricing with software licenses for designated cores.

Talk about disruptive; plus it brings scalability, reliability, high performance, and rock-solid security of the latest mainframe. LinuxONE can handle 8000 virtual servers in a single system, tens of thousands of containers. Try doing that with an x86 machine or even a dozen.

Customers of traditional taxi companies or guests at conventional hotels have had to rethink their transportation or accommodation options in the face of Uberization and the arrival of other disruptive alternatives like Airbnb. So too, x86 platform shops will have to rethink their technology platform options. On either a per-workload basis or a total cost of ownership (TCO) basis, the mainframe has been cost competitive for years. Now with the Uberization of the Linux platform by LinuxONE and IBM’s latest pricing options for it, the time to rethink an x86 platform strategy clearly has arrived. Many long-held misconceptions about the mainframe will have to be dropped or, at least, updated.

The biggest risk to businesses used to come from a new rival with a better or cheaper offering, making it relatively simple to alter strategies. Today, entrenched players are being threatened by new entrants with completely different business models, as well as smaller, more agile players unencumbered by legacy infrastructure. Except for the part of being smaller, IBM’s LinuxONE definitely meets the criteria as a threatening disruptive entrant in the Linux platform space.

IBM even is bring new business models to the effort too, including hybrid cloud and a services-driven approach as well as its new pricing. How about renting a LinuxONE mainframe short term? You can with one of IBM’s new pricing options: just rent a LinuxONE machine monthly with no upfront payment.  At the end of the 36-month rental (can return the hardware after 1 year) you choose to return, buy, or replace. Try that with enterprise-class x86 machines.

The introduction of support for both KVM and Ubuntu on the z platform opens even more possibilities. With the LinuxONE announcement Ubuntu has been elevated to a fully z-supported Linux distribution. Together IBM and Canonical are bringing a distribution of Linux incorporating Ubuntu’s scale-out and cloud expertise on the IBM z Systems platform, further expanding the reach of both. Ubuntu combined with KVM should make LinuxONE very attractive for OpenStack-based hybrid cloud computing that may involve thousands of VMs and containers. And don’t forget a broader range of tools, including an expanded set of open-source and industry tools and software, including Apache Spark, Node.js, MongoDB, MariaDB, PostgreSQL, Chef and Coker.

Deon Newman, VP of Marketing for IBM z Systems, can recite the LinuxONE scalability stats off the top of his head: The entry-level, single-frame LinuxONE server, named Rockhopper, starts at 80 virtual Linux machines, and hundreds and hundreds of containers while the high-end double-frame server, Emperor, features six IFLs that support up to 350 virtual machines and can scale all the way to 8,000 virtual machines. On the Emperor server, you can literally have hundreds of thousands of containers on a single platform. Newman deliberately emphasizes that LinuxONE machines are servers.  x86 server users take note. LinuxONE definitely is not your father’s mainframe.

In the latest C-suite study all C-suite executives—regardless of role—identified for the first time technology as the most important external force impacting their enterprise. These executives believe cloud computing, mobile solutions, the Internet of Things, and cognitive computing are the technologies most likely to revolutionize or Uberize their business.

DancingDinosaur is Alan Radding, a veteran information technology analyst and writer. Please follow DancingDinosaur on Twitter, @mainframeblog. See more of his IT writing at technologywriter.com and here.

 

 

 

 

 

Syncsort Brings z System Integration Software to Open Source Tools

October 13, 2015

In a series of announcements last month, Syncsort integrated its DMX-h data integration software with Apache Kafka, an open distributed messaging system. This will enable mainframe shops to tap DMX-h’s easy-to-use GUI to subscribe, transform, enrich, and distribute enterprise-wide data for real-time Kafka messaging.

Spark graphic

Courtesy of IBM

Syncsort also delivered an open source contribution of an IBM z Systems mainframe connector that makes mainframe data available to the Apache Spark open-source analytics platform. Not stopping there, Syncsort is integrating the Intelligent Execution capabilities of its DMX data integration product suite with Apache Spark too. Intelligent Execution allows users to visually design data transformations once and then run them anywhere – across Hadoop, MapReduce, Spark, Linux, Windows, or Unix, on premise or in the cloud.

Said Tendü Yoğurtçu, General Manager of Syncsort’s big data business, in the latest announcement: “We are seeing increased demand for real-time analytics in industries such as healthcare, financial services, retail, and telecommunications.” With these announcements, Syncsort sees itself delivering the next generation streaming ETL and Internet of Things data integration platform.

Of course, the Syncsort offer should be unnecessary for most z System users except those that are long term Syncsort shops or are enamored of Syncsort’s GUI.  IBM already  offers Spark native on z/OS and Linux on z so there is no additional cost.  BTW, Syncsort itself was just acquired. What happens with its various products remains to be seen.

Still  IBM has been on a 12-year journey to expand mainframe workloads—Linux to Hadoop and Spark and beyond—the company has been urging mainframe shops as fast as fast as possible to become fully engaged in big data, open source, and more. The Syncsort announcements come at a precipitous time; mainframe data centers can more easily participate in the hottest use cases: real-time data analytics, streaming data analytics across diverse data sources, and more at the time when the need for such analytics is increasing.

Apache Spark and some of these other technologies should already be a bit familiar to z System data centers; Apache Kafka will be less familiar. DancingDinosaur noted Spark and others here, when LinuxOne was introduced.

To refresh, Apache Spark consists of a fast engine for large-scale data processing that provides over 80 high-level operators to make it easy to build parallel apps or use them interactively from the Scala, Python, and R shells. It also offers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application.  As noted above Syncsort offers an open source version of the IBM z Systems mainframe connector that makes mainframe data available to the Apache Spark open-source analytics platform.

Spark already has emerged as one of the most active big data open source projects, initially as a fast memory-optimized processing engine for machine learning and now as the single compute platform for all types of workloads including real-time data processing, interactive queries, social graph analysis, and others. Given Spark’s success, there is a growing need to securely access data from a diverse set of sources, including mainframes, and to transform the data into a format that is easily understandable by Spark.

Apache Kafka, essentially an enterprise service bus, is less widely known. Apache Kafka brings a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system. Kafka is often used in place of traditional message brokers like JMS and AMQP because of its higher throughput, reliability and replication. Syncsort has integrated its data integration software with Apache Kafka’s distributed messaging system to enable users to leverage DMX-h’s GUI as part of an effort to subscribe, transform, enrich, and distribute enterprise-wide data for real-time Kafka messaging.

According to Matei Zaharia, creator of Apache Spark and co-founder & CTO of Databricks: “Organizations look to Spark to enable a variety of use cases, including streaming data analytics across diverse data sources”.  He continues: “Syncsort has recognized the importance of Spark in the big data ecosystem for real-time streaming applications and is focused on making it easy to bring diverse data sets into Spark.” IBM certainly recognizes this too, and the z System is the right platform for making all of this happen.

DancingDinosaur is Alan Radding, a veteran IT analyst and writer. Please follow DancingDinosaur on Twitter, @mainframeblog. See more of his IT writing at technologywriter.com and here.

developerWorks Open Reinforces IBM’s Commitment to Open Source Cloud

July 30, 2015

Maybe IBM’s decades-long legacy of proprietary platforms and systems makes people a little hesitant to fully embrace its open source initiatives. Still, IBM has been supporting Linux on the z System for over a decade, the Eclipse initiative for as long or longer, and gives no sign of getting ready to pull the plug on any of its open source initiatives.

Rise of Open Source Linux and OpenStackCourtesy of IBM (click to enlarge)

Or take Bluemix, an implementation of IBM’s Open Cloud Architecture based on Cloud Foundry, an open source Platform as a Service (PaaS) initiative. And the company only gets more open source by the day. Just last week IBM continued to pour more open source components into Bluemix. It announced developerWorks Open, a cloud-based environment for developers to not only download open sourced IBM code but also have access to blogs, videos, tools and techniques to accelerate their own efforts on behalf of clients.

The current model of development in the open source community, according to IBM, lacks a strategic focus on business requirements. To address this IBM is launching a set of projects in industries like healthcare, mobile, retail, insurance, and banking that ensure a strategic business focus and address real-world business challenges.

The creation of developerWorks Open, notes IBM, comes at an important time for cloud developers. Organizations are wrestling with getting the most out of their multiple clouds environments. For instance, a developer building cloud applications on Bluemix for an insurance accident claim system likely will require storing videos and repair photos for an insurance accident claim application. The developer may have difficulty making a storage choice and then implementing it. But developerWorks Open helps simplify that choice and provides the reassurance that the choice has industry support.

To that end IBM is offering a broad range of technologies for developerWorks Open that aim to help bridge this perceived development gap and remove the obstacles that inhibit developers from turning open source code into sustainable applications that solve real business issues. IBM will also offer these services on its programming platform for cloud software development, Bluemix. The goal is to remove the obstacles that hinder developers from turning open source code into sustainable enterprise-caliber applications that solve real business issues.

For that reason the company will open source a number of apps from its MobileFirst portfolio (MobileFirst was recently cited by Gartner as a leader in its Magic Quadrant mobile application development segment) that will assist developers in the following markets:

  • IBM Ready App for Healthcare tracks patient progress for at-home physical therapy programs via mobile device.
  • IBM Ready App for Retail personalizes and reshapes the specialty retail store shopping experience through direct line of communication
  • IBM Ready App for Insurance improves the relationship between homeowners and insurers and uses Internet of Things sensors to synch home with utilities.
  • IBM Ready App for Banking helps financial institution’s address the mobile needs business owners and attract prospects.

Additionally, IBM is open sourcing several analytics technologies including:

  • Activity Streams provides developers with a standard model and encoding format for describing how users engage with both the application and with one another.
  • Agentless System Crawler offers a unified cloud monitoring and analytics framework that enables visibility into all types of cloud platforms and runtimes
  • IBM Analytics for Apache Spark puts the full analytics power and capabilities of Spark at the developers fingertips. (Beta now available on Bluemix.)

IBM will also continue to open source cloud data services, including IBM Object Storage on Bluemix Service Broker, which can be used to integrate OpenStack Swift with Cloud Foundry to enable fast access to cloud data without needing to know where the data is stored.

The introduction of developerWorks Open comes at a time when organizations are starting to realize that their software and apps increasingly are their products, especially cloud, mobile, and collaboration apps, and they need a fast and efficient way to build and update them. In other cases, IBM notes, organizations are wrestling with getting the most out of their multiple clouds environments.

IBM is committed to open source; there is no going back. Company executives see it as the foundation of innovative application development in the cloud.  “With developerWorks Open we are open sourcing additional IBM innovations that we feel have the potential to grow the community and ecosystem and eventually become established technologies,” declared IBM Vice President of Cloud Architecture and Technology Dr. Angel Diaz recently.

Currently, IBM participates in and contributes to more than 150 open source projects. These projects include Spark, OpenStack, Cloud Foundry, Open Contain Project, Node.js, CouchDb, Linux, Eclipse and an already established relationship with Apache. As IBM notes, open source projects increase the skills and knowledge base around the company’s software product set. developerWorks Open is the next step in IBM’s strategy to help businesses create, use, and innovate around cloud computing systems. Coming right behind is a similar community initiative for IoT development. Stay tuned.

DancingDinosaur is Alan Radding, a veteran IT analyst and writer. Please follow DancingDinosaur on Twitter, @mainframeblog. See more of his IT writing at technologywriter.com and here.

IBM Continues Open Source Commitment with Apache Spark

June 18, 2015

If anyone believes IBM’s commitment to open source is a passing fad, forget it. IBM has invested billions in Linux, open Power through the Open Power Foundation, and more. Its latest is the announcement of a major commitment to Apache Spark, a fast open source and general cluster computing system for big data.

spark VGN8668

Courtesy of IBM: developers work with Spark at Galvanize Hackathon

As IBM sees it, Spark brings essential advances to large-scale data processing. Specifically, it dramatically improves the performance of data dependent-apps and is expected to play a big role in the Internet of Things (IoT). In addition, it radically simplifies the process of developing intelligent apps, which are fueled by data. It does so by providing high-level APIs in Scala, Java, and Python, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Spark Streaming for stream processing.

IBM is contributing its breakthrough IBM SystemML machine learning technology to the Spark open source ecosystem. Spark brings essential advances to large-scale data processing, such as improvements in the performance of data dependent apps. It also radically simplifies the process of developing intelligent apps, which are fueled by data. But maybe the biggest advantage is that it can handle data coming from multiple, disparate sources.

What IBM likes in Spark is that it’s agile, fast, and easy to use. It also likes it being open source, which ensures it is improved continuously by a worldwide community. That’s also some of the main reasons mainframe and Power Systems data centers should pay attention to Spark.  Spark will make it easier to connect applications to data residing in your data center. If you haven’t yet noticed an uptick in mobile transactions coming into your data center, they will be coming. These benefit from Spark. And if you look out just a year or two, expect to see IoT applications adding to and needing to combine all sorts of data, much of it ending up on the mainframe or Power System in one form or another. So make sure Spark is on your radar screen.

Over the course of the next few months, IBM scientists and engineers will work with the Apache Spark open community to accelerate access to advanced machine learning capabilities and help drive speed-to-innovation in the development of smart business apps. By contributing SystemML, IBM hopes data scientists iterate faster to address the changing needs of business and to enable a growing ecosystem of app developers who will apply deep intelligence to everything.

To ensure that happens, IBM will commit more than 3,500 researchers and developers to work on Spark-related projects at more than a dozen labs worldwide, and open a Spark Technology Center in San Francisco for the Data Science and Developer community to foster design-led innovation in intelligent applications. IBM also aims to educate more than 1 million data scientists and data engineers on Spark through extensive partnerships with AMPLab, DataCamp, MetiStream, Galvanize, and Big Data University MOOC (Massive Open Online Course).

Of course, Spark isn’t going to be the end of tools to expedite the latest app dev. With IoT just beginning to gain widespread interest expect a flood of tools to expedite developing IoT data-intensive applications and more tools to facilitate connecting all these coming connected devices, estimated to number in the tens of billions within a few years.

DancingDinosaur applauds IBM’s decade-plus commitment to open source and its willingness to put real money and real code behind it. That means the IBM z System mainframe, the POWER platform, Linux, and the rest will be around for some time. That’s good; DancingDinosaur is not quite ready to retire.

DancingDinosaur is Alan Radding, a veteran IT analyst and writer. Please follow DancingDinosaur on Twitter, @mainframeblog. See more of his IT writing on Technologywriter.com and here.


%d bloggers like this: