Hewlett Packard Enterprise (HPE) has announced an expansion to its AI-at-scale offerings with the acquisition of Pachyderm. The latter is a startup that delivers software. based on open-source technology, to automate reproducible machine learning pipelines that target large-scale artificial intelligence (AI) applications.
Reproducing a machine learning pipeline enables use of the same dataset to achieve the same results each time to increase transparency, trustworthiness, and accuracy in predictions while optimising time and resources. It is critical to successful AI-at-scale initiatives, which represent the next revolutionary step in realising AI’s potential to increase the accuracy of predictions and achieve results faster. To attain these outcomes, organisations need to adopt technologies to efficiently build and train larger machine learning models that require a high volume of complex data.
“As AI projects become larger and increasingly involve complex data sets, data scientists will need reproducible AI solutions to efficiently maximise their machine learning initiatives, optimise their infrastructure cost, and ensure data is reliable and safe no matter where they are in their AI journey,” says Justin Hotard, executive vice president and general manager, HPC and AI, at HPE. “Pachyderm’s reproducible AI software augments HPE’s existing AI-at-scale offerings to automate and accelerate AI and unlock greater opportunities in image, video, and text analysis, generative AI, and other emerging large-language-model needs to realise transformative outcomes.”
HPE expands AI at-scale portfolio with addition of Pachyderm
HPE unlocks AI-at-scale opportunities for its customers by bringing together its supercomputing technologies that are foundational for optimised AI infrastructure, and the HPE Machine Learning Development Environment, a machine-learning software that enables users to rapidly develop, iterate, and scale high-quality models from proof-of-concept to production. The combined solution already helps users train more accurate AI models faster, and at scale, on several of the world’s supercomputers that have been purpose-built for demanding AI workloads.
Building on to these solutions, HPE will integrate Pachyderm’s reproducible AI capabilities in one integrated platform to deliver an advanced data-driven pipeline that automatically refines, prepares, tracks, and manages repeatable machine learning processes used throughout the development and training environment.
Delivering an end-to-end machine learning software platform to enable production AI at-scale
AI-at-scale capabilities advance popular use cases involving natural language processing, computer vision, and video and image processing that are growing across industries such as transportation, life sciences, defense, financial services, and manufacturing.
By integrating Pachyderm’s machine learning pipeline capabilities with its existing AI offerings, HPE will enable faster development and deployment of more accurate and performant large-scale AI applications with the following benefits:
- Data lineage – Visibility on the origin of the data and where it moves over time during the machine learning lifecycle and analytics process to easily trace errors back to the root cause.
- Data versioning – Ability to track different versions of data to understand when data was created or changed at any point in time, to increase efficiency in making any changes.
- Efficient incremental data processing – As data changes over time, only incremental data needs to be processed to update AI applications. Pachyderm makes incremental data processing automatic and efficient.
Lockheed Martin deploys HPE’s AI at-scale solutions for mission-critical applications
Lockheed Martin’s AI Factory, an open architecture approach to AI-at-scale, integrates Pachyderm’s software, the HPE Machine Learning Development Environment, and other modular solutions as part of their foundational AI ecosystem. Leveraging these capabilities allows Lockheed Martin to increase trust, maximise performance, and standardise AI technologies across a broad range of contested environments in support of national security missions.
This acquisition builds on HPE’s February 2022 investment in Pachyderm through its venture capital arm, Hewlett Packard Pathfinder, to speed time-to-market for AI innovation at lower data processing and operating costs. The transaction is not subject to any regulatory approvals and is expected to close this month.
Product integration and availability
Pachyderm’s software is available today to integrate with HPE’s existing supercomputing and AI software solutions. Additionally, HPE plans to integrate Pachyderm with upcoming versions of the HPE Machine Learning Development System, which eliminates the complexity and cost to build and train models with a complete, ready-to-use solution.
Follow us and Comment on Twitter @TheEE_io