Since the advent of the internet and the subsequent rise of the dot-com boom to the present prominence of Artificial Intelligence, data has been the crucial tool powering and driving the business value and success of enterprises that have recognized and realized its importance. There is a competitive arms race among enterprises of all spheres of business to become data-driven in the way they operate and offer experiences to their customers. Several new ways have been developed to share and communicate information which has led to rapid exponential growth in the volume of data with predictions indicating that 200 zettabytes will be stored in databases all over the world by the year 2025. Apart from the growth in volume, the rise of social media in the past decade has led to the change in nature and quality of data with nearly 95 percent of total data globally being unstructured in nature such as image, audio, video etc. With advances in technology in nearly every aspect of IT, new applications, databases, storage systems, CRMs, ETL tools etc., are emerging which businesses have to take advantage of to maintain their competitive edge.

The process of migrating data, called data migration has become the need of the hour. Data migration comes into action when businesses focus on transitioning their data centers from on-premises to cloud, transferring data when upgrading or changing their database management systems such as from SQL Server to Oracle or from legacy IDS to PostgreSQL, upgrading storage equipment such as from the older and slower HDDs (Hard disk drives) to the newer and much faster SSDs (Solid State Drives), for website consolidation most often as a result of business optimization and reorganization and M&As, conducting server maintenance, physically relocating or upgrading a data center etc.

Data migration

Data Migration in the BFS Sector

Migrating from legacy systems to a better storage system was once considered to be an ambitious activity in the financial services industry but now it is a reality in the industry and done smoothly.
The trend for data migration in the financial services sector is driven by two factors: changing customer requirements and the need to improve performance. Customers in the financial services industry demand a user interface that is flexible as well as scalable. With data in a better advanced storage system or cloud, the Financial sector gets multiple benefits which are:

⦁ Lower Costs: Using a legacy and obsolete database costs businesses a significant amount of money in overhead costs. When a business uses an old system, they may have to install extra applications and systems to speed up normal processes, which increases the cost. An outdated database will with time result in providing low productivity, lost money, and opportunities. Migrating to a better-advanced storage system will serve its purpose with more efficiency. Businesses can ultimately save on the infrastructure and manpower required for ongoing support. For many organizations, legacy systems are seen as holding back the business initiatives and business processes that rely on them. For example, The World Bank cut its platform management costs by $8 million by simply migrating to the cloud-based Office 365. DBS Bank in Singapore lowered its operating costs by 75% by shifting it to the cloud.

⦁ Centralized access: As we know, Financial institutions handle enormous amounts of critical data than any other sector, so, it is required that all data are in one place. Data Migration helps in achieving this goal as it moves all enterprise data to a single, centralized location. 57% of financial institutions worry about losing control of their data, so, if the data is present at a single/centralized location it will eliminate their fear of losing the critical data they have in their system. This shift helps in minimizing the redundant data caused by having numerous platforms containing data from different divisions in a Bank or Finance company. Also if data is in a single place it helps in sharing information and communication in the team. When data is migrated to the cloud then team members can also work simultaneously on shared documents, eliminating the need to send emails.

⦁ Enhanced Product Offerings and Customer Experience: Financial institutions can now drive innovative solutions for their customers more effectively with advanced storage systems or with the cloud, as it allows to upload new products and applications online in no time. This also allows to quickly recognize and respond to customer feedback and behavior, thus, improving Customer Experience.

⦁ Improved Flexibility and Scalability: With the data in advanced systems or in cloud, financial institutions can maintain a highly open and adaptable platform. Migrating data to the cloud also makes scalability easy as the need of relying on physical servers gets eliminated and they can increase the capacity on demand. For example, if a banks system utilizes 100% capacity for only 1 month in a year and the rest of the year it uses only 10% capacity then keeping the same capacity entire year does not make sense, so, this tells us the importance of Scalability feature present in the new advanced systems and cloud.

Guidelines for Cloud Data Migration

Below are the key points to keep in mind during Data Migration. We have considered cloud Data Migration as Financial institutions today focus on migrating their critical data to the cloud.

⦁ Define the goal: The common mistake most companies make when migrating to the cloud is that they fix a vendor and then see what needs or requirements are getting fulfilled. Instead, identify the goal of the business and requirements first and work backwards to the provider. When choosing a vendor after identifying the goal state, make sure to select one who offers the extensibility to meet future business needs.

⦁ Catalog the data: The creation of an intelligent data catalog that provides insights into what data the business has, where it is, what is in current use and how it needs to be protected provides a starting point for the data migration strategy and makes it easier to locate and find specific pieces of data as needed. Identification of high value data becomes quicker and this data can be moved to the cloud data warehouse or cloud data lake. This provides an opportunity to serve data consumers with this new technology to gather feedback on the process.

⦁ Standardization and Data Cleansing: Focus on data quality and governance should be of utmost importance as according to an IBM study, poor data quality results in a $3.1 trillion loss in failed and repeated data migration processes in the US alone. Cleaned data will also reduce the later work needed to migrate the data to cloud, saving time and effort in the future.

⦁ Metadata Management: Metadata Management is key to automating the process of discovering, tagging, relating, and provisioning data into your cloud data warehouse or data lake. Choose a solution that can scan all enterprise systems and collect all technical, business, operational, infrastructure, and usage metadata—from database schemas and glossary terms to volume metrics and user access patterns. In addition, your metadata management solution should be able to curate your metadata, augment it with business context, and infer data lineage and relationships between entities.

cloud data migration

Challenges of Data Migration

Data migration facilitates the adaptation to superior technology whose benefits have long term value, but the operation has some demands and pain points associated with it. Below are a few points that describe the critical nature of data migration and if these are not taken care of, businesses may fail to perform data migration successfully.
⦁ Security of Data: Data has become one of the most crucial assets for any enterprise as it provides a wide array of benefits and any approach to data migration should incorporate processes to ensure absolute security of data. Any compromise to data security is a risk that businesses would not want to deal with. The same applies when this data is migrated to the cloud for instance. Organizations might resist data migration if the transition to another data storage system is not secure. For example, if a cloud infrastructure consists of patchworks of open-source code, it may create and raise concerns about security vulnerabilities. The existence of compliance regulations such as General Data Protection regulation (GDPR) and Health Insurance Portability, Accountability Act (HIPAA), ensure that companies take the issue data security during data migration as an essential first step to consider when evaluating migration tools offered by different vendors and providers. For the BFS sector data security is more crucial as financial services firms are 300 times as likely as other companies to be targeted and attacked as they deal with the enormous amount of sensitive data.

⦁ Lack of proper understanding during the assessment of source data: Poor knowledge of the source data is a problem that has already been observed over several data migration processes across industries. Issues such as spelling errors, duplicates, etc. act as obstacles while ensuring complete and proper data migration. Often, organizations are over-confident and believe that they can configure their data without any complications. However, even a single issue can lead to the failure of the data migration process.

⦁ Vendor Management: It is required for a business to have complete trust in their vendor before completing the data migration process. It may happen that technical issues on the vendor’s side could affect data security. Therefore, it becomes necessary that data migration vendors provide SLAs in order to reduce the concerns. For example, Microsoft has employed 3500 cybersecurity experts and has invested more than $1billion annually in cybersecurity research. This shows the credibility of Microsoft as a company serious about data security while migrating data to the cloud.

⦁ Time-consuming data cleaning process: Data cleaning is the process of altering data in a way suitable for migration. The process focuses on null data values, data relevance, and data duplication which further get corrected using relevant methods. Though data cleaning is needed to increase the data accuracy in a system, it demands so much time which eventually results in reducing the efficiency.
Tools such as Talend Open Studio from Talend and Apex Data Loader from Salesforce enable the ETL team to automate the cleansing process to reduce the cost of time and effort involved. Also, among popular solutions are Open Studio for Data Quality, Data Ladder, SAS Data Quality, Informatica Data Quality, and IBM InfoSphere QualityStage, to name a few.

The Bottom Line
Data migration is often viewed as a necessary evil rather than a value-adding process. And this seems to be the key root of many if not all difficulties. Considering migration an important innovation project worthy of special focus is half the battle won.

Latest posts by Dinesh Chintamreddy (see all)