IT Infra and Virtualization Services for Financial Services Industry

Client Information
The client is a global leader in facilitating e-commerce across 6.5 million locations and over 3,750 financial institutions in more than 90 countries. The client and its associates are specialized in servicing businesses of all sizes including start-ups, and amount to more than USD 2 million transactions per year.

Client Requirements
As per the client’s requirement, Veritis team of experts offered managed IT Infrastructure and Virtualization consulting services.

This project involved building, security, patching and updating of countless LINUX, AIX and attached subsystems. These subsystems support existing systems and are responsible for leading technical aspects of the project in terms of building new systems and also provide technical review for the go-live checklists across UNIX servers.

The client’s UNIX server team required the assistance of personnel with special skill-set in UNIX server administration, automation scripting skills in Shell or Python languages and experience in handling the project, which would prove critical and sensitive to financial transactions provided by our experts.

The project prospects involved:

• Ensuring adherence to established technology standards and consulting with senior leadership to deploy technology solutions
• Supporting existing systems and being responsible for leading technical aspects of the project
• Installing and supporting Red Hat, CentOS, Oracle Linux and/or IBM AIX operating systems
• Performing additional job duties as required
• Leading complex projects and issues within the functional technical area and cross-functional area
• Working independently with IT functional units and business units
• Recommending strategies for technical aspects of projects as well as broad system improvements, with a focus on AIX and Linux Systems

Veritis Approach
Veritis team of experts assisted the client with IT Infrastructure and Virtualization services meeting the right technology standards. The team eliminated the traditional manual practice of validating server configurations by automating the process and created new standards, procedures and processes for builds and documents among other infrastructure-related tasks.

Client Challenge Areas
Before Veritis took up the project, the client’s server admin team had a manual validation process for server configurations in order to ensure that the systems comply with the standards. The approach involved comparing configurations on the server with a document that has all the server-specific information. The process was a painful task, where more than 40,000 servers required manual checking.

With Veritis intervention and the team’s expertise, the whole process was automated using bash and KSH scripting skills.

Key Responsibilities Held

• Installed, configured and performed day-to-day administration to solve system-related issues and user assistance activities on Red Hat Linux 4.x, 5.x, 6.x, 7.x, Open SuSE Linux 9.x, 10.x, 11.x. 12.x, CentOS Linux 6.x, 7.x, Oracle Linux 6.x, 7.x, IBM AIX 6.1/7.1 servers in both physical and virtual system environments
• Created new standards, processes and procedures for builds and documents. Provided documentation for new builds to be created in the environment
• Performed administration tasks like setup, maintenance and management of user accounts and groups, and allocated quotas to users as well as system startup and shutdown
• Installed patches, packages related to hardware, vendor software and installed third-party software for system customization
• Monitored the performance of the UNIX systems using PS, prstat, vmstat, netstat, iostat, and RPCinfo to check the CPU utilization, bottlenecks of I/O devices, memory usage and network traffic
• Provided drill down reporting for application teams to be used in monitoring client’s application resource usage/performance
• Worked closely with database administration staff to ensure optimal performance of databases and maintained development applications and databases
• Monitored system resources, logs, disk usage, scheduled and restored backups using Nagios and VERITAS Netbackup
• Diagnosed, troubleshoot and resolved hardware, software and system problems. Quickly arranged for repair in occasion of hardware failure
• Configured High Availability (HA) solutions to include multiple cluster configurations and support
• Implemented network security measures to protect data, software and hardware
• Worked closely with Hardware vendor (IBM, HP, Dell, and Cisco) support to detect and resolve any issues related to the environment
• Performed day-to-day health check and resolved any errors during the health check
• Maintained logs related to system functions, as well as maintained and repaired records
• Assisted in critical outage situations to isolate the root cause, put a temporary fix in place, and recommended/implemented long-term corrective action
• Network troubleshoot using NDD, traceroute, netstat, ifconfig and snoop, among others
• Administered, maintained and troubleshoot EMC Clariion arrays by creating volumes, LUNs and implementing different RAID levels
• Implemented enterprise storage solutions SAN (created VGs and LVs on SAN disks), NAS (mounted and shared NAS devices for servers) and Logical Volume Management (LVM)
• Configured and scheduled various automated jobs for backups, database, and proprietary jobs using CRON, Autosys and BMC Control-M and troubleshoot daily job problems
• Performed system backup and recovery from tapes using ufsdump, ufsrestore, tar and CPIO
• Performed restores from the tape backups and recovered database from hot and cold backups
• Wrote UNIX/Linux shell scripts to automate tasks (SH, bash, KSH, AWK and SED)
• Performed UNIX server build CLIP’s, work build and production queue IR’s and provided technical review for the go-live checklists across UNIX servers
• Held regular meetings to review usage as well as create, revise, and report any new measurements required to manage mainframe or distributed environments
• Recommends strategies for technical aspects of projects as well as broad system improvements, with a focus on AIX and Linux Systems

Environments Used
Redhat LINUX, CentOS LINUX, Oracle LINUX, IBM AIX, HP Server Automation, VMware vSphere, Saltstack, GIT, BlueCat IP Address Management, BMC Remedy, Cisco switches.

Value Delivered
The client was planning to replace all the Red-Hat Enterprise Linux servers with Oracle Enterprise Linus OS and were going to approach external consultants for the same. Veritis team informed the client that they have previous experience for the same and chalked out a plan with the client’s staff for execution. The installation was successfully completed and the client spent USD 55-60k less than what they would spend on external consultants.