Description: Service Delivery Cloud Migration
High Level Summary:
The client was performing an extensive internal change, due to growth via Mergers & Acquisition, requiring a change in support vendor. A consolidation of multiple Managed Service Providers was in flight to reduce to a more manageable number, and create efficient internal workflows.
Objective:
Primary objective was to provide guidance and governance to the cloud services (Azure and Office365) support vendor, SoftChoice. Oversight of workloads and workflows in addition to the performance monitoring via standardized KPI’s was required.
Secondary objective was the day to day management of internal requests and tickets. This required management of in-house systems, Active Directory, on-premise E-mail and spam solutions, and Cisco CUCM.
Description: Agile Transformation
High Level Summary:
The client was performing an extensive internal change, which required the re-organization of the Software Development team from an unstructured and chaotic team into a fluid Agile and Scrum based team.
Objective:
Primary objective was the implementation of Agile and Scrum methodology and cadences with a Java and IBM WebSphere Commerce development team to support Continuous Integration/Continuous Deployment (CI/CD) pipelines. Project Management of software projects.
Secondary objective was the creation of supporting tools, such as Atlassian JIRA to provide full reporting and feedback on targeted KPI’s. Agile projects tracked within JIRA leveraged Sprints, and sprint related ceremony and cadences to enable future planning and backlog grooming.
Description: Migration to Office 365
High Level Summary:
The client was performing a planned migration from an on-premise mixed environment of Microsoft Exchange 2010 and Lotus Domino 9 to a completely hosted Office 365 environment. The volume of mail objects were 2000+, including user mailboxes, shared mailboxes, group mailboxes, shared calendars and resources.
Objective:
Primary objective was to perform a clean cut-over migration to the Office 365 platform during a weekend maintenance window. All mailboxes were required to be migrated, synchronized and fully accessible with offline local archives brought into the cloud platform. Distribution groups were to be migrated to Office 365 Groups, contacts were to be cleaned up and imported, and consolidated resource management (meeting rooms, projectors, etc.) was to be introduced.
Secondary objective was the implementation and roll-out of the Skype for Business platform to support the enablement of online real-time communication collaboration.
Challenges and Risk Mitigation:
The primary challenge involved reactive changes to the planned Office 365 architecture and deployment, from a large quantity of unknown’s due to internal staff churn, and changing requirements/project scope. We maintained timelines and velocity with daily stand-up meetings, open collaboration and granular allocation of tasks and duties with various team members.
The secondary challenge met was the migration of Lotus Domino mailboxes into Office365, requiring a 3rd party tool, BitTitan, to complete the process. The tool involved a small amount of complexity, and required comprehensive testing and planning to ensure fluid data transition.
Description: Migration to Office 365
High Level Summary:
The client was undergoing extensive internal restructuring, right-sizing staffing and migrating services to the cloud to reduce costs and mitigate risks. The Office 365 was selected as the premium marketplace leader, with rich functionality that could be scaled with simplicity. A fully cloud-based environment was supportive to restructuring and streamlining of the internal IT department, and facilitating outsourced and off-shore support.
Objective:
Primary objective was to migrate the historical mail, and configurations intact in a manner that mitigated extended downtime, whilst right-sizing mailbox/user counts to reduce OPEX costs. Secondary objectives were directed towards utilizing the additional feature functionality and integration available within the Microsoft Office 365 technology ecosphere.
Challenges and Risk Mitigation:
Primary challenges were resultant to internal restructuring creating dynamically changing requirements and implementation strategies. Extensive analysis of existing configurations was required to ensure the new platform was provisioned for the company’s future and decoupled from legacy complexities. Secondary challenges were driven by cost control and budgetary planning and multiple analyses performed for short-term and long-term change.
Description: JIRA Agile Cloud Migration
High Level Summary:
Client was migrating all on-premise services into cloud/SaaS versions to support a change in internal IT Strategy, and re-org reducing IT headcount. The Atlassian JIRA product was to be maintained, although utilizing the “on demand” cloud version.
Objective:
Primary objective was to migrate new projects into the cloud version, whilst maintaining legacy/historical projects in the on-premise version as read-only. As the project progressed we strategized that migrating the historical projects into the cloud was the best fit, as the on-premise version could be fully deprecated and no additional costs would be incurred with the cloud platform.
Challenges and Risk Mitigation:
Primary challenges were caused by the on-premise version of JIRA being nearly two full revisions behind the cloud version. With marginal downtime, due to near 24×7 operations, we had to provision a same version of the on-premise server in parallel to the production version, and migrate data and perform several staged upgrades. Negotiations with Atlassian enabled us to source temporary maintenance licenses to allow the upgrade before migrating to the cloud.
Secondary challenges occurred due to shifting timelines with the client for availability to deploy, caused by client backlog of project sprints, compounded by migration bugs encountered during the initial window. An open and clear dialogue was maintained with the client highlight on-potential complications with delayed dates with the primary risk involved with revision drift from the on-premise JIRA server and the Atlassian Cloud, as well as on-going internal JIRA plug-in development and integrations.
Description: Cloudification of SaaS Product
High Level Summary:
Progressive migration of in-house and on-premise services/infrastructure into a high-availability robust cloud platform. Amazon’s AWS platform was selected due to its maturity and identified suitability to software and development platforms.
Objective:
Primary objectives were driven by lowering CAPEX costs, reducing OPEX with minimal compute power which could burst with demand, and lowering payroll costs linked to traditional infrastructure.
Secondary objectives were driven by increased high-availability, improved SDLC time-frames, simplified management, and scalable backup/DR functions utilizing native AWS modules and API.
Challenges and Risk Mitigation:
Primary challenges involved changes to infrastructure automation and SDLC automation, which were mitigated with extensive pre-planning to understand future requirements leading to an early implementation of supporting systems; i.e. migration to Jenkins platform from Atlassian Bamboo, and reconfiguration of IP based API calls to split-DNS based calls.
Secondary challenges involved infrastructure and integrated architecture changes, which were mitigated with simplistic tweaks to existing architecture and substitution of fat systems with native AWS services; i.e. replacing SendMail and PostFix relay clusters with native AWS SES integration.
Description: Implementation SaaS Product
High Level Summary:
A traditional hosted customer facing CMS system was in production that was originally designed for an alternate market, with simple web GUI drive access and a cumbersome offline client. Highroads were designing a new “V2” system and was in development as a fully SaaS product with both web GUI access and customer available API access. Infrastructure was required to support the SaaS product, provide reliable operations, secure access and customer data, and enable customer-driven Single Sign-On integration.
Objective:
Due to extensive technical debt, a new production stack was architected and implemented, with tertiary systems running on legacy shared infrastructure. Future growth was expected to be significant, so a key objective outside of feature/functionality was to design a vendor agnostic architecture that could scale outside of a single data centre.
Challenges and Risk Mitigation:
Primary challenges involved security due to the nature of customers on the platform. Although no PII, PHI or HIPAA data was stored, customers had to manage those controls and by default, Highroads had to be able to achieve the same or similar levels of controls and security. The infrastructure was architected to provide a level of security and abstraction between SOA microservices, with network segregation controlled by multiple firewalls within a single multi-tenanted environment. All SOA microservices were designed to be stateless worker nodes carrying out transient processing via API calls, with data only being stored in the backend database. Infrastructure was tested in-house to verify data isolation and ensure zero tenant bleed with penetration testing performed bi-annually with an external security partner.
Secondary challenges involved the requirements for low latency responses within the application whilst performing complex queries followed by simplistic changes to multiple client data plans. Initially, a continuance of utilization of the Oracle 10g platform was implemented, however, the costs of implementing Oracle RAC was prohibitive. We reviewed various options, including migrating data through the application stack into caches, as well as reviewing alternative vendors. The final decision was made to implement a NoSQL backend that better served the application requirements, whilst enabling full HA clustering at a lesser cost than regular Oracle 10g licensing. A full migration including development changes and staged QA analysis was carried out over a three-month period to the CI/CD stack and subsequently production stacks.
Description: Corporate Asset Sale
High Level Summary:
During Q4 2015 Highroads sold the assets and IP for their “V1” services, including physical assets, IP, licensing, customer base and staff to Enrollment Advisors, now DirectPath, LLC.
The asset sale required a project to divest software, development environments, QA environments, staging environments, demo environments, training environments, DR environments, and production environments for the “V1” software product. Secondary systems including monitoring, security analysis, edge networking (firewalls, load balancers, etc.,) and backup systems had to be either migrated to the purchasing company with replacements provisioned in Highroads infrastructure, or newly provisioned identical systems provisioned in DirectPath infrastructure. Tertiary systems requiring migration and/or new provisioning included corporate applications (Wiki’s, CI/CD software, JIRA, etc.,) physical office installations and office hardware.
To be compliant with the Sale Agreement and Transitional Services Agreement 80% of the work had to be carried out by Highroads with zero downtime to either organization in a timely manner as defined in the sale documentation. A significant percentage of the sale value was held in escrow until completion and approval of the purchasing company.
Objective:
The primary objective was to stand up a brand-new infrastructure for the purchasing company that was identical to the existing Highroads infrastructure, driven by the physical assets, licensing, and resource reallocation as defined in the sale agreement. Where required, either company would have to purchase a new/replacement component for divested infrastructure/services.
Challenges and Risk Mitigation:
Highroads lost 40% of the IT Operations staff due to the sale, with staff being moved to DirectPath, whilst realizing a further 20% attrition of resources during Q4 2015. A significant risk was present as the staff moving to DirectPath were long-tenured staff, and the Highroads retained staff were all of less than 1 year’s tenure to the company, this was realized a major deficit in understanding of the company-specific unique configurations of systems. Mitigation was implemented by bringing in dedicated private consultants focused on networking/security, systems and virtualization, and generalists to address the resource/capacity shortfall. Further supplementation was made by bringing in a dedicated project coordinator to work directly at an equal level with both purchasing and selling companies, as well as provide direction and management to consulting staff.
Secondary challenges involved the complexities of divesting, reallocation, and migration of vendor services, subscriptions and licensing. Various issues arose that were conflicting to the definitions of the Sale Agreement and Transitional Services Agreement due to legal agreements and restrictions in place with vendor contracts and MSA’s in place with Highroads. Mitigation revolved around on-going discussions, mediations, and negotiations with Senior Executives in DirectPath in parallel with legal counsel.
Description: Canadian Operations Closure
High Level Summary:
In Q4 2016 Highroads announced the closure of their Canadian offices, with the subsequent planned closure of the Canadian company. IT Operations was to be fully handed over to various outsourced vendors, physical presence within the Ottawa office was to be immediately deprecated and the closure of the data centre supporting the “V1” application was to be planned.
Objective:
Primary objective was to identify all internal IT Operations functions that we carried out purely from the Ottawa office and relocate those processes and functions to outsourced vendors. Key focus was placed on risk mitigation from lack of presence and established staff.
Secondary objectives included the planned and controlled closure of Ottawa office services, infrastructure and systems post-layoffs as a “lights out” facility, with the planned deprecation of data centre presence after customer migration to the “V2” platform.
Challenges and Risk Mitigation:
The primary challenge was a sweeping and overarching risk with the handover of all IT Operations to outsourced vendors. Due to the relatively short tenure of staff, wholesale changes to the company structure and infrastructure systems from the asset sale of 2015/2016 and compounded by the focus on supporting development changes with an under-resourced team there was a significant lack of documentation and cohesive knowledgebase for corporate systems, CI/CD systems, and production systems. The risk was easily mitigated by focusing exiting staff on documentation work during their working notice, with repeated blind-testing knowledgebase pages for efficacy with internal staff and on-boarded vendors.
Secondary challenges involved the management of vendor contracts and leasing agreements associated with the Ottawa office, and transitioning agreements with the Canadian company to the US company for ownership. The loss of all Highroads administration staff prior to the closure required extensive work and diligence to capture all contracts, and put in place fail-safes for capturing outstanding issues post closure.
Tertiary challenges were to fully document and detail the planned closure and deprecation of all data centre based systems, and functionality for future closure. The data centre was to be retained in operation, with remote management from a Boston, MA vendor until the last customer was migrated from V1 with an ETA of Q2 2017. Plans and schedules were put in place to guide the remote vendor through hands-off deprecation to mitigate costs and limit hands-on work to a minimal requirement that could be completed by a local vendor at a reduced cost versus international travel.
Description: Virtualization Strategy
High Level Summary:
The Dymon corporate infrastructure was operating on unique physical hardware per server and per service. We identified 8 physical servers of identical capacity that we running at less than 20% utilization, in one instance 0% utilization. Switching and networks were analyzed and we identified 3 totally unused Layer 3 switches with zero utilization.
A plan was put forward to migrate to a fully virtualized infrastructure, segregate corporate traffic with various VLANs to support operations, POS systems, publicly accessible web services, CCTV data, and credit transactions in addition to iSCSI traffic, virtualization traffic, monitoring traffic, and Building Automation traffic.
Objective:
Primary objective was to re-architect the corporate network to support separation of data based on type with QoS and rate limiting, implement a robust and fault tolerant virtualization stack to eliminate all single points of failure and utilize un-allocated equipment to enable growth and enhance redundancy.
Secondary objectives were to streamline the requirements for operation and eliminate frequent downtime occurrences, whilst improving the efficiency and efficacy of the network as a whole, and enhancing backup procedures.
Challenges and Risk Mitigation:
The primary challenge was soft challenge realized as a distrust and fear of virtualization technologies. We mitigated this with extensive training and exposure to real-world examples of implementation with our test lab. Key focus was placed on demonstrating the benefits of virtualization with physical failures failing over within the cluster, speed to create and implement VMs, and the benefits of features such as snapshot technologies.
Our secondary challenge was working with a zero-value budget. Physical servers were re-purposed by creating a rolling migration plan to migrate a physical system and then convert the hardware into a hypervisor. Xenserver was selected as the hypervisor of choice due to its zero cost whilst supporting HA clustering, with the option for vendor support from Citrix in the future as required. Various QNAP SANs were being utilized for CCTV storage, and these were re-purposed into a “poor mans” cluster with simplistic pooling and data replication handled either by the SANs or via VMs.
Description: Thin Client Rollout to Health Care
High Level Summary:
The Dymon Health Care retirement residences were operating with heavily aged systems that incurred excessive amounts of downtimes, ran various operating systems and various versions of software. A frequent churn of nursing staff required unique profiles to be created on each system repeatedly. We identified a number of different solutions, with selection being given to a Terminal Server (RDS) / Thin Client solution where roaming profiles were available to each staff member, and data was centralized into the corporate servers away from the local desktops.
Objective:
Our objective was to leverage technology to reduce the overhead of managing systems, and staff related issues, whilst consolidating services, centralizing data and incorporating into corporate backups.
Challenges and Risk Mitigation:
Our primary challenge was managing a tight budget, with a focus to being as close to zero spend as possible. The Dymon Storage companies already leveraged a Terminal Server licensed application that we upgraded to Microsoft Server 2012R2 with Remote Desktop Services (RDS) and created segregated appliance for the Dymon Health Care workers. Existing hardware was repurposed where possible using “AnywhereTS” Open Source Thin Client operating system, or sourcing Dell Wyse Terminals when required. Simple changes to DHCP scopes enabled zero effort deployment for new/replacement systems.
Description: Point of Sale Upgrade
High Level Summary:
A vendor driven change to end-user credit transaction PIN pads required Dymon Storage to update their credit transaction software, Line of Business integration module, merchant banking authentication and physical handsets with zero downtime realized to storage facilities.
Objective:
The POS Upgrade object was to replace the PIN pads, upgrade and reconfigure each unit and complete a site within 45 minutes. Pre-testing verified requirements, installation testing a transaction of $0.01 verified the installation, and post-implementation reporting verified the back-end allocations.
Challenges and Risk Mitigation:
Extensive pre-migration testing showed that software changes could be applied to the processing system after EOD batch processing at 1am, with LOB software updates able to be performed after that. Updating the physical PIN pads required an on-site presence to physically connect and secure the PIN pads, software and security updates to the POS terminal and simultaneous reconfiguration of each POS settings within the LOB application, merchant bank authentication portal, and transaction software. Development staff and staff accountant were seconded to the IT project to provide additional on-site presence to facilitate the changes in a timely manner as downtime was expected to be zero. Due to the security nature of POS administration we were unable to leverage store staff or regular staff to support this change.
Secondary challenges required extreme due diligence for the configuration process as the changes required allocation of terminals, terminal code usage, and transaction processing to be correctly allocated to a merchant account for either the physical site’s business, consolidated operating business or web-based business transactions. Any error would require a new PIN pad to be requested from the vendor and downtime.
Description: Corporate Restructuring
High Level Summary:
During the years 2013 and 2014 GeoDigital went through successive periods of expansion and contraction, in both company size and market ownership. A boom market saw an accelerated growth of sales requiring a growth in resources which drove a corporate expansion via M&A. The growth required not only input from a Due Diligence perspective, but also technical and corporate challenges on-boarding new acquisitions and incorporating staff and process with a corporate alignment to the GeoDigital philosophies.
Objective:
Support growth of the company with growth of IT Operations teams, ensuring the most efficient use of resources to balance end-user support, production support, and internal project work. Support and facilitate the on-boarding of new offices and staff inherited via M&A activities, and roll-out standardized systems for incorporation into corporate strategy.
Manage and address budgetary requirements, and departmental organization during time of corporate contraction. Facilitate reduction in OPEX across all business units, including right-sizing of vendor agreements to support internal change and corporate restructuring. Renegotiation of vendor contracts to support changes, and facilitate temporary stretch-credit terms.
Challenges and Risk Mitigation:
Growth challenges required extensive planning and agile changes to change to support on-boarded companies. Standardization of systems and efficacious consolidation from previous company architectures into company standard financial systems, ERP, MRP, project tracking and knowledgebase systems.
Corporate contraction challenges were primarily budget focused. CAPEX investment was shifted to future years, with extended life-cycles of infrastructure provisioned. Relocation of corporate data and the creation of a ‘virtualized office’ to capture infrastructure, systems, and data that were integrated to a closed facility for virtual access by remote staff and historical access. Planned and unplanned departmental attrition required extensive cross-training, prioritization of resources, and contract changes.
Description: Creation of SaaS product
High Level Summary:
GeoDigital created a market pioneering SaaS application utilized by premium US insurance companies to provide a Photo Acquisition Service (PAS) system to provide on-the-fly imagery and extrapolated engineering metrics for roofing claims. API integration provided the customer’s agents the ability to receive zip bundles of imagery along with PDF descriptors of building dimensions, including pitch, slope and height via single click integration to customer’s proprietary Line of Business (LoB) application.
Objective:
Primary objective was to design, architect, and develop a fully sustainable and scalable SaaS infrastructure that could grow vertically and horizontally as utilization increased with growth of customer of customer base, and could grow in feature functionality as additional market requirements were serviced.
Secondary objectives required compliance to client security requirements, ease of data mining of usage, efficient backups of data and client access audit tracking metrics, and a high level of tolerance to support a 99.9% uptime SLA.
Challenges and Risk Mitigation:
Architectural and process challenges were of significant impact as this was a new market venture for a pioneering product, and the organization’s historical was focus was LiDAR and engineering. Extensive pre-planning was required, with constant agile adjustments to the teams and progress as the organization grew into the requirements, leveraging internal resources as well as technical partners and vendors. A cross-departmental tiger team was created to support the overall management of development progress, SDLC release cycles, infrastructure architecture and implementation, and scheduled maintenance.
During the early development stages, we worked closely with an early adopter, and faced multiple struggles with initial go-live release and product readiness. To mitigate a direct and tangible risk with the customer, I worked closely with the teams on a daily basis, providing daily status updates to the primary client, significantly enhancing client buy-in, tolerance to development issues and delayed releases and generally enhancing client-supplier relations. The client was kept abreast of internal issues with comprehensive and detailed Root Cause Analysis (RCA’s,) provided with fully detailed resolution and mitigation documentation and processes to provide a high level of transparency and build an on-going relationship.