At the end of November 2014, Axis41 upgraded CMO.com to Adobe Experience Manager 6.0 from CQ 5.5 and had the opportunity to migrate the site to Adobe Managed Services (AMS). It was a challenging process to manage the moving parts given the coordination between four distinct teams (Axis41, CMO.com, Adobe@Adobe, and Adobe Managed Services). While we have talked about the process or what it means to host a site on AEM “in the Cloud” before, we wanted to take a few minutes to discuss what it means to host in the Cloud with Adobe Managed Services. I want to emphasize that I am not an expert in Adobe Managed Services offerings but wanted to share our experiences in working with them on the CMO.com project. And, a special thanks to Patrick Kynaston, who was my principal contact and the Customer Success Engineer for CMO.com at Adobe Managed Services, and helped us through the migration from our system to AMS.
AMS uses Amazon Web Services for its hosting platform, which is actually what Axis41 uses when we host a website in the cloud. We like its feature set and its ability to scale up and down very easily. Because CMO was set to go live a week before Thanksgiving, we chose to have them host the site on a larger instance, so it would compensate for any unforeseen possible issues (no one wants to get a call during their vacation). We made that change with the proverbial flick of the wrist, without the need to wait for new hardware.
Similar to Axis41’s Systems Support team, AMS controls the Production environment—and the only action they allow in that environment is authoring. Everyone other than an AMS employee is locked out of System Console, CRXDE Lite, and Package Manager. However, they will create a mirror of Production called Stage, which is a complete copy of Production. As changes or fixes are worked on in the respective development environments, they can be brought up to Stage and then thoroughly tested to ensure they will work correctly when deployed to Production.
Pre-Launch Activities
Because of how new this process was for us, to work with Adobe Managed Services procedures, I felt that it would be a good idea to walk through a variety of the pre-launch activities that the AMS team requires their customers to go through in order to certify that a site is ready to be hosted on their systems. There are other areas that I will not cover because we are highlighting the major areas of coordination we had to go through. Your experience may differ.
RunBook
The RunBook is an evolving document that must be completed before AMS can deploy the site live. It is the heart of a site for the AMS team, as it contains all the relevant documentation about the site. A customer of the AMS team must document all of the necessary features, integrations, etc., that are critical for the site to run. The general idea is that anyone working within AMS should be able to be brought on board to the project, read the RunBook, and have the relevant information to manage the site infrastructure. The process of documenting all areas of the site for the RunBook allowed us to see potential areas of deprecation and resolve issues prior to deployment. The Runbook is probably the toughest requirement in working with AMS because of its complexity and requirement for high detail. Deficiencies in documentation at this point will only lead to frustration and confusion down the line, potentially at a critical period of time.
Review of Packages, OSGI Bundles, and Backend Linkages
The Runbook contains some areas that needed more in-depth review than others, and were divided into three categories: (1) responsibility of the AMS lead (Maintenance, System Monitoring), (2) AMS client responsibility (Cloud Services, Custom Packages, Custom Bundles, and Backend Linkages), and (3) mutual responsibility (dispatcher configs, DNS, Events and Responses). Let me briefly focus on the areas of biggest concern to us: Packages, Custom Bundles, and Backend Linkages.
We had to spend quite a bit of time documenting and preparing for a deployment to a new environment. As anyone who has worked with AEM is aware, Packages are where you put the components, content, styles, and other front-end logic for your site. Without them, you don’t have a site. As in any other systems development, over the years the site had accumulated some services and packages that were unnecessary and needed to be removed.
Custom Bundles provide the underlying services that drive the packaged components, as well as interact with external services and provide configuration specific to the site’s needs. As with the packages, we found that in some instances, the service was no longer relevant; in other cases, we needed to alter some OSGi configurations to align them with AEM 6.0, and to improve the process in general.
Backend Linkages are a listing of the third-party systems that the site calls upon to either bring in data or run a particular function. These backend linkages include things like Adobe Marketing Cloud products, reCAPTCHA, etc. All third-party integrations and external communications had to be documented, with the endpoints identified. In some cases, this required us working with external vendors to get information that was not readily available to the public, in order for the AMS team to understand how things were working.
User Acceptance Testing (UAT)
Adobe Managed Services requires that you prepare a User Acceptance Test set of test cases and ensure that the site passes these test cases. This is a good practice, regardless of hosting location. AMS did not require a completely robust set of test cases, only the critical areas of primary site control and authoring. In our case, we developed the test cases and did the testing for the customer.
Performance Testing
Another of our responsibilities was load performance testing to validate servers would handle a larger than normal load, which ensures reliability during traffic spikes and other unusual patterns. Although the AMS team would be responsible for the server, the responsibility for load testing the delivered platform fell to Axis41 and our Systems Support team. We used Apache JMeter to perform this testing, creating several scenarios—varying from a sustained load at more than double our previous peak traffic, to spikes of up to four times peak traffic. After we conducted testing, the process proved positive as we were able to see some areas of improvement that needed to be implemented in the out-of-the-box AEM list component and cache process.
Change Approval Board (CAB Process)
The process to move new code or configurations up to the Production environment is via a Change Approval Board, or CAB. The general rationale is when new changes are ready to be introduced to a site, the AMS team needs to be able to review and understand the updates and implications. The main function of the CAB is to certify that any changes have been tested and documented. Any bugs, errors, or issues are the responsibility of the AMS customer, not AMS. AMS will indicate that the customer had tested it in the Stage environment.
When you submit a CAB, you must also complete a document that explains what the deployment is fixing or enhancing. The document must include definitions of what is being done, indicate its risk assessment, give the step-by-step process for the AMS individual deploying the code, and lastly outline the reversion plan in case there is an issue at deployment. The document is submitted to the AMS lead and they begin the approval process. AMS has at least five individuals on their side that must review the document and provide approval or denial. The customer can also add additional people from their side to review things, such as a development supervisor, but this depends on how involved the customer wants to be. Because of the long-standing relationship of trust that the CMO.com team has with Axis41, they do not require reviews on their side and we can submit things when we deem it necessary.
In the case of CMO.com, there is a weekly release schedule. We could submit a CAB whenever we want, but, similar to software development best practices, we felt that it was best to maintain a regular, rigorous release pattern. This means that in order for a deployment to occur within the release window, a CAB must be submitted three days in advance. Yes, three full days in advance. This forces us to work regularly in a good cadence with the CMO.com author team to ensure they have reviewed and approved items in time for us to submit to the CAB.
ECAB
After learning about the CAB process, our first question was, what do we do if there is an emergency? In the past, we had complete access to make any needed change, on the spot. We saw this as a shift and paradigm change for both us and the CMO.com team. The answer from AMS was the Emergency Change Approval Board, or ECAB. The requirement to submit a CAB is still in place, and eventually it must be reviewed by their board, but they are willing to abbreviate the process in order to get the issue resolved and maintain uptime or site integrity. After the switchover, we had to request a few ECABs as there were some configurations that needed to be adjusted, but an ECAB is a special exception—not a standard practice. I wouldn’t recommend getting into a habit of multiple ECABs as it is likely to raise a few eyebrows on the AMS side, if not within your own development team. It is better to spend the proper amount of time needed to test the code/system/upgrades than push out code and hope everything is ok.
SharePoint
AMS uses the Microsoft SharePoint system as a place to maintain and manage all relevant documents and CABs. If you are unfamiliar with this tool, your AMS adviser will show you some of the basics to navigate the system.
Day Care Ticket
AMS offers 24/7 support, and customers looking for assistance are assigned a specific worker who will be available during normal business hours. But, what happens if something goes wrong during non-business hours? The process for communicating a problem is to create a support ticket with Adobe. AMS has employees on staff 24/7 and can assign an issue to the on-call/site worker. They will then review the Runbook to understand how things are handled in your specific implementation.
Conclusion
Although we were excited about the challenge of migrating to AMS and working with the AMS team, we were accustomed to managing the site from end to end and had to adjust our processes. We initially had some concerns about loss of control over the site (especially access to CRX in the Author environment and being locked out of Publish completely) when we are the stewards and day-to-day managers of the site. Navigating this new relationship continues to be an evolving process, as we work hand-in-hand with the AMS team in their responsibility for maintaining the infrastructure uptime and our responsiblity to the CMO.com team to make constant updates, enhancements, and improvements to the CMO.com site.
In conclusion, I know that this list is not exhaustive and that your experience may differ, but this should hopefully give you a good idea of the types of things that you will need to do should you ever choose to have Adobe Managed Service host your site “In The Cloud”.
CMO.com by Adobe delivers marketing insights, expertise, and inspiration for and by marketing leaders—all aimed at helping CMOs and senior marketers lead their brands in this new digital world.
Adobe Managed Services provides cloud hosting and expert application support as well as 24×7 maintenance and support and a single point of contact for each customer throughout the application lifecycle.
Adobe@Adobe partners with Adobe product teams and business owners to design and implement solutions that showcase Adobe’s Digital Marketing and Digital Media products by delivering business value using Adobe technology within Adobe and on Adobe.com.
Axis41 is fully integrated creative and technical development teams working together to strategically create, deploy and optimize marketing content that inspires and strengthens profitable relationships for our customers.