The COE project

Background

This was a massive crunch, unlike any migration I’d ever seen. I’ve never experienced a business actually going for big-bang migrations. We had various technical dimensions that made this an incredibly challenging job:

  • Merging Wellington, Hutt Valley DHB’s & Wairarapas (end-user) computer systems onto a common operating environment,
  • Rushing HVDHB off Novell NetWare/ZenWorks,
  • bringing an unmanaged XP environment (at CDHB) into a very-heavily managed Citrix design.
  • Additionally, we were introducing application virtualisation where there wasn’t even a packaging process previously,
  • Changing Windows to a 64-bit architecture on both the desktop and Citrix farm
  • Migrating Office versions

Each of which, when on its own would result in: a project, a team of people and a bunch of money budgeted. This is all aside from the ‘get off XP’ part of the project…

Prior to the CEO’s involvement, we had a design settled; completed several pilot migrations and were starting to line up and pick off the initial business units in a linear, low risk, one-at-a-time method (expecting to take 24-36 months just for CCDHB)!

In all, once the CEO decided to make the project a priority, we did everything excepting Wiararapa’s migration roughly within a year (at the time of writing WRDHB is ‘in flight’). Throughout the year we added additional packagers until the team was up to 9, pure packagers too, no infrastructure/deployment/discovery or UAT involved!

Our design approach changed after Citrix dropped application streaming and we committed to App-V 5.0 SP1. Immidio Flex+ was used for user profile configuration/management. This was delivered almost exclusively on Citrix, where each server is essentially a commodity – no separate silos or additional management overhead.

We achieved roughly 95% App-V and almost the same with Citrix delivery, somewhere over 90%. There were certainly a few applications where we persisted over a few weeks with challenges through multiple revisions – while not necessarily cost-effective at first glance, sometimes it was simply because it would be worth it in the long-run (the user-base might have been quite significant/Citrix delivery might be necessary and we wanted to avoid having separate silos). Additionally, replacing App-V’s is far simpler to MSI deployments!

  • Really our major issues barring App-V-delivered packages were retail products that activated online (and we didn’t want a user-state App-V repair to invalidate the license).
  • Our biggest blocker for Citrix delivery lay with hardware devices such as foot pedals or medical components.

We pulled out so many PC’s from both hospitals; the savings with support and lower costs of terminals were fantastic!

Having Citrix-delivered App-V packages were such a dream to supersede (we avoided connection groups, some parts of the design may look odd, but matched the hospital usage-scenarios well. The way the hospitals use their devices and the scenarios employed were unlike any corporate I’d ever seen!).

The big-bang migrations were pulled off very smoothly. CCDHB & HVDHB each took about 4 separate weekends to migrate. There were some nerves initially, but we didn’t have one application come back as an ‘urgent repackage’. Certainly we had a few alterations and updates, but nothing urgent that could ‘join the queue’, we certainly had enough packagers however! (The team expanded to 8 concurrent, pure-packagers at the highest point. We sustained this for about 8 months I think, culminating while I was away for the second half of CCDHB’s migrations)

The team

Rory Pollock – Packager

Rory was great – he was an ex-desktop support guy from HVDHB that we were training up. He took to packaging like nothing I’ve ever seen before. Rory completely understood the registry and the implications of virtualisation, integration and the data in the sandbox. He was inexperienced, but quickly became quite advanced in his understanding of the App-V engine and implications within (paths missing from VFS, pre-creating items to cause a merge during sequencing etc). I ended up often running hypothetical scenarios past him to check my own thinking in certain situations. Rory became particularly strong at considering where the ‘data’ lies with an application and the way an App-V can be structured to cater for that. Rory welcomed my push to automating the sequence through the use of custom package-specific templates and pre-sequence actions.

I hold Rory’s abilities and cool-headed logic in high regard and would desire him in my team in future. I had to promise not to steal him!

Thomas Cole – Packager

Thomas did really well – he was an ex-desktop support guy we were training up, he didn’t desire to ‘change careers’ as a packager (and previously moved away from server work too) but still became a solid contributor and earnt the role of the first shim expert in the team. Thomas could be relied on to deal with incomplete information, start chewing on a package, and cut it a few different was – studiously stepping through the experiments until he settled on something worked, or found a better outcome.

I would welcome Thomas in my team in future.

Wayne Cresswell – Senior packager

Wayne was a perfect early joiner – He’s got many years of experience already so I loved his help mentoring the juniors. I could trust him to take a few of the really curly apps while I was buried in other facets or training up people. He didn’t say much but often had us in stitches all the same. He was crucial in the process and standards development – we were both completely new to App-V! Wayne didn’t get to see the migrations in the end but I would have him back in a heartbeat!

Aki – Packager

Aki joined us fairly late in the piece, well into the booming phase where we had very a large team and were competing in a numbers game. He easily slipped into the team dynamics and got to work. He was always cheerful and quickly became well-liked. We has brought back in after the initial packaging phase was completed to assist with broader system administration tasks after BAU staff resigned. I have worked with Aki previously at gen-i and we’ve always gotten along well. I look forward to working with him again.

Matthew Kelly – Senior packager

Matt joined us fairly late in the piece with another packager, in the booming phase where the team was growing. We had lost some key SCCM resources – Matt’s experience was quickly utilised and he happily spread himself across the two teams – I half wonder if he just enjoyed having more toys to play with? Matt was another fairly senior packager and was often useful with his automation of everything he could get his hands on. He had no issues coming to grips with the complicated approach and tools involved and his scripts still teach me about PowerShell. I was sad to see him move back to Christchurch as I hoped we would be able to work together more in future.

Mark Wilson – Senior packager/Citrix engineer

Mark was such a blessing, we had more juniors coming and management had just stepped in and re-arranged the priorities. I was struggling and Mark came along as another very experienced, highly-skilled packager who I could trust not only with the hard jobs, but also mentoring the new guys. Straight after the interview I was leaning on the PM lest someone else take him. He stayed with us and became a crucial part of the Citrix team after the lead resigned – he slotted in perfectly with the BAU engineer leading and the two of them became a very tight unit. He polished off some extremely complicated tasks (I was quietly relieved they were in someone else’s good hands) and never showed a sign of stress – He’s one cool head, don’t play poker with him!

Callum Holmes – Project director.

Callum was …a force of nature. He came in, I think at the request of the CEO to review the project, and was an agent of change. We already had inertia and settled-in to a risk-free approach and honestly, I’ve never seen an organisation allocate budget, increase the team and compress the time frames drastically like this before – the risks were considerable, but Callum kept the management not just involved but often had to push them into understanding what was going on and whether or not they would sign off.

When he arrived, he quite rightly, told us we wouldn’t like him; that he was here to kick us in the pants and get us fired up. He turned the clinical risk issues that had held us captive inside out and got the business on-side.

But once he had us firing, it was like he then became ‘set on the department heads’ and the rest of the management layers. He was instrumental in getting clinicians involved in what was going on (whereas previously we had almost no level of engagement – these guys are already hideously busy, with enough on their plate to manage – risks and otherwise). Honestly, once he had us ‘humming along’ I thought very highly of the way he handled the political situation.

And then, when things started happening and the had D-Day with Hutt valley was successfully avoided; he stood back and pointed the praise at the workers. I should’ve hugged him.

I’ve learnt a lot from Callum and consider him a fantastic manager who gets things done. Despite being a little bit intimidated, I think I’d come back for more…

Jeff Moran – ‘Packaging management unit’/’project clarity officer’

Jeff was another ray of light in our team. He came on just before the boom phase; the project Directory was pushing us to each average 1 app per day, no defects. Jeff quickly worked on our inputs (which were often something vague and incomplete) and the available lists of applications. He became a master of Excel and was the source of truth about what apps existed, where they were in the process, and who was (meant to be) working on them. At the time I was struggling to motivate the team and let go of discovery and other responsibilities. Jeff was our management reporting extraordinaire. He tweaked our progress tracking spreadsheets (not something I’d been strong at) and quickly started charting progress, instantly creating motivation in the team. He was instrumental at defending and reducing our set of responsibilities so we could focus on package development – something the Project director was very hot on. This gave me a lot of room to breathe as ‘letting go of discovery’ and clinical risks were still a considerable tax on my attentions in the early phase.

In a previous job, I had spent years maintaining the earliest example I have ever seen of a scripted-image hybrid build before. This was something Jeff had implemented. He was quickly liked by the team and I continue to hold him in very high regard – both as an architect who’s not shy of some dirty work but also as a manager who took on an established team and was liked by all.

Benjamin Guy – Designer & Technical lead.

None of this would have been possible without Ben. When the 2 other leads left (separately) I was and worried he might do the same. He stayed on and finished the implementation on his own. When Wairarapa DHB was ready, Ben was the key that organised it.

Ben is incredibly relaxed; I honestly don’t know how he managed to sleep with the stresses that were going on with the Novell exit at HVDHB. He is well liked and approachable, is respected by others and so has no problem addressing conflicts and motivating others to work. There were times where he would champion the principle of a design despite short-term awkwardness and won over reluctant engineers. This was Ben’s first desktop migration and he absolutely smashed it out of the park.

He worked very closely with the SCCM and Citrix design leading to the POC well before the project really kicked off. They had some major hurdles changing technologies as Citrix dropped their application virtualisation tool (forcing the change to App-V) and it was decided to change from Altiris to SCCM late in the POC. The three of them put together a solid design that addresses the complexities in the clinical workflow and application delivery.

Oren – Designer & Tech lead, Citrix

Oren was a crucial part of the design phase, and a big champion of Immidio Flex+. Part of the success in the design is due to his adherence to its ideals – Oren had a direct impact on how fully the design became such a solid implementation. He was a really hard customer and hard to please, but with his input to the team we shipped many packages that simply didn’t look like they were suitable.  Oren’s troubleshooting and analytical abilities were crucial in many cases. Often he would use the applications within Citrix, digest enormous perfmon traces, and feed it back to us with altered requirements. He’s an incredibly technical person, now working directly with Citrix as an escalation engineer. Oren ‘drove us’ brilliantly, and was instrumental in mentoring one of our juniors who quickly became senior.

Sadly Oren left before the migration phase had completed – I personally found this intimidating as we had worked so closely together and I had developed such a trust in his abilities.

Mike Pohatu – Designer & Tech lead, SCCM.

Mike was instrumental in the early design phase. When it came time to take up the tools, Mike was the build development king, and was responsible for the workstation-related deployments – Often awful MSI’s with activation-related problems, fortunately there were very few of these. Mike really ploughed himself into the SCCM development that much of this is hinged off. Mike was well aware of the unmanaged XP install base and was instrumental in championing the cause to our early pilot groups. Together we defined the way passing of packages ‘across for SCCM configuration’ and the specific additional requirements as needed (Flex+ user profile footprints, group policy or dependency chains etc). Mike designed built a tool to manage the print queues between the Citrix and Win7 environments and scaled it to the multi-tennanted scenario the COE design became.

David Gray – Senior Citrix engineer

Dave came in after the Citrix Tech lead (and one of the designers) resigned to work directly for Citrix. Dave really took the reins about a month before the COE was implemented for HVDHB and passed the trial by fire with top marks. We quickly clicked, he has a very affable character and despite being in ‘the deep end’, he took on the massive tasks and stress load and saw the implementation though. Dave was a joy to work with, and mentored another packager into the Citrix team prior to becoming a father – now the pair of them work very closely together and Dave is starting to mentor another resource from the desktop team.

Dave was never afraid to ask questions and was always open to helping others; he was a really key part of the team. I’ll miss him when I’m gone…

Shyam Menon – SCCM engineer

Shyam was our COE build engineer and SCCM deployment king. He was great as we were somewhat used to Citrix servers which deploy in a highly managed fashion via scheduled maintenance windows and so on. The PC fleet, while not particularly large, was often used in weird and wonderful ways: The design utilised many kiosk-like scenarios where service accounts were always logged on 24/7. Shyam kept things structured and cool-headed.

Shyam was regularly producing .HTA’s for us or the other support teams, such as tools to query AD or SCCM collections, group update scripts etc. There’s always one heavy scripter in the team, it was nice being away from that for a change (Shyam was way ahead of me).

Shyam took over a complicated custom-printing tool that kept the users queues synchronised between Win7 & Citrix servers, across the 3 customers. This was something that had been architected by one of the designers, Mike Pohatu, and built for CCDHB but had only loosely been ported for a multi-tenanted environment. Shaym took on this bespoke code and maintained it through the implementation of the 3 customers, allowing staff to work across DHBs with seamless printing.   At the time of writing this is in for testing at present.

Shane Eagle – End User Computing Team lead

Shane was originally the senior server engineer from Hutt Valley and became the team lead over the End User Computing team (Application packagers, Citrix and SCCM/deployment engineers). Shane was a deep thinker and would often push for the better solution despite a perceived complication others might have been adverse to – he often understood the business better and always let that drive his judgement. Shane always had a cool head and was able to cut to the heart of matters. He was a fantastic head over the team, regularly pushing for complete, whole solutions instead of something more reactionary.

Tyronne Schalkwyk – Packager / Deployment / Citrix / Swiss army knife

Tyronne was originally a Desktop Engineer who was looking for work to do. He started working with us as a packager and succeeded in taking on board the application virtualisation implications. After a few months he succeeded in moving into the Server Engineering team officially and so the EUC team lead moved across into assisting the Citrix and SCCM delivery. He was a useful assistant to the configuration and Citrix leads at the time and is one of the few resources who have worked across the board in the end-to-end delivery context. This more or less encapsulates Tyronnes approach – he takes things on, gets them done and helps out those around him. He’s an all-rounder who hungrily takes up new tools and seeks to learn how they work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s