ICMC: Building a Corporate-Wide Certification Program - Techniques that (might) work

Tammy Green, Security and Certification Architect at Blue Coat, came into what seemed like a simple task. She had had some Common Criteria experience when she joined Blue Coat, their product had previously been evaluated at FIPS-140-2 level 2, and they had already signed contracts with a consulting firm (Corsec) and lab.  Should've been easy, right?  Her new boss said it should take about a year and only 20% of her time.  Two and half full time years later... yikes.

One of the biggest problems was getting internal teams to work with her - even people involved in the previous validations didn't even want to talk to Ms. Green about it.

Nobody wanted to do this - they want to work on the new shiny features that they can sell, how does a process that takes 2 years (often not complete until after a product is EOL) help them?

It's hard to see the long term picture - you want to sell to the government, you need FIPS-140-2 validations.

Ms. Green didn't want to do this herself again afterwards. Instead of running away, she worked on setting up a certification team locally (her boss hiring a program manager helped to encourage it).

In addition to having the program manager, you need a certification architect.  You can't use the same architect as the product architect, because that person is busy designing shiny new features.

You need to work with the development team well in advance - fit your FIPS-140-2/Common Criteria schedule into their development schedule. You can't screw on the necessary tests and requirements as an afterthought, and you don't want to delay a schedule because requirements are dropped in at the end.

Target the right release: because FIPS-140-2 takes so long, you need to pick a release you plan on supporting for a long time.

Ms. Green found that after time... engineers stopped replying to her emails and answering her phone calls.  You need to identify key engineering resources to work with and their management needs to commit to those engineers dedicating 10-20% of their time to these validations.

Once you get this set up and have educated engineering, you'll find they'll reach out to you in advance - better timing!

Her team keeps track of what needs to be done: file bugs and track them.  You'd think the project manager for the product team would do this, but what she's found is that the bugs get reprioritized and reassigned to a future release.  Someone who understands validations needs to track these issues.

Ms. Green recommends that you create the security policy from existing documents: don't rely on engineers doing this. They simply don't understand what goes into this document or why it's important.  Instead, use engineering and QA to validate content.

It's important to convince QA continue to test FIPS mode and related features, as some customers may still want to run in FIPS mode (even though it wasn't validated) or that the release would be ready for validation if something went horribly wrong with the older release in validation.

Schedule time to prep. Ms. Green has 4-8 hour long meetings to make sure everyone understands what's important. Take time to prepare, make sure everyone knows what will be expected from the lab visit and have a test plan formalized in advance.  It's actually a lot of work to set up failures (the lab evaluators require that you demonstrate what happens when  a failure happens, even though you have to inject the failure to force it).  Debug builds, builds you know will fail, multiple test machines, platforms, etc.

To keep your team from killing you... or damaging morale, celebrate the milestones. Mention the progress in every status report, celebrate the milestones, do corporate wide announcements when you finish.

Do a post mortem to understand how this can be improved: give your engineering team a voice! Listen and take action based on what worked and didn't.

Update tools and features to make this easier next time: keywords to bugs and features, modifying product life-cycle, add questions related to FIPS to templates.

Suggestions/questions from the audience:
Make sales your best friend.  Validations/certifications are not fun, nobody does them for fun - you do this to make money.
Get the certification team involved as early as possible: from the very beginning - marketing design meetings.
Why don't you run your FIPS-140 mode tests all the time?  Time consuming, slower, not seen as a priority when there are no plans to validate.


This post syndicated from: Thoughts on security, beer, theater and biking!