ITSEC Evaluation, what could be improved?
'ITSEC Evaluation, what could be improved?' is an interesting question.
There is plenty of scope for significant reduction in evaluation costs, improvements in
information to potential users and policing of vendor claims, but that assumes ITSEC
survives long enough to justify changes.
There is also a very big question about the value of certification schemes.
What is often forgotten is that security is all about time and deterrence. IT
systems are not a special case. One company can live with an encryption system that takes
8 hours to break while another may have information that has to be protected for much
longer - and its usually not a government secret that requires real strength of
protection, but something like a drug formula or the design of a relatively mundane
product with a huge time critical potential market. Another company may be so cash poor
that they cant spend any more money on IT and still remain solvent, so they just have to
pray that they are never attacked.
So really the only intelligent way forward is Dynamic Risk Management rather than security
but then ITSEC and the other criteria do not provide adequate information to map onto a
DRM matrix and you have to fall back onto CRAMM or some other methodology that allows some
wild assumptions and assertions.
Reality of life is that an increasing number of companies will certify to ISO 9000 and BS
7799, mandate ITSEC E3 for IT products, register under Data Protection, and provide a very
good living for consultants but a very poor level of Risk Management. When something goes
horribly wrong no one gets fired because they ticked all the Standards boxes.
The life of ITSEC as a criteria is uncertain, but to some degree limited. This year
the UK ITSEC Scheme Secretariat, UKITSECSS, agreed a programme with the UK ITSEC
Commercial Licensed Evaluation Facilities, CLEFs (whoever said that govies couldnt
come up with short sexy titles for things), whereby they would begin running parallel
evaluations to Common Criteria on all products being submitted this year for ITSEC
The first US CC evaluation facilities were set up earlier this year and facilities had
already been running in Canada with, I think, the Milky Way firewall being the first
product to complete CC evaluation and be certified. With these North American evaluation
centres becoming available, and a high percentage of IT product originating in North
America, its highly likely that the demand on ITSEC CLEFs will reduce dramatically.
At present, the main advantage of ITSEC is that the mutual recognition agreements have
finally been signed between UK, Germany, and France and ALL evaluated products are listed
(including those that are restricted to government use only) in the product listings
issued by each of the three governments, irrespective of the location of the ITSEC CLEF or
the national Scheme Secretariat issuing the certificate.
It is an interesting note that the working parties originally took just under six months
to agree the criteria and the methods of evaluation, but its taken eight years for the
first three mutual recognition agreements to be developed by the lawyers and agreed by the
politicians. It is also interesting to note that it took us six months primarily because
the criteria had to be down graded to keep the US happy. The first cut criteria genuinely
reported Assurance, Integrity, and Availability as whole system measurements so that a
user could see not only that a given level of assurance was provided, but also that the
certified product would function effectively as an IT product. That was subverted somewhat
by the desire to change the F grading system to appear a close mapping to
Orange Book and to delete the resilience/availability reporting. The other change was to
drop the obligation on vendors to disclose their Target Of Evaluation documents.
After the US Senate Committee objected to simply adopting ITSEC (Not Invented Here
Syndrome) in place of TCSEC, NIST and NSA got together to produce FC-FIPS which was very
close to the original ITSEC criteria and included some Quality Assurance elements. Common
Criteria is the traditional political compromise that attempts to combine what are
perceived as the best features of TCSEC, ITSEC, and FC-FIPS but probably fails to achieve
that because its usually the lowest common denominator that gets selected.
For those who are interested in history, and differing views on the subject, I believe
that FC-FIPS is still available as a document and is still listed in the Federal
Information Processing Standard document lists of the National Institute of Standards and
Technology, NIST. I never bothered to get a copy of the general publication but Version 1
(Draft issue December 1992) was a cracking read and only a little over an inch thick for
Having been involved in TCSEC, UKLC, and ITSEC evaluations, I have some views on security
evaluations from a vendor perspective and some further views from a user perspective.
One interesting fact of TCSEC evaluations was that the evaluation work and issue of
certificate by NCSC was at no cost to the product vendor. The cost placed on the vendor,
and therefore eventually on the user, was that of engineering support costs and the cost
of employing a Vendor Security Analyst with an NCSC ticket who would present the product
to the evaluators and creatively describe a modified commercial product as being a product
developed from ground up to meet a precise US Fed specification - or to be flexible and
economic with the truth (thats a consequence of the Orange Book depending heavily on
Mil-Std 2167A which is fine for specifying army boots or aircraft carriers but not
designed for IT kit where there is no initial tight govt specification to which the
product is designed). That situation, together with the fact that products were only
evaluated in the (US) national interest, led to the most effective products
being strictly controlled and available only to a small range of US and some NATO
government agencies. It also resulted in certified product costing considerably more than
the commercial product from which it derived.
ITSEC was intended to change all that and, to a limited extent, it has succeeded. The
basic concept was that most ITSEC certified products would be available to anyone who
wanted to buy them and that this would create a huge market that would result in certified
products costing little more than standard commercial equivalents. Whats actually
happened is many vendors, particularly firewall vendors, are rushing to get a certificate
and then selling product that is either different from the evaluated product or is
deployed and configured in a different form so that it does not necessarily provide the
same assurance as the product which has been evaluated. Sometimes thats because no
one would seriously consider using the TOE configurations because they need something that
or processes information.
As an extreme example, I could go to PC World and buy a bog standard Intel clone PC with
the usual quality selection of packaged software and present that for
evaluation as a secure product. At least in theory, I could construct a TOE that claimed
the product to be E3 or E4 assurance if a particular installation process is followed.
This process requires all cables to be cropped as close to the case as is possible with
side cutters, an 8 lb lump hammer is then applied enthusiastically until the PC has been
reduced to a pile of fragments, and the results are then mixed with concrete and poured
into a suitable container. If I provided a mathematical model, maybe using Z or VDM and
NPL security clauses, I might even be able to produce an E6 TOE. Product cost increase
would be very small apart from the evaluation costs which would be excessive at E3, crude
extortion at E4, and astronomical at E6.
Therefore, an ITSEC Certificate is only really meaningful if you have a set of ITSEC
documentation, a copy of the product TOE and the time and skill to decide if the product
you are attempting to install actually matches all of that in the configuration which you
intend to employ. The main value of the certificate is that most vendors have actually
been forced to consider their product more carefully, and remove some of the bugs that
they are normally happy to ship out to customers in standard product. The documentation is
usually also a great improvement.For that privilege you will be paying a premium to cover
the very high cost of evaluation and certification because the vendor gets billed by the
CLEF for the evaluation, also has to carry evaluation support costs for his own
engineers, and the CLEF then passes on the costs they are charged by the Scheme
From a vendor perspective the CLEF charges can be excessive. I received a quote from one
CLEF (probably since renamed Creative Accounting Inc) for an evaluation that included a
four week visit to a Californian development lab to investigate their quality assurance
methods (although this facility held a variety of US and UK govt tickets for producing
very high spec products for aerospace defence application). The really interesting factor
was that the CLEF folk seemed particularly interested in visiting when it was good surfing
conditions. Of course if you ask a CLEF why they are obscenely expensive, they will blame
the charges from their Scheme Secretariat, who of course claim that their charges are
lower than they should be and its all down to the CLEF.
It is probably impossible to produce a really effective system of evaluation because the
moment a computer is installed people add products to it, change configurations, and
subvert any security that it had on day one. BS7799 isnt an answer because its also
a fudge to produce something that might be adopted generally. What could be done though is
for ITSEC, or any other scheme, to have product inter operability tables.
The main problem here is that most popular applications wouldnt ever get listed
because they just dont offer the integrity and assurance and the vendors probably
would not agree to providing the key vendor with enough access to their products to effect
An example of the approach would be where a vendor of a trusted operating system had his
product evaluated on a specific range of hardware and then added a selection of products
to provide a range of functionality popular with users. Each item would be evaluated and
then the integrated set of products would be evaluated to identify any covert channels or
other subversion of the assurance and integrity of the platform product. It would
almost certainly result in a range of availability levels depending on the mix of
applications and their configuration. Its also probable that there would be several
Assurance and Integrity levels for the same reasons.
It wouldnt provide total solutions but it could provide workstation and server kits
that could be bought as evaluated system modules and a risk policy could make assertions
and assumptions about the network links, based on the form of authentication and
ITSEC as it stands today could be used to evaluate in this manner but it would be horribly
expensive and the first attempts would produce a steep learning curve for the CLEF
selected to do the evaluation.
However, the main challenge would be in getting enough vendors together to work
co-operatively. Its not impossible because there are examples where a level of
co-operation has been achieved in the IT industry, notably in the binary compatibility
agreements that have made it possible for Intel and Intel clone hardware vendors to
produce platforms that MS operating systems and products such as SCO and Linux will
co-exist and inter-operate, providing a composite platform on which applications from a
host of other vendors can be mounted. There might be considerable scope for improvement,
but that co-operation has been largely responsible for the explosive uptake of computer
products during the last 20 years.