Does Adaptive Exam Devalue the CISSP?

[gphalpin]

Hi Everyone, 

This may have been covered a few months ago when the news was first announced, but I just recently learned the CISSP exam became adaptive. Colleagues asked me about my exam experience. That was three years ago when the exam was up to 6 hours long and 250 questions. So I showed them the web site to go over the domains, exam info, etc. 

I was very surprised to see the exam (English language) is now 100-150 questions. While the material is still demanding, I think the CISSP had a strong reputation as the premier information security certification because it was so rigorous with 250 questions. It was a long, tough exam. And people respected (sometimes grudgingly) those who passed. 

At 100-150 questions, does this devalue the CISSP? The Security+ is 90 questions. People used to believe the CISSP was several notches above Security+. Now people might think the CISSP is just one notch above or lump them together.  

I'm not trying to take anything away from those who passed the adaptive exam. I'm concerned about the long term implications this has on the value of the CISSP certification in the eyes of IT security professionals.  

Thanks,

Greg

[Lamont29]

---

@denbestenwrote:

@meinckewrote:

CAT in no way nullifies brain dumping. CAT still draws from a pool of questions. If somebody gets their hands on the test bank then they can just as easily memorize the questions as they would with a traditional test. Only exam security and frequent rotation of questions can combat dumping.

---

No question that brain dumping remains possible, but it is also less fruitful for one simple reason. Now they will only get 100 to 150 questions from the bank, whereas they linear test guaranteed them 250 questions.  So, the question bank is a bit more protected than it was.

There are a few more techniques that can be used to defend an exam, as this article describes.

---

I should have know never to convey 'never' as there's always going to be a work-a-round somewhere. I just think that ISC2 does a great job of mitigation. I used Shon Harris and ISC2 approved materials. I encountered no similar questions from any of the practice test questions. But knowing what the subject matter is, is always a great guide. But if you have the experience, you should be fine with passing this test. I just didn't see the big deal. I am glad that I only had to sit for the 2 hours (maybe less) but I could have certainly enhanced my tendency to lose interest with a couple of caffeine pills or a couple of strong lattes if the 250-question test was still a requirement.

Nice article attached as well. I think that it's a good thing if all the major vendors collaborated to thwart the cheating on industry tests. It hurts me as an IT Security professional.

[Baechle]

Raymond,

---

@meinckewrote:

Here's the thing I think proponents of CAT are missing; the 6-hour exam was a right of passage. The length of the exam was something to be respected. I could tell someone who didn't know anything about IT or security that I had to sit for a 6-hour test with only 90 seconds to spare between questions and they would genuinely be impressed. 

It's not nearly as impressive to say that your exam ended early because an algorithm determined you'd pass a longer test.

---

I agree with you whole heartedly.  There are many things that I have participated in that I feel were a rite of passage from the paper CISSP, to courses I took in the military before they went from practical to CBT/CAT.  The experience of the paper test for the CISSP (and concentrations) unreasonably bled into to the perception of how grueling the material knowledge itself was.

When I take a deep breath and let that nostalgia subside, I come to my senses. 

There is no CISSP domain requirement that we have to be able to hold our pee, grit out arthritic pain, or stave off diabetic distress for 6 hours. 

When the CAT eliminates questions from the test and the test-taker reaches a passing score quicker, it was the easier questions that were skipped.  What is the point of asking a test taker several leading questions from “2+2=x” to “y=mx+b”, when they are able to jump right to solving questions involving Multivariate Quadratics?

Sincerely,

Eric B.

[meincke]

---

@Baechlewrote:

Raymond,

---

@meinckewrote:

Here's the thing I think proponents of CAT are missing; the 6-hour exam was a right of passage. The length of the exam was something to be respected. I could tell someone who didn't know anything about IT or security that I had to sit for a 6-hour test with only 90 seconds to spare between questions and they would genuinely be impressed. 

It's not nearly as impressive to say that your exam ended early because an algorithm determined you'd pass a longer test.

---

I agree with you whole heartedly.  There are many things that I have participated in that I feel were a rite of passage from the paper CISSP, to courses I took in the military before they went from practical to CBT/CAT.  The experience of the paper test for the CISSP (and concentrations) unreasonably bled into to the perception of how grueling the material knowledge itself was.

When I take a deep breath and let that nostalgia subside, I come to my senses. 

There is no CISSP domain requirement that we have to be able to hold our pee, grit out arthritic pain, or stave off diabetic distress for 6 hours. 

When the CAT eliminates questions from the test and the test-taker reaches a passing score quicker, it was the easier questions that were skipped.  What is the point of asking a test taker several leading questions from “2+2=x” to “y=mx+b”, when they are able to jump right to solving questions involving Multivariate Quadratics?

Sincerely,

Eric B.

---

The goal is to be a security practitioner. Disaster doesn't only strike when you are well rested with an empty bladder. The ability to still correctly respond to a situation when you're not at 100% SHOULD be part of the exam. Otherwise, the CISSP just becomes a piece of paper validating something you already know, and how is that any different than Security+? 

[Baechle]

Raymond,

---

@meinckewrote:
The goal is to be a security practitioner. Disaster doesn't only strike when you are well rested with an empty bladder. The ability to still correctly respond to a situation when you're not at 100% SHOULD be part of the exam. Otherwise, the CISSP just becomes a piece of paper validating something you already know, and how is that any different than Security+? 

---

I appreciate your passion, but I have to point out that not every CISSP holder is in a Disaster Recovery work role.  Additionally, not every organization requires such a Disaster Recovery response as to require people to come in to work in the middle of the night while in the midst of suffering the flu.

It’s the position itself that may have a medical or health requirement.  But, at least in the USA, you’ll have to tip toe around the Americans with Disabilities Act.  In my occupation there is a medical requirement and even a maximum entry age coupled with a mandatory retirement age.  And in fact, in some of our job qualifications we are required to be physically stressed by running for five minutes and doing a series of push-ups before taking the test.  But it’s the actual work role that has that medical requirement.  I could apply a significant portion of my  CISSP CBK knowledge in a work role or occupation that doesn’t have the medical requirement.

Sincerely,

Eric B.

[CZ]

This is an interesting question: I have taken complex adaptive and linear tests in the past (ECNE/MCNE as well as CISSP) and I remember the adaptive tests for the ECNE. The problem was you had to answer a certain number of questions at a certain level to pass, and you could usually figure out where you were on the spectrum if the questions suddenly got easier. So there are tells you can use to help on the test.

My guess is they pull from all domains and might rank difficulty within a domain (ie: you have to get a certain level of question difficulty on each domain) which would dilute that somewhat, depends on how they formatted the test.

I liked what they did with the traditional linear test format when they salted the questions with 10 or so questions that had no right answer. Because of that if you tried the classic trick of "they were asking me a question like this before, therefore one of the answers there relate to the question/answer set here" you could find yourself using one of these "no right answer" questions as the basis, which would give you a very wrong answer.

I remember spending 3 hours on the test, then going to take a nap because some of the questions simply seemed idiotic. Then when I woke up I realized they were probably the blind questions, so I ran through my answers, submitted, and passed (for all I know I got 70.01% 🙂

Not being able to go back and change answers is not a handicap, usually the first answer you pick is the best one (assuming you know the info) and it is very easy to spend too much time thinking about the answer and rationalizing a wrong choice instead.

But still, I prefer the long format. There are a lot of domains to cover in the CISSP, it's not like a more focused certification and my concern would be that a right guess on a hard question could help you more than hurt especially if there is a limited pool to cover all domains.

Or they could ask you 100 questions on class K fire extinguishers.

Chris Z

Novell Enterprise CNE

Novell Master CNE (Infrastructure/Advanced Access)

CISSP

Fun at parties

[skvivian]

I suspect that, for those who don't really understand the CISSP, a "shorter" exam might lead them to believe it's not as rigorous as it used to be, or that the cert has been watered down somehow. 

I also don't know that you can predict whether the exam has become easier based on the number of holders, because back when I got my cert (on the long-form exam), not as many people knew about the cert yet. As awareness has spread, it's likely that there are increasing numbers of people trying the exam, so the pass rate could be the same, increasing, or dropping, but there's no way to measure the pass rate without knowing the attempt rate.

I will throw in that I personally hate CAT exams, simply because they leave no room for error and produce their result at least partly based on luck. What's to say that the algorithm won't hit that one question you aren't sure about, at a critical junction, and dump you into the "needs more analysis" branch? Or alternately, they might hit one or two questions at critical junctions that someone happens to know perfectly even though they don't have a good handle on the rest of the material, and they get through the exam when they probably shouldn't have? With the long-form exam, you might not know a few answers, but are able to demonstrate that you know most of the material, and will get through, or you might know only a few things but not know all the material, and end up failing because it really tested you on *everything* rather than some selected subset.

And I also think that the long-form exam works better on a mental level. The CAT process adds stress, because not only do you have to be able to answer the material, but you are constantly worrying that you'll miss a couple of questions at critical points and end up blowing the exam, not to mention the shorter time limit; even with fewer questions it feels like there's less time per question. At least with the long form, you know that you can miss a couple of questions and it won't change the rest of the test.

[Baechle]

---

@skvivian wrote:

I will throw in that I personally hate CAT exams, simply because they leave no room for error and produce their result at least partly based on luck. What's to say that the algorithm won't hit that one question you aren't sure about, at a critical junction, and dump you into the "needs more analysis" branch? Or alternately, they might hit one or two questions at critical junctions that someone happens to know perfectly even though they don't have a good handle on the rest of the material, and they get through the exam when they probably shouldn't have? With the long-form exam, you might not know a few answers, but are able to demonstrate that you know most of the material, and will get through, or you might know only a few things but not know all the material, and end up failing because it really tested you on *everything* rather than some selected subset.

And I also think that the long-form exam works better on a mental level. The CAT process adds stress, because not only do you have to be able to answer the material, but you are constantly worrying that you'll miss a couple of questions at critical points and end up blowing the exam, not to mention the shorter time limit; even with fewer questions it feels like there's less time per question. At least with the long form, you know that you can miss a couple of questions and it won't change the rest of the test.

---

Did the scoring change?  If I remember correctly, the exam required a high score on only two domains with an overall minimally-passing score on the whole exam.  I took this to mean, you could outright fail several domains, as long as your kept an overall passing score and did really well on two domains.

[skvivian]

I really don't remember. I took the exam some 12 years ago. 

[Lamont29]

---

@skvivian wrote:

I suspect that, for those who don't really understand the CISSP, a "shorter" exam might lead them to believe it's not as rigorous as it used to be, or that the cert has been watered down somehow. 

---

Here’s what I know… a pretty intelligent guy who holds a senior-level information security position has taken both versions of this exam (4 tries, two each) and has failed all four. Now you might be saying to yourself, “Well, he couldn’t be that smart!” And if you’ve known this guy as I do, you would never arrive at that opinion. Truth is, some people just don’t test well. Me on the other hand, I have never sat for a test for which I have put in serious enough study time and failed. But all that really says about me is that I have developed good study habits that seem to work for me – not that I am better than anyone else.

I give a lot of push-back to those who would suggest that the CAT version is somehow ‘easy’ compared to the legacy linear tests. You got more time on the linear 250-question test, and there was the added advantage of going back and changing your answer on the linear test. All of these advantages have been taken away in ISC2’s switch to the CAT format.

Furthermore, I see no reduction in the amount of ‘whiners’ since the CAT format has been offered. This tells me that the CISSP certification maintains its integrity!

[unixgeek21]

I completely agree with what Lamont29 had said...my boss/supervisor is literally a genius...and yet he failed even on the entry level CCNA (Cisco Certified Network Associate) exam not because he didn't know the answer but he over-analyzed at every possible questions and answers he encountered.   With his genius mind, he would take things apart saying that this answer would not be right, etc..and in the end, he run out of time.    Even if he's given all the time in the world, he will probably never come to conclusion as to why this answer would be sufficient or correct.

Having said that, I recently took and passed the CISSP exam. I have to say that it's one of the difficult exams I've taken in life if not the most difficult!   I got tons of questions where the answers were all wrong or all correct in some sense and I'm to choose which is the least wrong (former) or the most correct (latter).   Without any experience and dedication to studying, I couldn't have possibly passed the exam.  The exam tested me on my ability to make the decision given a situation where there's no good or right answer but to choose the "best" answer given the possible choices/answers and in the allotted time.   If one can't make any decision, it doesn't matter if one is given 3 or 6 hours...

ISC2 has done a great job in protecting the integrity of the exam and making sure that if one pass the exam, one is truly worthy (in my opinion) to be called a CISSP!   I've never been so proud and grateful for this very emotional journey...and very humbled to be part of this club.