topic Re: Does Adaptive Exam Devalue the CISSP? in Exams

Does Adaptive Exam Devalue the CISSP?

gphalpin — Sun, 22 Apr 2018 15:09:44 GMT

Hi Everyone,

This may have been covered a few months ago when the news was first announced, but I just recently learned the CISSP exam became adaptive. Colleagues asked me about my exam experience. That was three years ago when the exam was up to 6 hours long and 250 questions. So I showed them the web site to go over the domains, exam info, etc.

I was very surprised to see the exam (English language) is now 100-150 questions. While the material is still demanding, I think the CISSP had a strong reputation as the premier information security certification because it was so rigorous with 250 questions. It was a long, tough exam. And people respected (sometimes grudgingly) those who passed.

At 100-150 questions, does this devalue the CISSP? The Security+ is 90 questions. People used to believe the CISSP was several notches above Security+. Now people might think the CISSP is just one notch above or lump them together.

I'm not trying to take anything away from those who passed the adaptive exam. I'm concerned about the long term implications this has on the value of the CISSP certification in the eyes of IT security professionals.

Thanks,

Greg

Re: Does Adaptive Exam Devalue the CISSP?

Jaswhee — Sun, 22 Apr 2018 19:10:53 GMT

Greg,

I would invite you to read up on the CAT (Computer Adaptive Testing) . Basically the CAT in most cases is able to determine within 95% statistical confidence that you would be a pass or fail even if you had 250 questions. I do not believe that more candidates are passing as a result of the CAT. I personally know quite a few people who have failed the CAT.

Jason

https://www.isc2.org/Certifications/CISSP/CISSP-CAT

Re: Does Adaptive Exam Devalue the CISSP?

perryjurancich — Sun, 22 Apr 2018 21:36:37 GMT

I would suggest that if the fail rates go down, that indicates 1. the CAT algorithms are not doing a good job, or 2. the test is effectively easier thereby reducing (IMHO) the value to those who have passed. I can tell you from several different types and levels of tests (CISSP, CCSP, GSEC, PMP, BlackBelt, etc) CISSP used to be a good challenge, hope it stays that way.

Just my $.02

Re: Does Adaptive Exam Devalue the CISSP?

CEMyers — Mon, 23 Apr 2018 09:15:09 GMT

We should all bear in mind that quality is always a better yardstick than quantity.

Were the exam to be a straight set of questions there might be stronger reason for doubt, but the adaptive nature of this framework is there specifically to address the quality question and to maintain the current high standard of the qualification.

Nothing worthwhile is ever easy, remember that, when comparing with other qualifications. Also remember the qualification is not just about passing the exam but also about having the (appropriately recognised) experience to support the knowledge. Knowledge supported by experience for me is always the best form of assessment. Remember the human race moves forward through learning from its failures as much as from its successes.

Re: Does Adaptive Exam Devalue the CISSP?

MiniMe — Tue, 24 Apr 2018 12:19:02 GMT

I am willing to bet a person, who thinks CAT makes CISSP exam easier, has never taken a CAT exam in their life.

I have recently taken the CISSP in CAT format. Having taken GRE when I applied for grad studies, I was familiar with the CAT concept.

If I had an option, I'd have taken the 6 hour linear exam where I could go over "A" number of very easy, "B" number of easy, "C" number of hard, and "D" number of very hard questions.

1. If you're not sure about an answer, you can skip the question... you do not have that luxury with CAT format.You have to answer the questions in the order presented and your answers are final.

2. More than often, in an exam with hundreds of questions, you end up inferring the answer of some previous questions just by reading the proceeding questions. Well, once you answer a question, you cannot go back in CAT format.

3. Going through a hard question may discourage you but all you need to do is reading the next question to gain your courage back. Well, in CAT format, if you answer a hard question correctly, you get a "harder" question. And if you get an easy question after a hard one in CAT, you know that you screwed up the previous one.

4. You might be really good at time management and you can easily come up with a plan when you're taking a linear exam. Guess what, you have the same amount of time whether you're done at question 100 or need to answer 50 more in CAT format.

Having said that, maybe it's a lot easier for some folks to take CAT exams... for me, linear tests are very predictable and a professional test taker can easily increase his/her chance of acing the test by preparing for the exam itself as well as the content of the exam.

Thanks,

JohnZ

Re: Does Adaptive Exam Devalue the CISSP?

gphalpin — Tue, 24 Apr 2018 12:51:43 GMT

Thanks for the responses, everyone.

The question is not whether the exam is easier or harder. The question is whether the CISSP will have the same perceived value to other IT pros, IT security pros, and hiring managers now that it's a shorter exam?

Anyone who has a CISSP will know that the exam is still very hard. Years of experience and knowledge are necessary. It also takes many months to study for the exam. You have to be endorsed, etc.

But people who are not yet familiar or as familiar with the CISSP may wonder why it stands out since it's not much longer than other exams. In fact, one of my colleagues who expressed interest it in said it looks a little harder than Security+ but not by much. I don't want the CISSP lumped in with Security+.

Again, I'm concerned the CISSP may just be another cert in five years, not the premier IT security certification.

Re: Does Adaptive Exam Devalue the CISSP?

mgoblue93 — Wed, 25 Apr 2018 14:11:46 GMT

No. What devalues the CISSP is the people who have it, take a title of cybersecurity engineer, and then do not do any engineering work.

My personal opinion is adaptive exams are harder. Someone sitting for that test pretty much has to be mistake free. In the previous test, one could miss a question and recover. In adaptive exams, you miss a question and you're going to see more of the same material. 2 thumbs up to people who pass (any) adaptive exams!

In general, I don't think comparing old version to the new version's question count or their duration and somehow equating that to a degree of difficulty is applicable or productive to the community.

I have witnessed in the past, people throw shade at others because their CISSP number is too high -- the assumption is they got into the game late or just because of the gov't and therefore aren't a real CISSP. To me, I interpret some of this old test/new test thing as a continuation of the "your number is too high" silliness. Adaptive tests are no joke.

Re: Does Adaptive Exam Devalue the CISSP?

CISOScott — Tue, 24 Apr 2018 15:31:37 GMT

I don't feel like it devalues it. Adaptive tests are supposed to ask progressively harder questions, so if you know about encryption and it's correct application to a certain situation, whether you get asked 3 questions or 30 doesn't necessarily make you more knowledgeable in that situation.

Re: Does Adaptive Exam Devalue the CISSP?

Beads — Wed, 25 Apr 2018 15:53:42 GMT

ISC(2) doesn't publish the pass rate so their is no way of understanding if the test is easier, harder or about the same. We simply don't have enough information to make that claim one way or another. I will direct our readership to the one clue we do have access - exam numbers.

This will be a bit controversial so if your easily offended please pass on reading below this line.

Consider the year and month you received your official membership number and the number of exams taken by all candidates before. This will become your baseline. Now advance a few years and think about those resumes with membership numbers from more recent members. Those numbers go up not down. If you have a really low number say in the 50,000s from 200x and compare to a very recent member with a membership over three-quarters of a million. Well, you can compile a pretty rational idea as to if people are passing more frequently than in the past.

Its just maths and some observation skill but I have confidence you can make an argument either way.

We have over 125,000 members and can remember when we hit 50,000. The later being a huge relief and celebratory moment for the organization.

Re: Does Adaptive Exam Devalue the CISSP?

meincke — Sat, 05 May 2018 03:38:12 GMT

I wonder about that too.

The exam was difficult for me. Not due to the difficulty of the questions, I was well prepared by the time I took the exam, I answered the vast majority of them with a high degree of confidence. I didn't rush, I read every question carefully, weighed each answer and came to a conclusion I was happy with. After answering each one at a time I had 11 minutes left on the clock. I didn't go back to review my answers, I was too tired, and I was reasonably sure that a review wouldn't turn up any better answers.

I felt like a ran a mental marathon, I certainly had the knowledge to pass but there was no getting around 6 hours of testing. That was the hardest part. I feel like a CAT test would've been a breeze for me or anyone else who was well prepared.

Re: Does Adaptive Exam Devalue the CISSP?

denbesten — Sat, 05 May 2018 16:11:34 GMT

@Beadswrote:

your official membership number ... compile a pretty rational idea as to if people are passing more frequently than in the past.

The member number is issued when one signs up for the web site and represents everyone who has one or more (ISC)² certificates, has ever scheduled an exam, or even might just be considering it.

You can use the member number to estimate the age of somebody's oldest cert and you can use it to gauge the overall level of interest in (ISC)² certification, but it does not help estimate how frequently people are passing the exam. You would be better off tracking the published member counts to estimate that.

Re: Does Adaptive Exam Devalue the CISSP?

Lamont29 — Sat, 05 May 2018 21:27:29 GMT

The

@gphalpinwrote:
Hi Everyone,

This may have been covered a few months ago when the news was first announced, but I just recently learned the CISSP exam became adaptive. Colleagues asked me about my exam experience. That was three years ago when the exam was up to 6 hours long and 250 questions. So I showed them the web site to go over the domains, exam info, etc.

The biggest gripe I see online is with people who say that they studied for the 250 question linear exam but took the CAT exam instead and failed. You got 6 hours on the old format and 3 hours on the CAT format. On the old format, you can go back and check or correct something, on the CAT format, it's a 'one and done' deal. I guess because I know how CAT exams are statistically measured, I don't really get caught up in the 'watered down' theory.

ISC2's mission is to maintain the integrity of the testing for CISSPs. If the CAT format made things 'easier' then I know that I would have never bothered to sit for the exam. The critical thinker may interject and ask critical questions such as: 'Why is it easier?' 'Has the person who made such a judgement taken both versions?' What's the failure rate between the CAT format and the linear exam?

Based on anecdotal evidence that I've encountered, there are even MORE disgruntled test failures now than when the linear-only format was in place.Just my thoughts.

Re: Does Adaptive Exam Devalue the CISSP?

Lamont29 — Sat, 05 May 2018 21:40:23 GMT

@MiniMewrote:

Having said that, maybe it's a lot easier for some folks to take CAT exams... for me, linear tests are very predictable and a professional test taker can easily increase his/her chance of acing the test by preparing for the exam itself as well as the content of the exam.

Thanks,
JohnZ

JohnZ,

You have taken the words right out of my mouth! I try explaining this concept to people who are not professional students like we are and the message never seems to get over. Exhibit A: I have NEVER taken a linear test and failed - NEVER! On a linear exam, you can always brain dump. You can go back and recheck a question and change an answer. You can always get cues from other people who shared their experience with the linear test, and all of this aggregate information allows a person who is a professional test-taker to have a much easier time with linear exams - they can be 500-question tests and it wouldn't make a difference.

CAT examination all but nullifies the ability of brain-dump test taker success.

Re: Does Adaptive Exam Devalue the CISSP?

Lamont29 — Sat, 05 May 2018 21:44:54 GMT

@denbestenwrote:
The member number is issued when one signs up for the web site and represents everyone who has one or more (ISC)² certificates, has ever scheduled an exam, or even might just be considering it.

You can use the member number to estimate the age of somebody's oldest cert and you can use it to gauge the overall level of interest in (ISC)² certification, but it does not help estimate how frequently people are passing the exam. You would be better off tracking the published member counts to estimate that.

Great observation and response...

Re: Does Adaptive Exam Devalue the CISSP?

meincke — Sat, 05 May 2018 23:01:08 GMT

CAT in no way nullifies brain dumping. CAT still draws from a pool of questions. If somebody gets their hands on the test bank then they can just as easily memorize the questions as they would with a traditional test. Only exam security and frequent rotation of questions can combat dumping.

Re: Does Adaptive Exam Devalue the CISSP?

meincke — Sat, 05 May 2018 23:18:04 GMT

Here's the thing I think proponents of CAT are missing; the 6-hour exam was a right of passage. The length of the exam was something to be respected. I could tell someone who didn't know anything about IT or security that I had to sit for a 6-hour test with only 90 seconds to spare between questions and they would genuinely be impressed.

It's not nearly as impressive to say that your exam ended early because an algorithm determined you'd pass a longer test.

Re: Does Adaptive Exam Devalue the CISSP?

Lamont29 — Sun, 06 May 2018 00:49:09 GMT

@meinckewrote:
CAT in no way nullifies brain dumping. CAT still draws from a pool of questions. If somebody gets their hands on the test bank then they can just as easily memorize the questions as they would with a traditional test. Only exam security and frequent rotation of questions can combat dumping.

I will add that I am in no way sure that the linear test was favorable to brain dumps because I never taken the linear exam. However, ALL of the people that I know who took the linear exam suggested to me that the test banks and brain dumps were viable resources for passing the linear exam when they took it. Maybe my experience might have been different. I do know that I have NEVER failed a linear exam (I am at 100%) so long as I studied similar questions from test banks - akin to brain dumps.I have on the other hand experienced difficulty on CAT exams, as they are in my opinion tougher than linear exams. You will NEVER find similar questions from a test engine or other resource on the CISSP CAT exam (based on my experience). Nearly every test question was based on my experience, which if that makes it easy then, I guess it was easy for me. I found the test to be uneventful. I have other colleagues who have taken this test several times without success - both CAT and linear format. Furthermore, ALL of the referenced individuals have earned advanced degrees.

Lastly, I don't know of anyone who passed both versions of the exam. In this case, this thread is for bravado and very opinion based.

Re: Does Adaptive Exam Devalue the CISSP?

meincke — Sun, 06 May 2018 04:18:51 GMT

@Lamont29wrote:
I will add that I am in no way sure that the linear test was favorable to brain dumps because I never taken the linear exam. However, ALL of the people that I know who took the linear exam suggested to me that the test banks and brain dumps were viable resources for passing the linear exam when they took it. Maybe my experience might have been different. I do know that I have NEVER failed a linear exam (I am at 100%) so long as I studied similar questions from test banks - akin to brain dumps.I have on the other hand experienced difficulty on CAT exams, as they are in my opinion tougher than linear exams. You will NEVER find similar questions from a test engine or other resource on the CISSP CAT exam (based on my experience). Nearly every test question was based on my experience, which if that makes it easy then, I guess it was easy for me. I found the test to be uneventful. I have other colleagues who have taken this test several times without success - both CAT and linear format. Furthermore, ALL of the referenced individuals have earned advanced degrees.

Lastly, I don't know of anyone who passed both versions of the exam. In this case, this thread is for bravado and very opinion based.

Completly anecdotal but I had the opposite response. People I've known who are open about dumping have failed the CISSP after trying to use a dump as their only study method.

A CAT isn't going to change this, the software isn't generating questions on the spot, there's still a pool of predetermined questions. The only difference is that the questions are ranked from easy to hard and you might not need to answer every question once you hit that magic number for a given category.

Re: Does Adaptive Exam Devalue the CISSP?

Lamont29 — Sun, 06 May 2018 04:35:20 GMT

@meinckewrote:
Completly anecdotal but I had the opposite response. People I've known who are open about dumping have failed the CISSP after trying to use a dump as their only study method.

A CAT isn't going to change this, the software isn't generating questions on the spot, there's still a pool of predetermined questions. The only difference is that the questions are ranked from easy to hard and you might not need to answer every question once you hit that magic number for a given category.

I completely agree. Your anecdotes are just as plausible as the one I gave. Now, on the other hand (I was just thinking of this experience) when I was the deputy in charge of battalion operations, the operations officer had everyone working long hours. I mean 16-18 hours on most days. That person had to go on mission which left me in charge. I told my executive officer that I was going to add efficiencies to our processes and get everyone home at a decent hour. To make a long story short (see the pun?), I improved our battalion's overall performance and got everyone (including the commander) home in time for them to have dinner or a life with their families. So longer doesn't equal to smarter or better performance. My processes were adopted even after the senior operations officer came back. He may never tell me - but yes, I think he was happy as well.

I tend to lose interest after about an hour and a half of reading. Now, give me a couple of 200 mg caffeine pills and I will jump all day and into the night. So anyone sitting for three hours could have just as well sat for 6 hours. I did not use all 3 hours - and I would avoid that at all costs knowing that my best performance is going to be around 1 1/2 to 2 hours. But if I had to, there's would have been no cheating in me taking the caffeine pills if the exam would have stayed linear.

Re: Does Adaptive Exam Devalue the CISSP?

denbesten — Sun, 06 May 2018 04:50:22 GMT

@meinckewrote:
CAT in no way nullifies brain dumping. CAT still draws from a pool of questions. If somebody gets their hands on the test bank then they can just as easily memorize the questions as they would with a traditional test. Only exam security and frequent rotation of questions can combat dumping.

No question that brain dumping remains possible, but it is also less fruitful for one simple reason. Now they will only get 100 to 150 questions from the bank, whereas they linear test guaranteed them 250 questions. So, the question bank is a bit more protected than it was.

There are a few more techniques that can be used to defend an exam, as this article describes.