topic Verification fails - well, sometimes it does.. in Member Support

Verification fails - well, sometimes it does..

fortean — Fri, 17 Jan 2020 21:20:50 GMT

Folks,

When we set up the Dutch Chapter (https://www.chapter.isc2.nl) I wrote a set of PHP programs which implement our membership registration system. It's a simple system; you'll surf to the registry (https://registry.chapter.isc2.nl), fill out a couple of forms and (given that the data is consistent and can be verified) are registered as a member of the Dutch Chapter.

If you specify an (ISC)² membership number during registration my software verifies that you are actually an (ISC)² member. Such functionality is normally provided over a well regulated application program interface (API), but (ISC)² did not provide such an API. So, I ended up writing my own kludge, which used the on-line verification page. But kludge or not - it worked just fine for years.

Then (ISC)² migrated to Okta/Salesforce. They use a totally different approach, so my kludge stopped working and automatic verification did not work anymore. Dang!

We tried to get some response from (ISC)², but to no avail. Then one of our Board members "did some experiments" (hacked stuff) and he figured out how to do verification in the new situation. It proved that, intentionally or by accident, the new verification page DID provide an API! Yay!

So, I set out to write a proof of concept that used that API. Worked like a charm! I enthousiastically reported back to (ISC)² that I was happy as a bear in a berry patch, and all was well.

A few days later I found the time to integrate my POC into the existing software. But - it did not work. Gnashing of teeth. I figured that I probably had done something wrong during the integration, so I went back to the POC and tried that again. Ow.. that did not work anymore either! More gnashing of teeth.

I had used my own data (last name and membership number) to test the software. On a hunch I replaced that data with some other chapter member's last name and membership number - and lo! IT WORKED..??

WT..?

I now started doubting the world: according to the API, I was not registered anymore!

I fired up my browser, hurried over to the 'normal' (ISC)² verification site, typed in my last name and number - no cigar! I, the President of the Dutch Chapter of (ISC)², proud member of the CAC, and a hallmark of integrity and honesty, having duly paid my fees and atop of a sea of CPE's - was cast out! ..

O, the pain, o the horror! Hastily and with pounding heart I logged in into the (ISC)² member portal. Thank Goodness - I could still log in! And all was well there: member in good standing, all requirements fulfilled, ocean's of CPE's..

SO WHY THEN DOESN'T THE DARNED VERIFICATION PAGE SHOW MY DETAILS?

Can anybody shed any light on this?

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Sat, 18 Jan 2020 09:21:10 GMT

I can't shed any light on why it happens, but I have also noticed the new verification page seems to have intermittent issues.

On at least two different occasions now over the past four weeks or so when I have come to use it I received a "no results found" message, so I checked it using my own details and surprisingly also received the same message.

On each occasion normal service was resumed within a few hours.

Re: Verification fails - well, sometimes it does..

fortean — Sat, 18 Jan 2020 20:44:28 GMT

$ php ./proof-of-concept-ISC2-API.php 
Heinrich holds 2 accreditation(s): 
1: CISSP
1311717600 from 
1596146400 until 
2: CCSP
1500588000 from 
1596146400 until 
from 1311717600 to 1596146400

So, yes, it seems to work again

But.. for how long? Oh, come on, there must be some logging or other tracing available that helps a good sysadmin over there find out what causes this.. verification of membership should not result in having to guess what it is: "member is not in good standing" or "member is in good standing but our software is not" ..

Re: Verification fails - well, sometimes it does..

fortean — Sun, 26 Jan 2020 14:38:18 GMT

Folks,

much to my dismay I have to report that the verification site is failing again, and again is similar ways it failed before.

I can't get my head around it: how is it that this is not seen as an urgent matter? Or, if it IS seen as an urgent matter, I would appreciate to hear some update: is action being taken? Is the firing squad already assembled? Are the criminals, that deny an honourable member of (ISC)2 in good standing his verification sent to serve triple-lifelong sentences yet?

Tongue in cheek, of course. But seriously, I can't rely on the site anymore to check if somebody is properly registered and a member in good standing. Not with, not without the new (much appreciated!) API. Nor can employers that want to check if a person really IS certified.

That, in my not so humble opinion is the core business of (ISC)2: certifiy people and ensure that sufficient mechanisms are in place to verify if a member is (still) in good standing. It seems we are failing our core business.

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Mon, 27 Jan 2020 09:37:57 GMT

As of this moment the verification page is working for me.

Given this doesn't affect me that much and I know how to work around it when it does I haven't been bothered to report it to Member Services so I'm not sure if they're even aware of the issue - maybe drop them an email if it's causing you major problems.

In the meantime can you add something to your script where a null result for a new member you're checking triggers a check of a known good member? If that also comes back with a null result you know the system is down and to try again later.

Re: Verification fails - well, sometimes it does..

fortean — Tue, 28 Jan 2020 22:05:32 GMT

Well, whenever I try to verify myself, and it fails, I will indeed try to verify another member, and it typically will work. So, it may well be that there is something wrong with just my account, or a number of accounts. In any case, as a member in good standing, one should ALWAYS either be shown a page that the site is down, or some other error page - or be shown the proper result.

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Wed, 29 Jan 2020 21:48:44 GMT

@fortean wrote:
...
...
In any case, as a member in good standing, one should ALWAYS either be shown a page that the site is down, or some other error page - or be shown the proper result.

I wholeheartedly agree with you.

So, what did Member Services say when you reported it?

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Thu, 30 Jan 2020 10:18:10 GMT

@fortean - Just in case you haven't reported this issue to Member Services, I have now sent them an email about it referencing this thread.

If you have specific times when your script was failing due to the issue it would be helpful if you could post them please.

Re: Verification fails - well, sometimes it does..

fortean — Fri, 31 Jan 2020 10:43:03 GMT

Thanks for mailing Member Support. I have treated this as a Chapter issue, hence mailed Jayda and Paula about this. It no doubt has their attention. I have set up a coarse monitoring system that fires a query to the API each hour. Will let that run for a week, and see what it says.

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Tue, 18 Feb 2020 16:12:34 GMT

Currently getting the "no results found" message when trying to check my own status!

Timestamp: 15:42 GMT on 18/02/2020

Re: Verification fails - well, sometimes it does..

fortean — Wed, 19 Feb 2020 12:02:57 GMT

Well, as promised, I did set up a crude monitor for this functionality. It tries to verify my own verification each hour and logs the results.

According to my logs, the system failed on Monday, February 17th, somewhere between 14:53:02 CET 2020 and 15:53:01 CET 2020. It has not been restored yet.

BTW, it may be a coincidence, but it struck me that 1500 CET is 9 AM in the US (EDT). One could imagine a mr Parkinson, arriving at the office at 9, reading a mail that says there is something wrong with the system, and that the old version needs to be restored, which Parkinson dutyfully does. Only to re-introduce the nasty bug that ensures that the system does not work anymore..

However, can somebody pretty please look into it and resolve the matter, preferably once-and-for-all..?

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Wed, 19 Feb 2020 13:45:18 GMT

@fortean

Similar to what I have seen before, the system was reporting the correct results a couple of hours after I made my last post, and is currently reporting the correct results - strange your systems still shows it as being down!

Re: Verification fails - well, sometimes it does..

fortean — Fri, 21 Feb 2020 21:41:43 GMT

The system has been disfunctional from Mon Feb 17 15:53:01 CET 2020 until Wed Feb 19 16:53:02 CET 2020. It resumed working on Wed Feb 19 17:53:01 CET 2020

It was down for about 50 hours.

It currently is still working, I'm still actively monitoring it to see if I can detect some pattern.

Re: Verification fails - well, sometimes it does..

fortean — Wed, 25 Mar 2020 20:59:40 GMT

Just an update: the verification site seems to be mostly working now. I'm running a script each hour, which does a query over the API that the verification site offers. These are the outages that I registered during the last month:

Sun Mar 15 15:53:01 CET 2020
Fri Mar 20 21:53:01 CET 2020
Sun Mar 22 13:53:01 CET 2020
Mon Mar 23 04:53:01 CET 2020
Tue Mar 24 14:53:01 CET 2020

Given these results, I re-instated the automatic checks during registry and also ran a batch to check the validity of our members data. I had some outages during that batch, but all in all I have decided to label it "sufficient for use" for now.

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Fri, 27 Mar 2020 13:40:56 GMT

@fortean

Hey Heinrich,

Just to let you know, I received a message from Member Services yesterday about the ticket I opened with them on this issue.

They were just checking with me to see if I had experienced the issue recently, as they believe it's now resolved.

I told them I hadn't run into the issue for at least a couple of weeks, and, as you have also reported no serious issues recently, it looks like the verification tool should operate more consistently now - great news!

Re: Verification fails - well, sometimes it does..

fortean — Wed, 22 Apr 2020 14:46:44 GMT

Well, according to my logs, it isn't that stable at all, assuming that the network connection is not the culprit. On these dates the connection failed at least once:

Fri Mar 20
Sun Mar 22
Mon Mar 23
Tue Mar 24
Sat Mar 28
Tue Apr 14
Wed Apr 15
Thu Apr 16
Wed Apr 22

Re: Verification fails - well, sometimes it does..

AlecTrevelyan — Wed, 22 Apr 2020 17:56:41 GMT

Yeah, it doesn't have perfect availability that's for sure!

I created my own system to monitor this too on the 31 March 2020.

Since then I have noted the following:

20200401203301 Verification check failed to run correctly
20200402173301 Verification check failed to run correctly
20200403123301 Verification check failed to run correctly
20200407063302 Verification check failed to run correctly
20200407083302 Verification check failed to run correctly
20200407103301 Verification check failed to run correctly
20200411221213 Verification tool is reporting incorrect results
20200411231212 Verification tool is reporting incorrect results
20200413043301 Verification check failed to run correctly
20200413063301 Verification check failed to run correctly
20200419113301 Verification check failed to run correctly
20200420103301 Verification check failed to run correctly
20200420153302 Verification check failed to run correctly

Times are UTC naturally.

The check runs once every hour, and uses Puppeteer to automate a headless browser connection to my personal member verification URL and then reports the results back.

Where the check failed to run that indicates the verification site was either down, or was providing very slow responses such as it was on Monday (20th) - potentially these could also be caused by my own Internet connection too.

Where it wasn't reporting correct results, those entries were triggered due to the order my certs were shown in changing on the tool, and my script not accounting for that. I've noticed this does still happen randomly, but my script no longer alerts on that since I adjusted it - it will only alert if not all of my certs are shown in the response.

EDIT: it's interesting to note how our logs don't seem to tally up. I didn't detect any errors on any of the days you did and vice versa!

Also, when I look at my logs, my check has been running for around 522 hours and only truly reported errors for 11 of those which is a 2.1% error rate or inversely results to 97.9% up time - that's not that bad really, all things considered!

Re: Verification fails - well, sometimes it does..

fortean — Thu, 23 Apr 2020 14:17:38 GMT

Good to see that you have been starting your own monitoring system, Alec. Kudos to you!

One of the possible explanations for the differences between your logs and mine is the method we use. I use the JSON API, you seem to use the standard HTTP(S) connection. I'd figure that they would end on the same system, but perhaps there is a difference.

Another explanation you already gave is that availability is hampered by network congestion or outage. That can be a local phenomenon or it can be an international issue - e.g. when the American site is not reachable from here, in Holland. In this case, perhaps (ISC)2 could consider employing something like Akamai.

I'll keep monitoring the connection, perhaps we can compare notes from time to time.

Re: Verification fails - well, sometimes it does..

AndreaMoore — Thu, 23 Apr 2020 19:46:19 GMT

@fortean has this been reported to membersupport@isc2.org? It looks like @AlecTrevelyan said he sent an email in a previous post. This could be a singular issue with your account.

Thanks,

Andrea