The ABR Online Testing Experience

Radiology   //   1

When I finally got around to taking the Neuroradiology CAQ Exam this fall, I finally had the pleasure of partaking in the ABR’s remote testing platform. I put a lot of work into my posts about the Core and Certifying Exams back in the day, but they do hail from the pre-pandemic era.

Here are some thoughts on taking an ABR exam from the comfort of not Chicago or Tucson:

Big Brother

ABR relies on a third-party company called Proctorio for monitoring.

Everything–including the real-time video monitoring, capturing a photo of your picture ID during the start-up process, and the “room scan”–rely on the webcam. This feature was clearly intended to be used on a laptop, which you could easily pick up and spin around the room. The ABR now requires a separate dedicated side-view webcam, which makes the acrobatics a bit more complicated.

Instead of buying a separate webcam, I used a super old DSLR with a $15 HDMI to USB video capture card (the knockoff of the better Cam Link). This works fine (and it’s what I use for Zoom calls to have that beautiful bokeh so that people like me more), but it’s worth noting that if you don’t have good autofocus (and mine is so old that it does not), that it may take a few tries to get the screengrab of your photo ID good enough to satisfy Proctorio’s artificially intelligent needs. Depending on your desk set up, a relatively short cable connecting the camera to the computer or a tripod to have the camera high enough to actually see anything if you don’t have a dresser or something suitable nearby can make the room scan awkward and a bit tenuous. A longer USB cord helps. After around 5 minutes of finagling, eventually, I was allowed to take the test.

You could also use your phone as a webcam by using one of many inexpensive apps like Camo or EpocCam, which is probably the best option, especially if you already have some sort of suitable stand or one of the infinite variety of cheapo phone tripods.

Anyway, what I’m trying to say is I don’t think most people need to buy a dedicated external webcam just for the purpose of taking one of these ABR tests. Especially since most webcams are terrible.

I will say that Proctorio didn’t make me install any apps on my phone, and the Chrome Extension is easy to uninstall. I don’t think anyone is going to have their identity stolen or be harassed by call center employees. While I am admittedly not very experienced with the myriad options for education spyware, it seems like the ABR picked one of the better players.

Computer Set-up

I really wanted to use my home workstation. It has direct fiber internet and a system I use every day for radiology-type stuff. However, Proctorio and the ABR have a 1 monitor rule, and I–in my radiologist glory–have 4 monitors. To satisfy Proctorio, any extra monitors in the space have to be missing from your system entirely, so simply turning them off isn’t good enough. They need to be totally unplugged from your computer so that they don’t show up. I use a dual PC system set-up to read simultaneously for the hospital and our outpatient PACS by swapping monitors and peripherals between two different PCs using a KVM switch. It’s awesome. But the nest of monitor cords is no joke, and the settings are just how I like them, and there was no way I was going spend the time unplugging stuff and jeopardize any of that in order to take a 3-hour exam with a bunch of low-quality JPGs.

Instead, I took it on my 2017 iMac over WiFi, which is across the house from the router, and it was fine. The biggest problem was getting Proctorio happy with the backlighting from the adjacent window overexposing the camera and making me look too dim.

Yes, I did roll over my attending chair, the Herman Miller Embody, from across the house for the experience, and it was worth it.

The Practice Test

The practice test is a 98 question process, but really it’s just designed to get you to make sure you can pass Proctorio’s welcome barriers and familiarize yourself with the software. The content is a random hodgepodge of questions, but there is no performance feedback or answer key, limiting usefulness as an actual practice exam experience.

The content parallels the Core Exam, which is to say I found it more difficult now as a super high-performing experienced attending radiologist than I would have as a resident. Go figure.

The Online Platform

I would show you some screenshots, but, you know, I can’t.

The software is totally fine. It works. You have a bunch of tools to manipulate the images, but since most images are just static screen grabs of middling quality, you usually don’t need to do anything. In fact, the image sizes are small enough and the questions themselves short enough that you really don’t need or even want a large monitor or anything fancy to take the exam. Your average laptop screen has plenty of real estate.

The ABR locks your past performance every 30 questions to prevent you from altering your responses to prior questions after you’ve left the surveillance zone for breaks (aka cheating). Locking so frequently is a touch annoying, and I’m sure the policy frustrates the people who want to blaze through the whole test and then agonize for hours on all the questions they flagged at the end. Psychologically, this is probably healthier, though it does make flagging a question basically useless.

I wish they just locked questions at breaks and not after so short a period, but given intermittent connectivity issues, the shorter batch size probably helps substantially reduce their tech support demands. It’s a worthwhile price to pay for the convenience.

Another irritating feature is that the ABR really wants you to move forward through the exam and not backward to revisit old questions, so much so that if you try to visit the previous question, the software prompts you with a pop-up saying (paraphrasing) “forward is so much better, are you sure you want to go back?”…which is annoying. I think the idea here is that they especially don’t want you to see a question, not answer, go backward, lose access in some way, and then have to unlock the question for you and/or question your integrity. 

On exam day, I got a brief “connection lost” warning twice, lasting for a few seconds before being reestablished automatically, otherwise no interruptions. I know people personally and online who had less good luck, sometimes being completely kicked out and needing to call to have questions unlocked. To their credit, I’ve only heard good things about ABR customer service when it comes to helping with exam connectivity issues.

Overall

Overall these inconveniences are a small price for being able to wear pajamas and take a break literally whenever you want, which is wonderful. The ability to use a bathroom mere feet away or get drinks and snacks on a whim creates a completely different ambiance than a typical testing experience.

Not having to pay for flights and hotels, waste days on both ends for travel, and leave your family is also almost priceless.

In some ways, it’s a little harder to get in the normal high-stakes exam headspace without the suffering, but it’s a breath of fresh air that I enjoyed nonetheless. I bought a twelve-pack of Diet Mountain Dew just for the occasion, and I made myself a nice fresh ice coffee as well (see above, hat tip double-walled insulated glasses) to sip while answering ludicrous questions about differentiating spinal cord tumors. 

If that’s not the dream, I don’t know what is.

 

 

Review: Sulcus Neuroradiology CAQ Prep

Radiology, Reviews   //   1

The neuroradiology subspecialty exam (aka the CAQ or certificate of added qualification) is a tedious, basically redundant, and expensive waste of time taken by a relatively small number of people every year.

As a result, there are very few dedicated recourses.

In fact, there is only one, and it’s Sulcus.

Based on an informal survey I did of CAQ takers, the Sulcus bank is relatively popular despite what are generally poor reviews online. In my informal poll, almost half of respondents admitted to using it. I chalk this up to the CAQ exam being exceedingly expensive, people not wanting to fail it, and this being the only dedicated product.

Related note: Sulcus also now offers MSK and Mammography programs, neither of which I looked at.

Reviewer Disclosure

Sulcus did not pay for this review nor have they seen it prior to publication.

They did provide me with 7 days of free access. I have a life, so that wasn’t sufficient for me to view the entire product, but it was enough for me to form an opinion. I’m not sure I would have finished it regardless, and I did not purchase the product afterward for further studying.

I also have no discount code for you, dear reader.

Platform

It’s a little clunky, but it works.  

You must “create a quiz” as a separate step before you can actually take said quiz.

There is no tutor mode. 

The site is technically mobile-friendly but the left sidebar (which is just the question numbers) takes priority, meaning that you need to scroll down past the question navigation to do every single question and read every explanation. Very tedious.

Content

There are a total of 562 questions grouped into 175 cases.

As in, there are several questions per case/diagnosis/image(s). For example, doing my first 5 cases used up 16 questions.

The cases themselves are solid. But when each case has three or sometimes four questions, the first is usually high yield, the second is borderline, and the third is basically just for giggles.

The question format is nothing like CAQ. Most CAQ questions are diagnosis-based.

Many Sulcus questions are multiple-choice “pick the correct statement” nonsense, where you evaluate a bunch of statements and then pick the one that is true. It also contains questions that are negatively framed (“which of these are NOT…”) questions that are garbage psychometrically (in their defense, these still crop on ABR’s exams rarely as well).

You’ll also get a lot of extended multiple T/F questions, where you need to evaluate a whole bunch of statements about the diagnosis and get each of them right in order to be “correct” overall. So, the end result–as other reviewers have alluded to–is mostly irritating/demoralizing.

Extended matching questions are pretty rare but fine.

Overall, the bank trends too much toward hyper-detailed in-the-weeds stuff, making it very inefficient to review in quantity for a busy practicing radiologist. The CAQ is an image-based test mostly of diagnosis, so testing a whole bunch of second-order factoids is not worth your time.

In Sulcus’ defense, I have a suspicion that the CAQ exam has changed significantly over the years. It very well may be that its current format more closely resembled the CAQ of old before changes brought the question style more in line with the modern multiple choice offerings of the Core and Certifying Exams. I support this theory with two points.

  1. No one would choose to write questions like this for no reason.
  2. The current (but clearly out of date) CAQ exam study guide from 2017 suggests that most questions would be part of linked question sets: “After the candidate makes her/his choice, one to three follow-up questions will typically be asked.” This is simply not the case. Most questions are solitary single best answer and do not go on to hammer you on miscellaneous second-order details.

Survey Results

I did a brief survey online and got 38 responses. Of the 18 people who said they used Sulcus, there were only two written comments: “didn’t think was relevant” and “was the highest yield by far.”

I think the reality is somewhere in between.

Pricing

$399.

Free Sample

There is one sample case in a slideshow at the bottom of the main page.

Verdict

Eh, pass.

A decent collection of wide-ranging cases; not a bad learning tool but extremely expensive related to other options and not an efficient CAQ review for the practicing neuroradiologist.

Unless you really have a CME fund to burn and nothing else to spend it on, you don’t need something this tedious and demoralizing to get through for the purposes of passing the CAQ exam. Some regular cases books, like Neuroradiology: A Core Review, will get the job done.

If you really just want a chance to see some pretty solid cases, take some great and some excruciating multiple-choice questions, and really hammer some low-yield esoterica to impress trainees and specialists, then this is an excellent choice.

 

The ABR is Sorta Changing Its Fees

Radiology   //   1

In recent years, the American Board of Radiology (ABR) has utilized a membership fee model, where–for example–those working towards an initial certification in diagnostic radiology would pay a $640 annual fee until passing the Certifying Exam. Since one takes the Certifying Exam 15 months after finishing residency, that has meant recent diplomates have paid a specialty tax of around 1% of their gross income for a total of five years before enjoying the privilege of paying a mere $340 per year for MOC forever.

The fee schedule looked like this:

To illustrate, here’s my payment history (the annual fee actually increased a bit during my training because money).

As of September 2021, ABR has moved to “an exam fee model.” How does that look? Well, a one-time $640 application fee followed by a $1280 Core Exam fee and a $1280 Certifying Exam fee.

It doesn’t require a doctorate to note that the total cost for initial certification is the same: $3200.

That fee continues to put radiology in the highest echelon of medical specialties in terms of board costs, as enumerated in this 2017 paper (which incidentally undercounted the radiology costs).

What has changed is that this fee structure is now standard across other exams and is resulting in a decrease in the (otherwise ludicrous) subspecialty exam fees.

You see, until now, the much shorter half-day CAQ exams actually cost the most! As above, you can see I paid $3,280 this spring for the privilege of spending a morning taking a poorly formulated exam to pseudo-prove that I can totally do the thing I already do every day. That’s more than the cost of the combined total of the much, much bigger Core and Certifying Exams.

But, as of this September 17, 2021 update, it’s merely the same $640 application fee + $1280 exam fee for a total of $1,920 (a savings of $1,360!).

Of course, before you get any warm fuzzies about their generosity, keep in mind that the CAQ exams comprise a relatively small proportion of ABR revenues since only ~200 people take them every year, and, meanwhile, MOC revenues continue to grow year after year. The ABR, per its internal narrative and official documents, has recently been operating at a loss.

Thankfully, they have some retained earnings on hand to mitigate the red.

Equity, Organized Medicine, and the Radiology Value Chain

Medicine, Radiology   //   0

It’s often said that large organizations are difficult to steer and slow to change course, but that’s only part of why they sometimes act in seemingly inexplicable ways. There’s another more insidious reason, and that is conflicts of interest, not just within leadership but also in the changing demographics of the membership.

A passage from “Value Chain: Where Radiologists Should Put Their Focus in Threats Against Income” by Seth Hardy MD MBA in Applied Radiology:

So, while private/public equity firms can use leverage to amplify profits to the upside, leverage has an opposite effect when gross income is in decline. Any cuts to reimbursement would be truly devastating to these firms’ employees; since the debt holders get paid before the radiologists, the impact on employed radiologists’ salaries may be significant. As equity-employed radiologists make up a greater share of dues-paying members within organized medical societies, it is easy to understand why the proposed CMS cuts were characterized as draconian by those societies. But a clear understanding of value chain by physicians is increasingly critical to evaluate the rhetoric of our medical society leadership.

I am now a partner in a physician-owned independent radiology practice. A CMS paycut would mean that we earn commensurately less money–not that we will become insolvent.

That should count for something when choosing where to work.

To STAT or not to STAT

Medicine, Radiology   //   0

A passage about limited resources and optimizing imaging from The Emergency Mind: Wiring Your Brain for Performance Under Pressure by Dan Dworkis MD PhD:

Within the broader context of your responsibility however, there frequently will be significant variability in the relative urgencies of individuals being imaged. Some patients—like a person seemingly experiencing an acute stroke—do need to be scanned immediately. Others—such as a patient with abdominal pain, stable vitals, and a reassuring physical exam—while no less “deserving” of those resources, would receive nearly equal benefit from being scanned now as in an hour from now. Optimizing care across the field in this context would involve prioritizing CT scans for those patients who would receive outsized benefits from immediate imaging, even if this makes some other patients wait longer.

Put a different way: If everything is stat, nothing is stat.

Stat abuse is one of those crimes especially tempting to inpatient teams in busy hospitals. It’s natural to want answers (and dispo) as soon as possible, and we assume that we will get them faster if we increase the priority of the exam.

All a clinician knows is that sometimes something ordered routine takes forever and that ordering stat should generally result in it being performed faster. They may not even care if the read is prioritized in all cases so long as the patient is freed from the waiting and future transport.

It’s also human nature for there to be a distribution with certain individuals ordering an outsize proportion of “stat” exams. The negatives of over-ordering or inappropriate priority are almost always placed on other staff. In a zero-sum game, selfish behavior may be an optimal choice for individual success even if it makes the system less efficient overall. Hospitals very rarely scold their staff for such abuses.

I don’t think most clinicians even have any idea where along the spectrum their behavior falls. Knowledge of outlier performance one hopes might curb excesses, and that data would certainly be helpful for individuals to know (presuming those individuals are capable of feeling shame and said shame functions as a deterrent). Such information would have to be long-term and stratified well to be meaningful (we should expect different levels of stat exams as a fraction of orders from different hospital units, for example). Otherwise, data are dismissible.

Ultimately, pleading and punishment are often ineffective and/or undesirable.

A more helpful approach would include data to guide decision-making on a case by case basis:

The EMR should show in real-time the expected wait for different study types based on the current queue and exam types pending, both inpatient and outpatient (i.e. how many unnecessary exams are obtained during an inpatient stay due to fears of long delays for outpatient follow-up?). Yes, a routine study may unexpectedly get bumped further down the line, but a smart system would incorporate predictions based on the current patient census, admission diagnoses, time of year, and whatever else some machine learning algorithm would include its impenetrable black box of Skynet code.

It would be extremely helpful for all parties to know if an MRI should be expected today or tomorrow, sometime this afternoon or more likely at 3 am.

And so, yes, of course, people are working on this in the machine learning world. But hurry up. I for one will continue to welcome our AI overlords and their promised efficiency gains, but I’m still waiting.