MountainScenarios

Tag: delphi method

These are the prompts used to gather information about homelessness and to develop a Delphi Method study to research causes and possible interventions (locally).  This is part of our Regenerative AI project; recreate as needed, when needed, with the GenAI engines available to you at that time. (Reprinted with permission from authors at NonprofitPlan.org and some adaptation here.)

Select GenAI results as of April 2024 of Delphi Research on Homelessness (from ChatGPT 4.0, Gemini, Claude, and Copilot) are available in this pdf document.

H+AI: The Genius of Crowds and AI

This site has now taken a dive into AI, Generative Artificial Intelligence (GAI) in this case. AI Assistants are everywhere now, and proliferating. In search(ish) there is Google’s Gemini (formerly Bard); in Microsoft there’s Copilot which utilizes OpenAI’s technology. And, of course there OpenAI’s ChatGPT itself, with the open source version available to anyone who has the time and money to obtain data, pre-train, and implement their own GAI system — typically for a more specific application like internal customer service.

There is a new heading on DelphiPlan.com (ScenarioPlan.com) that addresses Delphi + AI, or Scenario + AI… https://delphiplan.com/delphi-ai-primer/

Consensus: Let’s agree to look for agreement, not consensus

Most of the hunters (academic researchers) searching for consensus in their Delphi research, are new to the sport. They believe that they must bag really big game or come home empty handed. But we don’t agree. In fact, once you have had a chance to experience Delphi hunting once or twice, your perception of the game changes.

Consensus is a BIG dilemma within Delphi research. However, it is generally an unnecessary consumer of time and energy. The original Delphi Technique used by the RAND Corporation wanted to aim for consensus in many cases. That is, the U.S. government could either enter an nuclear arms race or not; there really was no middle ground.  Consequently, it was counterproductive to build a technique that could not reach consensus.  It became binary: reach consensus and a plan could be recommended to the president; no consensus, and this too was useful, but less helpful, to inform the president. (The knowledge that the experts could not come up with a clear path forward, when exerting a structured assessment process, is also very good to know.)

Consensus. The consensus process – getting teams of experts to think through complex problems and come up with the best solutions – is critical to effective teamwork and to the Delphi process. In most cases, however, it is not necessary – or even desirable – to come up with the one and only best solution. So long as there is no confusion about the facts and the issues, forcing a consensus when there is none is counter-productive (Fink, Kosecoff, Chassin & Brook, 1984; Hall, 2009, pp. 20-21).

Table 1 shows the general characteristics of various types of nominal group study techniques (Hall & Jordan, 2013, p. 106). Note that the so called traditional Delphi Technique and the UCLA-RAND appropriateness approaches aim for consensus. The so call Modified Delphi might not search for consensus and might not utilize experts. Researchers use the UCLA-RAND approach extensively to look for the best medial treatment protocol when only limited data is available, relying heavily on the expertise of the doctors involved to suggest – sometimes based on their best and informed guess – what protocol might work best. The doctors can only recommend one protocol. Consensus is needed here. 

(Table reprinted with permission Hall and Jordan (2013), p. 106).

But consensus is rarely needed, although it is usually found, to some degree, in business research, and even in most academic research. For example, the most important factors may be best business practices. Of the total list of 10 to 30 factors, few are MOST important. Often, the second round of Delphi aims to prioritize those qualitative factors identified in round 1. There factors are usually natural separation points between the most important (e.g. 4.5 out of 5), those that are medium important (3 out of 5), and the low importance factors.

Those researchers who are fixated on consensus might spend time, maybe a lot of time, trying to find that often elusive component called consensus. There are usually varying levels of agreement. Five doctors might agree on one single best protocol, but 10 probably won’t, unanimously. Interestingly, as the number of participants increase, the ability to talk statistically significantly about the results increases; however, the likelihood of pure, 100% consensus, diminishes. For example, a very small study of five doctors reaches unanimous consensus; but when it is repeated with 30 doctors, there is only 87% agreement. Obviously, one would prefer the quantitative and statistically significant results from the second study. (Usually you are forecasting with Delphi; 100% agreement implies a degree of certainty in an uncertain future, essentially this can easily result in a misapplication of a very useful planning/research tool.)

This brings us to qualitative Delphi vs. a more quantitative, mixed-method, Delphi. Usually Delphi is considered QUAL for several reasons. It works with a small number of informed, or expert, panelists. It usually gathers qualitative information in round 1. However, the qualitative responses are prioritized and/or ranked and/or correlated in round 2, round 3, etc. If a larger sample of participants results in 30 or more respondents in round 2, then the study probably should be upgraded from a purely qualitative study to mixed-method. That is, if the level of quantitative information gathered in round 2 is sufficient, statistical analysis can be meaningfully applied. Then you would look for statistical results (central tendency, dispersion, and maybe even correlation). You will find a confidence interval for all of your factors, those that are very important (say 8 or higher out of 10, +/- 1.5) and those that aren’t important. In this way, you could find those factors that are both important and statistically more important than other factors: a great time to declare a “consensus” victory.

TIP: Consider using more detailed scales. As 5-point Likert-type scale will not provide the same statistical detail as a 7-point, 10-point or maybe even a ratio 100% scale if it makes sense.

Subsequently, in the big game hunt for consensus, most hunters continue to look for the long-extinct woolly mammoth. Maybe they should “modify” their Delphi game for an easier search for success instead . . .

What do you think?

References

Hall, E. (2009). The Delphi primer: Doing real-world or academic research using a mixed-method approach. In C. A. Lentz (Ed.), The refractive thinker: Vol. 2: Research Methodology, (pp. 3-27). Las Vegas, NV: The Refractive Thinker® Press. Retrieved from: http://www.RefractiveThinker.com/

Hall, E. B., & Jordan, E. A. (2013). Strategic and scenario planning using Delphi: Long-term and rapid planning utilizing the genius of crowds. In C. A. Lentz (Ed.), The refractive thinker: Vol. II. Research methodology (3rd ed.). (pp. 103-123) Las Vegas, NV: The Refractive Thinker® Press.

Making sure the IRS Preparers are Prepared… Backcasting & Learning Theory

Dr Dave Schrader recently (December 2016) completed a very cool dissertation pertaining to the IRS and their (in)ability to assess tax preparers’ competency, and their (in)ability to test the preparers’ preparedness. {Sure, that’s easy for you to say!}

Over that last few years, the IRS has been charged with determining Tax Preparers’ competency. (Not the CPAs, mind you, but the millions of — shall we say — undocumented tax preparers.) The problem was that the IRS had not really determined what the preparers should know, before trying to test that they knew it.

Just as the IRS was starting to launch a “testing” of competencies, the civil courts forced Congress to pull the rug out. Another year or so has passed since a volunteer compliance program is in place…  Still no uniform requirements as to what those preparers should know in order to be prepared for the tests. But most importantly, now it’s not just tests, even if they start up again. With the change in Federal Law governing competency, tax preparers must be competent every single time they sign their name to a tax return. No matter how complicated!

What could go wrong with this? ! 🙂

So, Dave’s challenge was to do a dissertation into this murky quagmire. He found out the requirements, what they should know (generally), how they should learn it, and how competency should be assessed. As an afterthought, he tied this all into learning theory. To frame the skill identification, development, and assessment model, he tied the process into a construct for an effective total learning system.

If the dissertation sounds busy, that’s why. Lots of tables and charts to guide the reader through the mundane and the details.

Anyone teaching accounting should be interested in this dissertation. The management within the IRS should be calling Dr Dave in to assist with their Preparer Preparedness Program!.

From an Human Resources (HR) or management perspective, this is a very cool study. First is the skills needed. It works backwards from the skills needed to how and where to develop those skills: education, on-the-job training, or job experience. This is most of the way to “HR backcasting” for developing the skills needed for future jobs. Although backcasting is often used pertaining to economic development, the method, by necessity, must consider skills of the workers for those future jobs.

Can’t wait for the articles that will come out of this dissertation by this accountant (Accredited Accountant, Tax Preparer, and Advisor), teacher and newly minted Doctor.

References

Schrader, David M. (2016). Modified Delphi investigation of core competencies for tax preparers. D.B.A. dissertation, University of Phoenix, Arizona. Dissertations & Theses @ University of Phoenix database.

Delphi in the DoD Procurement Team (IPT) process

Jordan & Hall published an article in the Defense Sector edition (2016, Edition X) of the Refractive Thinker related to augmenting the DoD procurement process with Delphi team planning (Jordan & Hall, 2016). Here is the summary.

Delphi Method, or Delphi Technique, is an established method for bringing teams of informed panelists, or experts, together to analyze complex and interrelated problems. Organizations use group decision-making techniques to make sound plans, plans that gain support for the decisions made and build consensus. The U.S. Department of Defense (DoD) requires the use of Integrated Product Teams (IPTs) to ensure all disciplines are well represented in acquisition decisions. IPT planning process has several limitations, including the biases and inefficiencies associated with face-to-face meetings. The IPT process could be augmented to include Delphi analysis in order to develop more robust and more flexible procurement plans. Using the Delphi Method to augment IPTs could minimizing the costs and limitations of more traditional group planning while also significantly improve the quality of the procurement decisions. Delphi teams could be used with experts (or even with crowds) to provide sound analysis in many situations where the IPT process is ill equipped to produce unbiased and long-term results. Delphi teams would have the ability, as well, to look at bigger picture issues, and thereby avoid the narrow-scope, tunnel-vision analysis where most of the IPTs operate.

Reference

Jordan, E. A., & Hall, E. B. (2016). Group decision making and Integrated Product Teams: An alternative approach using Delphi.  In C. A. Lentz (Ed.), The refractive thinker: Vol. 10. Effective business strategies for the defense sector. (pp. 1-20) Las Vegas, NV: The Refractive Thinker® Press. ISBN #: 978-0-9840054-5-1. Retrieved from: http://refractivethinker.com/chapters/rt-vol-x-ch-1-defense-sector-procurement-planning-a-delphi-augmented-approach-to-group-decision-making/

The Volume 10 book: http://refractivethinker.com/books/the-refractive-thinker-vol-x-effective-business-strategies-for-the-defense-industry-sector/

Using Delphi Method for planning… including Scenario Planning and Horizon Plans.

This site discusses and tracks the use of Delphi-type methods in doing all kinds of research: academic, theoretical and real-world. Businesses can use the Delphi method to identify key issues, develop scenario plans and/or do horizon planning.

Strategic Business Planning that use similar methods as those used by Delphi Method. A strategic planning workshop for strat plan development uses a modified SWOT planning situational analysis method, for example. But the Delphi Method works best for horizon planning, future new product planning and scenario planning. We like to integrate disaster recovery planning (business continuity planning) into the scenario planning process.

Strategic Planning company (Hall, Hinkelman and associates) have done research and publishing on scenario planning and Delphi Method research.

Find these articles/books at:

  • SBPs Storefront at LuLu Press: LuLu.com/spotlight/SBPlan (Chapter 8 of the Guide 2.0 as well as the Economic Development Plan.)
  • Refractive Thinker(r) RefractiveThinker.com (Look for articles/chapters on Delphi research including the Delphi Primer.)

ScenarioPlanningTimeline

We like to look for that future deflection point were it would be clear to everyone, including the dog, “Toto, I’ve a feeling we’re not in Kansas anymore.”

Powered by WordPress & Theme by Anders Norén