Nuclearn FAQ

General

Do you deploy on-premise or on cloud?

Nuclearn can be deployed on-premise, on private cloud, or as hosted SaaS on GovCloud.  For customers that want us to do hosted SaaS, we do include a small price increase to support cloud hosting costs.

Are there any hidden costs with using Nuclearn?

The subscription license fee includes the software license, model fine-tuning, installation, support, training, and software upgrades.  We don’t charge additional professional service fees or change orders to get our software up, running and automating.  The only professional services we charge are if Nuclearn is contracted to develop and maintain highly customized integrations.

AutoCAP Screener – Automated Corrective Action Program Screening

What machine learning technologies are used to automate screening?

As of 2023, Nuclearn uses a type of deep neural network commonly known as a “Large Language Model”.  These models deliver state of the art performance on a wide range of machine learning tasks, especially those with unstructured input (e.g. condition report descriptions).  AI and machine learning are quickly evolving, and Nuclearn commonly releases updates to out of the box and customer specific models to improve performance.

How do you train the models?

Training these models is very resource intensive, and requires specialized hardware.  For model fine-tuning, we ask our customers to share with us a representative sample of condition report data.  We then fine-tune our Nuclear-specific models on that data to learn site-specific terminology and patterns in order to deliver the best performance.  The fine-tuned models are then deployed on-premise or on cloud as needed.

How much data do the models need to train?

As much as you can provide!  Model performance improves with additional data, even if older data is of a poor quality or differs from current screening processes.  >10,000 condition reports spanning at least one refueling cycle is typically sufficient.

How does the system choose what to automate?

When Nuclearn’s models process a new condition report, they make predictions for all screening decisions (e.g. Condition Adverse to Quality, responsible group, etc.), and provide a confidence value for each decision.  These predictions and confidence values are compared to the current automation configuration, and if any of the decisions do not meet the confidence threshold, the condition report will not be automated.

How does the system track accuracy?

Every condition report processed by Nuclearn is added to an audit trail, including all the information provided to Nuclearn, the model that was used, the automation configuration, and the model output.  After the condition report has been fully processed, the “ground truth” data is sent back to Nuclearn to calculate accuracy and system performance.

To track accuracy on automated records, a configurable randomly selected proportion of condition reports that would’ve been automated are instead sent for manual review.  This single blind sample-based approach allows Nuclearn to continuously estimate performance of automated records without requiring manual review of all records.

Can we control the automation configuration?

You have full control over the automation configuration, including what decisions to automate, desired levels of accuracy, and manual control sample rates.  You don’t need to contact Nuclearn to have these changed, although we are happy to assist.

Some of our condition reports are cryptic or poorly worded.  How are these handled?

This is one of the important reasons that we fine tune our models on your site’s data.  Different utilities have different acronyms, terminology, “style”, and variability in their condition reports.  The fine-tuning process allows the models to learn these patterns.  If a particular condition report is very vague or poorly worded, we typically see the models produce a low-confidence prediction that doesn’t get automated.

Do you have to start with high levels of automation on day 1?

We recommend that customers do a period of time in “full recommendation mode”.  In this mode, condition reports are fed through the automation pipeline and tracked, but no records are “automated”.  This allows the site to gather performance data and gain comfort with the process before automating.  After deciding to automate, we recommend customers ramp up automation by starting with conservative configurations with high manual sample rates, and incrementally increasing automation rates.

Evaluator – Automated Trend Coding & Analysis

How does automated trend coding work?

Nuclearn provides a model that can take any provided text and predict that certain trendcodes are applicable.  This model can be applied to any text, but typically works best with condition reports.  This model can be used to predict these codes for a single example, or to score many records at once and save the predictions for later analysis.

What trendcodes are available?

The current model provides predictions for the industry standard WANO Performance Objectives and Criteria (PO&C).  The next iteration of the model (2023 Q2 or Q3) will include codes from INPO 19-003 Staying on Top and INPO 15-005 Leadership and Team Effectiveness Attributes.

How long does it take to score a year’s worth of data?

Times will vary depending on condition report volume, length, and computing hardware.  However, a year’s worth of data can typically be scored in less than 8 hours for most sites.

What analytics do you provide out of the box?

Nuclearn includes an analytic engine specially designed for analyzing condition report data after it has been trendcoded.  The built in analytics include the ability to calculate time trends, intra-company comparisons, and industry benchmark comparisons, and display these visually in a cluster visualization to quickly identify trends.  Identified trends can quickly be saved as a “bucket”, and included in various reports.

I don’t agree with the predicted trendcodes sometimes.  How can I trust the trends?

Applying codes is often very subjective, and even people usually disagree amongst themselves on what codes should be applied.  Using Nuclearn’s model has a key advantage that people don’t have however: it is 100% consistent.  This is absolutely critical for trending across many years of data, as inconsistent coding (especially over time) often results in spurious trends unrelated to underlying performance issues.  While the model may be wrong on specific condition reports, applied over years of data these occasional inaccuracies are averaged out by the law of large numbers, allowing for effective trend analysis.

Can I connect PowerBI to Nuclearn?

While Nuclearn includes many built in analytics, you don’t have to solely rely upon them.  Nuclearn supports PowerBI integrations at all points in the analytics pipeline – from initial datasets, to raw trendcode predictions and analytic results.

How are the predictions validated? Both internally and across industry?

We test our models internally using a variety of traditional machine learning
methods. These include calculating accuracy, F1 score, log-loss, and other
common metrics on test/validation data. Additionally, we look at other more
qualitative metrics such as “coverage” across rare codes. We also
subjectively evaluate outputs on a variety of CAP and non-CAP inputs to ensure
results make sense.

When we work with new customers we typically provide a set of predictions for a provided sample of their data, and they evaluate against that. We don’t always have direct insights into their audit/review processes, but they will usually perform tasks like:

  • Evaluate predictions on a random sample of trend codes
  • Perform a trend analysis (either using Nuclearn Cluster Visualization or their own tools), and validate that either a) the trends line up with trends they have identified using other techniques or b) upon further inspection the trends appear to be valid or explainable
  • Compare predicted PO&Cs to previously applied internal trend codes (when available) to check if there are any major areas being missed

Why are the children trends captured and not just parents (e.g. why use specific trend codes instead of grouping up to the Objective level?

Grouping issues up at a very high level (e.g. at the objective level) is far too
broad and unlikely to identify important trends in our experience. For example, evaluation Areas for Improvement (AFIs) are usually created at an Objective level but they are almost always
representative of deficiencies in a sub-set of very specific things under that Objective. It’s very rare for a giant trend across most of the trend codes in a particular are.

It’s still important to group multiple low-level codes together, but usually in groups of 2-8. The “Cluster Visualizations” tools makes it easy to make “buckets” of codes easily.

5 Reasons AI is the Promising Future of Nuclear CAP

In the near future, the Nuclear Corrective Action Program (CAP) will be sleek, streamlined, and highly efficient; where occasionally humans participants are required to review and deliberate over only the most complicated issues requiring their vast experience and wisdom. For everything else, a trained army of CAP AI agents invisibly process issues, review and alert on trends, assign corrective actions, and take feedback from human coaches via purpose-designed human/AI interfaces.

No longer will a team of humans be subject to hours upon days of analysis for trend detection, a Senior Reactor Operator forced to process another condition report about a cracked sidewalk, or an Engineer left waiting for a corrective action item to be issued to her inbox. These functions will have been largely automated with the focused application of AI-based technology. Here are the five reasons this future is highly probable, based on both the current state of the Nuclear Industry and leading-edge AI technology.

Cost Savings and Improved Quality

It comes as no surprise to anyone that has worked in the Nuclear Industry that running an effective CAP program is expensive. CAP demands a significant investment into human resources that have adequate experience to effectively diagnose and resolve the problems experienced in an operating power plant. In practice, this requires either dedicated staffing or rotating employees out of primary roles to fulfill a CAP function.

By applying intelligent automation to the Screening, Work Generation, and Issue Trending processes, a resource reduction of approximately 45% is expected.

Beyond reducing the number of resources required, AI reduces the total amount of time required to execute portions of the CAP process. While a human screening team may only be able to process conditions on a daily basis, an AI system can review and screen conditions and issue work items immediately. More quickly getting workable tasks into the hands of employees saves money and improves CAP quality.

For those issues that may be too complex for AI to effectively handle, a human-in-the-loop system can be employed, where AI knows when it is unsure and can reach out for human assistance. By using human-in-the-loop the cost of the CAP program is reduced while keeping quality the same or better.

Additionally, AI can lower the threshold for issue documentation. Deployment of an information extraction AI lets employees more naturally capture issues using natural language, without filling out specialized forms. When issues become easier to document, they are documented more often, the overall information input into the CAP program increases, and the chance an issue is corrected becomes greater. AI that immediately evaluates the quality and completeness of the submitted report enables automated dialogue with the submitter. This can encourage behaviors such as adding information, clarify issues, correcting spelling, or otherwise encourage behaviors that promote report quality, increasing the effectiveness of the overall CAP program.

Scale

The most valuable problems to solve are frequently the largest. CAP and associated activities are one of the largest opportunities in Nuclear. CAP lies at the heart of the Nuclear Industry, and requires participation from almost every trade and profession at each site. The ubiquity of CAP combined with the savings potential provides an immense incentive for plant operators, industry vendors, and industry research groups to discover and implement ways to make these programs run more sustainably and efficiently. Specialized AI that can automate various tasks are at the top of mind of industry groups such as the Electric Power Research Institute, the Idaho National Laboratories, and various utility in-house teams.

A fortunate side effect of the CAP program is the production of large quantities of high-quality data – data ideal for training the AI systems that will be used to automate the same functions. Most of this data is captured in free-form text as natural language. Language with a specific Nuclear vocabulary and dialect, but natural language nonetheless. The scale of this data puts it on par with the datasets utilized by the large technology vendors and academic institutions to develop and train the most effective AI systems. Thanks to the scale of Nuclear CAP data, these large AI systems can be specialized to operate in the Nuclear domain – increasing performance and effectiveness for the tasks at hand.

Transportability

The most notable advancements in AI technology of the late 2010s were around the development of advanced natural language-based AI. This AI has the ability to understand human language more naturally and effectively than previously thought possible. Breakthroughs in this area are characterized by the ability of AI to transfer learning from one problem to another. An AI good at classifying condition report quality will be better at identifying equipment tags vs one specifically trained just to identify equipment tags.

The benefit for the nuclear industry is that an AI system trained at Plant X will be able to transfer its learning to Plant Y and be more performant than one trained at just Plant Y. This is similar to how a working professional at Diablo Canyon would more easily adapt and apply their knowledge when transferring to Turkey Point than someone not having worked in the nuclear industry at all. Similar to a human employee, an AI system will benefit from the variety of different knowledge obtained from general industry data. Learning specifics for any one plant will be faster, cheaper, and easier for any plant wishing to specialize the AI system for use in automation once trained on general industry data.

As a result, solutions developed at one site will be able to be shared. With commonly applicable training and similar problems, the industry can work to solve the big problems once with ‘large’ or ‘hard’ AI, and transport the solution from plant to plant for the benefit of the entire industry.

Automated Screening

One of the more specific solutions apparent when applying AI to the CAP process is the automation of the condition screening process. Condition screening is the process of reviewing a received report of a non-standard condition in or around the plant, then applying certain tags, codes, or classifications, assigning an owner, and generating the appropriate work items that address the condition. For some plants, this process involves dedicated groups of senior employees that work daily to manually perform this process. For others, this involves dispersed resources periodically gathering together to complete screening. In either case, the resources are usually senior-level and experienced, and thus expensive. The following estimation of resources spent by the industry for this process each year illustrates just how large an opportunity there is:

The screening process has certain properties: repeatability and complexity of task, quality of data, scale, cost, etc. that make it extremely promising to apply AI-powered automation — discussion worthy of a separate blog post…coming soon!

Automated Trending

Automated trending is the sequel to Automated Screening – it’s what comes after the conditions have been identified and actions issued. Normally done ‘cognitively’ or via brute force search of the condition data, trending is resource-intensive and largely manual. Read Nuclearn’s Nuclear CAP Coding AI – Better Performance at a Lower Cost to find out more about how AI can help automate and simplify the trending task.

Bonus: The Rapid Progress of AI Technology

The five points above are only achievable due to the explosion in the progress of various technologies that underpin how AI learns and behaves. The speed in recent years with which new AI tools achieve human-level performance on various vision and language tasks is unprecedented. As seen in the chart below, developing AI that can recognize simple numerical digits at human-level performance took over 10 years; to recognize cats, dogs, cars and other everyday objects in images took about 5 years. More recently, developing AI that can recognize and manipulate human language took only about 2 years.

The accelerating pace of AI advancements shows no sign of stopping anytime soon. This type of rapid advancement, combined with the scale, transportability, and savings of CAP, allows Nuclearn to confidently say AI is the future of Nuclear CAP.

 

DARSA: The Guide to Full Process Automation Using AI

You don’t automate right away…

Process automation using Artificial Intelligence is a complex endeavor. To successfully automate a process, automation systems and their implementers need to effectively incorporate complex technologies, a deep understanding of the business processes, risk-based decision making, and organizational change management all at once! This challenge can feel insurmountable to many organizations looking to start adopting AI-driven process automation. And unfortunately for some, it has proven to be so.

Luckily, there are battle-tested methods for bringing an automation system to life and avoiding the potential pitfalls. Here at Nuclearn, we have developed a project implementation process we call DARSA that helps guide us through automation projects. DARSA helps us deliver maximum value with minimal risk by leveraging an iterative, agile approach to AI-driven automation.

So what is DARSA? DARSA is a five-step linear process that stands for Decisions-Data-Direction, Assess, Recommend, Semi-Automate, and Automate. Each step in DARSA is a distinct phase with distinct characteristics, and transitions between these phases are planned explicitly and usually require system changes. To learn more about DARSA, we must dive into the first phase: “Decisions, Data and Direction”.

1) Decisions – Data – Direction

Before starting AI-driven automation, it is important to specify several key factors that will guide the project. These items are Decisions, Data, and Direction.

Decisions are the first, and most critical item to define early in an AI-driven automation project. At the end of the day, if you are embarking on an AI-driven automation project, you are doing so because you need to automate challenging decisions that currently require a human. If there is no decision to be made, then AI will not help automation in any meaningful way. If the decisions are trivial or based on simple rules, there is no need for AI. These processes should be automated by traditional software. So the first, essential part of an AI-driven automation project is identifying and defining the decisions that are planned to be automated.

Data is the next item that must be specified. An AI-driven automation project needs to identify two key sets of data: the input data being used to influence decisions, and the data created as a result of those decisions. These are critical, as AI-driven automation relies heavily on machine learning. To learn how to make decisions automatically, machine learning requires historical data. For each decision, there should be a detailed log of all data used at decision time, and a log of all historical human decisions.

Direction. AI projects must begin with clear direction, but unlike large traditional software projects, AI-driven automation projects cannot begin with a detailed project plan laying out detailed requirements gathering and design. AI and machine learning are notoriously unpredictable – even the most experienced practitioners have challenges predicting how well models will perform on new datasets and new challenges. Automation systems often have to evolve around the unpredictable strengths and weaknesses of the AI. As a result, it is important to specify a clear direction for the project. All members of the project should be aligned with this direction, and use it to guide their iterations and decisions. For example, in an AI-driven automation system for helping generate Pre-Job Brief forms, the direction for the project might be “Reduce the amount of time required to assemble Pre-Job Briefs while maintaining or improving the quality of Pre-Job Briefs”. This simple statement of direction goes a long way towards bounding the project scope, ruling out system design decisions that are unacceptable, and fostering potential innovation.

2) Assess

Once the Decisions, Data and Direction are specified, the most important factors for success in an AI-Driven automation project are fast iterations and good feedback. That is why the second phase in DARSA is “Assess”. During this stage of the project, nothing is actually automated! The “automation” results are being shared with subject matter experts, so they can assess the results and provide feedback. Automation system designers and Data Scientists are already quite familiar with testing how well their system works via various traditional methods. While these can help with generating accuracy metrics (the model is 95% accurate!), these methods are quite poor at evaluating exactly where the automations will fail or be inaccurate, why they are that way, and what the impacts are. The Assess phase is often where important risks and caveats are identified, and where additional considerations for the project are discovered.

Let’s take for example my experience with a project attempting to automate the screening of Condition Reports (CR) at a Nuclear Power Plant. One of the key decisions in screening a CR is determining whether the documented issue has the potential to affect the safe operation of the plant, often referred to as a “Condition Adverse to Quality”. Before even showing our AI model to users, my team had produced some highly accurate models, north of 95% accurate! We knew at the time that the human benchmark for screening was 98%, and we figured we were very close to that number, surely close enough to have successful automation. It was only after going through the “Assess” phase that we learned from our subject matter experts that we had missed a key part of the automation.

We learned during the Assess phase that not all Condition Reports are the same. In fact, there was a drastically asynchronous cost associated with wrong predictions. Overall accuracy was important, but what would make or break the project was the percentage of “Conditions Adverse to Quality” that we incorrectly classified as Not Conditions Adverse to Quality. The reverse error (classifying Not Adverse to Quality as Adverse to Quality) had a cost associated with it – we might end up performing some unnecessary paperwork. But get a few high-profile errors the other way, and we would potentially miss safety-impacting conditions, undermining regulatory trust in the automation system and the CAP Program as a whole.

As a result, we made some fundamental changes to the AI models, as well as the automation system that would eventually be implemented. The AI models were trained to be more conscious of higher-profile errors, and the automation system would take into consideration “confidence” levels of predictions, with a more conservative bias. A thorough Assessment phase reduced the risk of adverse consequences and ensured any pitfalls were detected and mitigated prior to implementation.

3) Recommend

After the Assess phase, Recommendation begins. The Recommend phase typically involves providing the AI results to the manual task performers in real-time, but not automating any of their decisions. This stage is often very low risk – if the AI system is wrong or incorrect there is someone manually reviewing and correcting the errors, preventing any major inaccuracies. This is also the first stage that realizes delivered value of an AI-driven automation system.

Increased manual efficiency is often recognized as a benefit in the Recommend phase. In the majority of cases, it is physically faster for someone to perform a review of the AI’s output and make small corrections versus working the decision task from start to finish. Paired with a proper Human/AI interface, the cognitive load, manual data entry, and the number of keystrokes/mouse clicks are drastically reduced. This helps drive human efficiencies that translate to cost savings.

The Recommendation phase also permits the capture of metrics tracking how well the automation system is performing under real-world use. This is absolutely critical if partial or complete automation is desired. By running in a recommendation setup, exact data about performance can be gathered and analyzed to help improve system performance and gain a deeper objective understanding of your automation risk. This is important for deciding how to proceed with any partial automation while providing evidence to help convince those skeptical of automation.

This stage may last as short as a few weeks, or as long as several years. If the automation system needs additional training data and tweaking, the recommendation phase provides a long runway for doing so in a safe manner. Since the system is already delivering value, there is relieved pressure to reach additional levels of automation. On some projects, this phase may provide enough ROI on its own that stakeholders no longer feel the need to take on additional risk with partial or complete automation.

4) Semi-Automate

The next step in DARSA is “Semi-Automate”. This is the first stage to both fully realize the benefits and risks of automation. This phase is characterized by true automation of a task – but only for a subset of the total tasks performed.

The metrics gathered in the Recommend phase play a key role here, as they can inform which parts of the task are acceptable for automation. As the system encounters different inputs and situations, total system confidence will vary. Based on this confidence, among other metrics, automation can be implemented as a graded approach. Low-risk, high-confidence tasks are usually automated first, and after the system continues to learn, and stakeholder confidence is improved, higher-risk automations can be turned on.

For example, take a system intended to automate the planning and scheduling functions required for Nuclear Work Management. Such a system would begin to partially automate the scheduling of work activities that have low safety and operational impacts, and the planning of repetitive activities that have little historical deviation in the execution work steps. These activities are low-risk (if something goes wrong there are minimal consequences) and high-confidence (the AI has lots of previous examples with defined conditions).

During semi-automation, it is prudent to still have a manual review of a portion of automated tasks to monitor model performance, as well as provide additional training data. Without manual review, there is no longer a “ground truth” for items that have been automated. This makes it challenging to know whether the system is working well! Additionally, AI performance may begin to stagnate without the inclusion of new training examples, similar to how human performance may stagnate without new learnings and experiences.

5) Automate

The final phase that every automation system aims to achieve: complete automation. This phase is characterized by the complete automation of all tasks planned in the scope of the project. The system has been running for long enough and has gathered enough data to prove that there is no human involvement necessary. From this point forward, the only costs associated with the task are the costs associated with running and maintaining the system. Complete automation is more common with tasks that have a lower overall level of risk, yet require a lot of manual effort without automation. The most common example of this in the Nuclear Industry today is automated Corrective Action Program trend coding.

It is expected to jump back and forth between partial and complete automation at some frequency. A common case where this can occur is when the automated task or decision is changed, and the automation system hasn’t learned what those changes are. There will need to be some amount of manual intervention until the system learns the new changes, and full automation can be turned back on. An example of this would be in a “trend-coding” automation system when the “codes” or “tags” applied to data are altered.

Start Using DARSA

DARSA provides a proven roadmap to designing, building, implementing, and iterating an AI-driven automation system. Using this process, organizations embarking on the development of new automation systems can deliver maximum value with minimal risk, using a methodology appropriate for modern AI in practical automation applications.

Visit https://nuclearn.ai to learn more about how Nuclearn uses DARSA to help Nuclear Power Plants achieve AI-driven automation.

Introducing Nuclearn

Today, we are proud to announce Nuclearn Inc, a company founded to help the nuclear power industry harness the benefits of artificial intelligence and machine learning technology.

nuclearn

Nuclearn?

Nuclearn is a machine learning platform that is ready right out-of-the-box with products needed to automate tasks and challenges commonly faced by nuclear power plants. The majority of these products utilize machine learning and artificial intelligence software that is pre-trained and ready for immediate use, specifically for the nuclear power industry.

Why is this important?

A nuclear power plant’s core competency is safely and efficiently operating a nuclear reactor – it isn’t developing software or training machine learning models. However, the nuclear power industry stands to benefit enormously from deploying machine learning to automate routine business tasks. Most industry participants perform tasks very unique to nuclear power, but very common from plant to plant. Many of these tasks are able to be automated with software employing machine learning and artificial intelligence techniques. Once such a task has been automated, the solution is almost always generally applicable to all industry participants.

Nuclearn Inc was started to develop and deploy such solutions.

One unified platform.

Nuclearn organizes all the software necessities for automation into one comprehensive platform. From the platform, you can access product categories that automate certain areas of your business like the Corrective Action Program or Regulatory Compliance, perform training and deployment of individual task automation models, or deploy Nuclearn’s solutions into your business immediately using our easy to use pre-built Excel spreadsheets or do a systems-level integration with REST APIs.

You choose, On-Prem or in the Cloud.

The Nuclearn team has first-hand experience complying with the challenges of nuclear export regulations and sometimes it is just plain easier to keep everything on-premise. This is why the Nuclearn platform was designed to be a true hybrid – equally as powerful and easy to use whether deployed on-premise or used in the cloud. The same Nuclearn platform you see at nuclearn.ai can be easily deployed on a single machine in your on-premise datacenter.

Nuclearn speaks Nuclear.

Nuclearn’s models and techniques aren’t just state-of-the-art, they are custom-tailored for the nuclear domain. This means that Nuclearn has adopted the best-available techniques from the largest technology companies and implemented the latest research from the highest caliber academic institutions and primed them with nuclear specific data. So when you tell a Nuclearn predictive model “Declared feed injection pump B inop after trip on lowflow to SG2, entered LCO 5.4.5”, Nuclearn’s models know what you are talking about.

Nuclearn knows Nuclear.

Nuclearn’s Founders were born in technology and raised in the nuclear power industry and have faced the challenges that come with working in a ‘special and unique’ industry. Nuclearn founder Jerrold Vincent started his career as a Nuclear Data Analyst overseeing nuclear analytics, data warehousing, and business intelligence. Nuclearn founder Bradley Fox started his career as a Nuclear Engineer overseeing the Inservice Testing Program. Together Jerrold and Brad started and grew the largest US nuclear plant’s data science and machine learning competency, culminating in the receipt of the Nuclear Energy Institute’s 2020 Best of the Best Technology Innovative Practice award for Process Automation using Machine Learning. That same skill and knowledge now help Nuclearn solve problems for the entire nuclear power industry.

We are pleased to introduce Nuclearn and look forward to serving the Nuclear Industry for years to come.

Bradley Fox & Jerrold Vincent