What’s New in Nuclearn v1.4.0?

Nuclearn Platform v1.4.0 is by far our biggest release yet! This release brings a lot of functionality we have been excited about for a long time to our customers. While the detailed release notes are quite extensive, there are 4 major enhancements that will interest most customers:

  • CAP Screening Automation & Workflow Automation General Availability
  • Improvements to Dataset and Dataset Analytics
  • Kubernetes Support
  • Azure AD Authentication Support

CAP Screening Automation & Workflow Automation General Availability

Nuclearn’s Workflow Automation features have been in preview since Q4 2021, and are core to enabling our CAP Screening Automation products. With Nuclearn v1.4.0, these features are now generally available for our customers using our CAP Screening Automation module! This release exposed the capabilities to build automation templates and configurations via the web interface, making it very easy to set up new CAP Screening Automations.

This release ties the Automation workflows much more closely in with our existing Dataset and Model functionality, making it even easier to deploy, maintain, monitor, and audit CAP Screening Automations. Additionally, the functionality added in this release makes it very easy to apply Nuclearn’s Workflow Automation to other processes beyond CAP Screening!

Improvements to Dataset and Dataset Analytics

v1.4.0 brings many new user experiences and stability enhancements to the Dataset and Dataset Analytics feature added in 1.3. These include a far more intuitive user interface, progress bars for monitoring the status of long-running scoring jobs, more flexible analytic options, and numerous bug fixes. These enhancements should make using Datasets for INPO Evaluation Readiness Assessments or Continuous Monitoring even easier!

Datasets UI Updates

Kubernetes Support

With the release of v1.4.0, Nuclearn is now supported on Kubernetes. As many enterprises move their containerized applications to Kubernetes, this is an important addition to the platform. Nuclearn Platform releases now included a Helm Chart for the entire platform and detailed instructions for configuring Nuclearn for Kubernetes. We have found that our deployments are actually easier to configure and install on Kubernetes, in addition to the horizontal scalability and fault tolerance a deployment on Kubernetes provides.

Azure AD Authentication Support

In addition to Active Directory authentication via LDAPS and native application authentication, Nuclearn v1.4.0 includes top-level support for Azure Active Directory (Azure AD) authentication. Customers leveraging Azure AD authentication within Nuclearn are able to SSO into the Nuclearn platform, and easily configure group permissions with Azure AD.

Beyond the most notable items already listed, there are even more miscellaneous enhancements and bug fixes. Additional details can be found in the detailed release notes below.

Nuclearn Platform Release Detailed Notes

v1.4.0

Highlights

  • Workflow Automation General Availability
  • Improvements to Dataset and Dataset Analytics
  • Support for Kubernetes
  • AzureAD authentication support

Updated Web Frontend To v1.2

  • Workflow Automation General Availability
    • Forms for creating, editing and deleting automation templates from scratch
    • Ability to view parent and child automation templates
    • Automation template overview page
    • Improvements to autoflow configuration page
    • Ability to kick off automation config test runs directly from the frontend
    • Several fixes to audit details page for correctness and performance
    • New KPI page for each automation template
    • Automation template can now either render automation configuration for a model or be a “parent” automation with children automations, and pass global parameters down to children
  • Improvements to datasets and dataset analytics
    • Redesigned dataset UI buttons to be more intuitive
    • Adding a progress bar during dataset scoring
    • Added ability to select whether to include first and last time periods in dataset analytics
    • Added “week” as an option for time grouping in dataset analytics
    • Added ability to directly download raw predictions from analytics page
    • Added ability to download an entire dataset as a csv file
    • Improved error messages for dataset uploads
  • Major enhancements to model and model version management
    • Changed model details routes to no longer be product specific
    • Standardized and improved model deployment buttons
    • Added new forms for creating models from scratch
    • Added new forms for creating new model versions and uploaing supporting whl files
    • Model admin page now uses collapsing cards for models and model versions to make UI easier to navigate
    • Most API calls related to models migrated from axios to ReactQuery, which will improve performance and enable better UI in the future
    • Most model react components migrated from legacy classes to react-hooks
    • “Predict Now” can now support models that require more than one input field
    • Fixed bug where UI would not render if a model did not have an associated model version
  • Misc
    • New login page for supporting AzureAD authentication
    • Fixed bug where users had to login twice after their session times out
    • Minor UI fixes to fit pages without scrolling more often
    • Improved loading icons/UI in many place across application

Update Model Engine to v1.3

  • Workflow Automation General Availability
    • Many additional routes for supporting Automation Templates, Configs, Audit, and KPIs on the frontend
    • Added ability to specify parent/child automation config templates
    • Added ability to provide configuration data for an automation config template
    • Refactored “Test Runs” to be generated from an automation template, dataset, and model version instead of just a model version
    • Automation configuration templates can now be tied to a “ground truth” dataset
    • Accuracy is now calculated and saved on the automation data record rather than calculating on the fly
    • Added unique constraint on AutomationConfigTemplate names
    • No max name length limit for automation configuration templates
    • Soft deletes for automation configuration templates
    • Removed hardcoded product ids and automation configuration template ids from routes and operations
    • Updated permissions and roles across all automation config routes
    • Updated testdata routes to still return model labels if no test runs have been completed
  • Dataset & Analytics Improvements
    • Added “week” as a valid option for dataset stat timeslices
    • A central dataset scoring queue is maintained so that multiple requests to score a dataset do not conflict
    • Added scoring progress route to check scoring progress
    • Improvements to csv upload validation, including checking for null UIDs, verifying encoding is either UTF-8 or ascii, and other misc improvements
    • Added route for downloading dataset data as a csv file
    • Added route for retrieving scored model predictions as a csv file
    • Added support to dataset stats for including/excluding first and last time periods
  • Model Deployment & Orchestration Overhaul
    • Support for multiple model backends
      • DockerOrchestrator and KubeOrchestrator added as supported model backends
      • Configuration for multiple backends provided via mre-config “ModelOrchestration” entry
      • Disable undeploying models on startup by setting ModelOrchestration -> undeploy_model_on_startup_failure = false
    • Orchestrators are now mostly “stateless”, and query backends to retrieve model status
    • Major improvements to model binary handling
      • Added routes for creating and deleting model binaries
      • Better support for uploading new model binaries and tying to model versions
      • Significant performance improvement in get_model_binary route
      • Ability to provide pre-signed temporary tokens to model orchestration interfaces to download binaries from MRE rather than MRE having to push model binaries to containers directly
    • Fixed bug where updating an existing model/dataset mapping would fail
    • Added routes for creating new models and model versions
    • Changed model deletions to be soft deletes
    • Removed “basemodel” tables and references as it was no longer used after model refactors
    • Better GRPC error handling
    • All model inference operation now support or use field translations to map input data to required model inputs
  • Kubernetes support
    • Support for MRE sitting behind a “/api” root path by setting the “RUNTIME_ENV” environment variable to “K8S”
    • Added KubeOrchestrator model orchestration interface
  • Azure AD Support
    • Azure AD added as a supported authentication provider
    • Users now have a “username” and a separate “user_display_name”. These are always the same except for users created via AzureAD as AzureAD does not use email addresses as a unique identifier.
    • Added functions for syncing user roles with remote authentication providers
  • Misc
    • Created new model version entries for the Wano PO&C Labeler and PO&C Labeler
      • Old model versions are undeployed by default and new model versions are deployed by default
      • Existing integrations via the APIs may break if they specify v1 of the labeler models
    • Configuration file location is now set via the “CONFIG_FILE” environment variable
    • Added support for deprecating and removing API routes
      • Deprecated routes can be forced to fail by setting the “STRICT_DEPRECTATION” environment variable to “true”
      • Deprecated following routes:
        • PUT /models/initial
      • Removed following routes:
        • DELETE /models/{model_id}/version/{version_number}/
        • GET /models/{model_id}/version/{version_number}/basemodel/binary/
        • PUT /models/{model_id}/version/{version_number}/binary/
        • POST /models/{model_id}/version/{version_number}/testrun/
    • Allow field descriptions for source data record fields to be null
    • Trigger added to SourceDataRecord table to keep full version history in a SourceDataRecordHistory table
    • Improved logging and error handling by logging in more places and creating more specific errors
    • Switched many routes to “async” to reduce blocking operations in FastAPI
    • Fixed bug preventing admins from updating user roles

Update Application Object Storage to v1.1

  • Updated WANO & PO&C model to 0.0.5, fixing a protocol bug where JSON serialization artifacts may be included in the input into the model

5 Reasons AI is the Promising Future of Nuclear CAP

In the near future, the Nuclear Corrective Action Program (CAP) will be sleek, streamlined, and highly efficient; where occasionally humans participants are required to review and deliberate over only the most complicated issues requiring their vast experience and wisdom. For everything else, a trained army of CAP AI agents invisibly process issues, review and alert on trends, assign corrective actions, and take feedback from human coaches via purpose-designed human/AI interfaces.

No longer will a team of humans be subject to hours upon days of analysis for trend detection, a Senior Reactor Operator forced to process another condition report about a cracked sidewalk, or an Engineer left waiting for a corrective action item to be issued to her inbox. These functions will have been largely automated with the focused application of AI-based technology. Here are the five reasons this future is highly probable, based on both the current state of the Nuclear Industry and leading-edge AI technology.

Cost Savings and Improved Quality

It comes as no surprise to anyone that has worked in the Nuclear Industry that running an effective CAP program is expensive. CAP demands a significant investment into human resources that have adequate experience to effectively diagnose and resolve the problems experienced in an operating power plant. In practice, this requires either dedicated staffing or rotating employees out of primary roles to fulfill a CAP function.

By applying intelligent automation to the Screening, Work Generation, and Issue Trending processes, a resource reduction of approximately 45% is expected.

Beyond reducing the number of resources required, AI reduces the total amount of time required to execute portions of the CAP process. While a human screening team may only be able to process conditions on a daily basis, an AI system can review and screen conditions and issue work items immediately. More quickly getting workable tasks into the hands of employees saves money and improves CAP quality.

For those issues that may be too complex for AI to effectively handle, a human-in-the-loop system can be employed, where AI knows when it is unsure and can reach out for human assistance. By using human-in-the-loop the cost of the CAP program is reduced while keeping quality the same or better.

Additionally, AI can lower the threshold for issue documentation. Deployment of an information extraction AI lets employees more naturally capture issues using natural language, without filling out specialized forms. When issues become easier to document, they are documented more often, the overall information input into the CAP program increases, and the chance an issue is corrected becomes greater. AI that immediately evaluates the quality and completeness of the submitted report enables automated dialogue with the submitter. This can encourage behaviors such as adding information, clarify issues, correcting spelling, or otherwise encourage behaviors that promote report quality, increasing the effectiveness of the overall CAP program.

Scale

The most valuable problems to solve are frequently the largest. CAP and associated activities are one of the largest opportunities in Nuclear. CAP lies at the heart of the Nuclear Industry, and requires participation from almost every trade and profession at each site. The ubiquity of CAP combined with the savings potential provides an immense incentive for plant operators, industry vendors, and industry research groups to discover and implement ways to make these programs run more sustainably and efficiently. Specialized AI that can automate various tasks are at the top of mind of industry groups such as the Electric Power Research Institute, the Idaho National Laboratories, and various utility in-house teams.

A fortunate side effect of the CAP program is the production of large quantities of high-quality data – data ideal for training the AI systems that will be used to automate the same functions. Most of this data is captured in free-form text as natural language. Language with a specific Nuclear vocabulary and dialect, but natural language nonetheless. The scale of this data puts it on par with the datasets utilized by the large technology vendors and academic institutions to develop and train the most effective AI systems. Thanks to the scale of Nuclear CAP data, these large AI systems can be specialized to operate in the Nuclear domain – increasing performance and effectiveness for the tasks at hand.

Transportability

The most notable advancements in AI technology of the late 2010s were around the development of advanced natural language-based AI. This AI has the ability to understand human language more naturally and effectively than previously thought possible. Breakthroughs in this area are characterized by the ability of AI to transfer learning from one problem to another. An AI good at classifying condition report quality will be better at identifying equipment tags vs one specifically trained just to identify equipment tags.

The benefit for the nuclear industry is that an AI system trained at Plant X will be able to transfer its learning to Plant Y and be more performant than one trained at just Plant Y. This is similar to how a working professional at Diablo Canyon would more easily adapt and apply their knowledge when transferring to Turkey Point than someone not having worked in the nuclear industry at all. Similar to a human employee, an AI system will benefit from the variety of different knowledge obtained from general industry data. Learning specifics for any one plant will be faster, cheaper, and easier for any plant wishing to specialize the AI system for use in automation once trained on general industry data.

As a result, solutions developed at one site will be able to be shared. With commonly applicable training and similar problems, the industry can work to solve the big problems once with ‘large’ or ‘hard’ AI, and transport the solution from plant to plant for the benefit of the entire industry.

Automated Screening

One of the more specific solutions apparent when applying AI to the CAP process is the automation of the condition screening process. Condition screening is the process of reviewing a received report of a non-standard condition in or around the plant, then applying certain tags, codes, or classifications, assigning an owner, and generating the appropriate work items that address the condition. For some plants, this process involves dedicated groups of senior employees that work daily to manually perform this process. For others, this involves dispersed resources periodically gathering together to complete screening. In either case, the resources are usually senior-level and experienced, and thus expensive. The following estimation of resources spent by the industry for this process each year illustrates just how large an opportunity there is:

The screening process has certain properties: repeatability and complexity of task, quality of data, scale, cost, etc. that make it extremely promising to apply AI-powered automation — discussion worthy of a separate blog post…coming soon!

Automated Trending

Automated trending is the sequel to Automated Screening – it’s what comes after the conditions have been identified and actions issued. Normally done ‘cognitively’ or via brute force search of the condition data, trending is resource-intensive and largely manual. Read Nuclearn’s Nuclear CAP Coding AI – Better Performance at a Lower Cost to find out more about how AI can help automate and simplify the trending task.

Bonus: The Rapid Progress of AI Technology

The five points above are only achievable due to the explosion in the progress of various technologies that underpin how AI learns and behaves. The speed in recent years with which new AI tools achieve human-level performance on various vision and language tasks is unprecedented. As seen in the chart below, developing AI that can recognize simple numerical digits at human-level performance took over 10 years; to recognize cats, dogs, cars and other everyday objects in images took about 5 years. More recently, developing AI that can recognize and manipulate human language took only about 2 years.

The accelerating pace of AI advancements shows no sign of stopping anytime soon. This type of rapid advancement, combined with the scale, transportability, and savings of CAP, allows Nuclearn to confidently say AI is the future of Nuclear CAP.

 

Nuclear CAP Coding AI – Better Performance at a Lower Cost

Over the last few years there has been a lot of discussion in the domestic Nuclear industry about using Artificial Intelligence to automate Nuclear Corrective Action Program (CAP) coding. The general idea is that sites can apply Artificial Intelligence algorithms to process large amounts of historical CAP data, those algorithms learn how Condition Reports and trend codes relate, new data is fed through those algorithms, and codes are automatically applied. While this is an exciting development in an industry with a long history of manual administrative processes, only a few sites have actually been able to adopt automated coding. To learn how sites can better adopt AI to automate Nuclear CAP coding, we will cover the economic pressures facing Nuclear Power, discuss the important role of coding in CAP, and present new tools for enabling AI-driven automation of Condition Report coding.

Cost Savings in Nuclear CAP

There is a simple, unavoidable fact looming over the Nuclear Power industry in the United States.  The cost per kilowatt-hour of Nuclear power has been increasing year over year for a decade, while existing Natural Gas and new Solar Generation costs are decreasing.  Simply put, Nuclear Power Plants need to become cheaper to operate in the very near future if they want to remain economically viable.

Data derived from Lazard Levelized Cost of Energy Analysis 2012-2020

This isn’t a secret to anyone who has worked in Nuclear Power over the last few years.  In fact, the entire industry mobilized to combat rising costs in a combined effort under the “Delivering the Nuclear Promise” initiative.  While this initiative helped slow down cost increases in the industry, it has still not been enough.  Costs for competing generation sources are closing in, and sites have to do more if they want to remain relevant.

One opportunity for savings comes from the “Corrective Action Program” (CAP) each site has.  Nuclear Corrective Action Programs are a key part of ensuring the safe continuing operation of a plant, are required by law, and are tremendously expensive to run and administer.  There are plenty of opportunities for cost savings in CAP, but in this article we will address one: using Artificial Intelligence to automate the trend coding of Condition Reports.  

But before we can get into the solution, let’s first cover the economics & challenges of today’s prevalent manual approach to Condition Report coding.

Trending Isn’t Easy…

The average Nuclear reactor will log between five and ten thousand Condition Reports in their CAP program each year.  It is not uncommon for multi-unit sites or fleets to have hundreds of thousands, even millions, of Condition Reports in in their historical datasets.

The Nuclear Regulatory Commission (NRC) & the Institute of Nuclear Power Operations (INPO) expect healthy CAP programs to include the analysis and trending of aggregate data.  This is a daunting task for many plants, exacerbated by an inherent attribute of CAP data.  Data collected in a Condition Report usually only includes the following information:

  • Title
  • Description of the issue
  • Affected equipment/procedures
  • Actions taken

3 of the 4 key data elements of a Condition Report are text.  Nuclear-specific, convoluted, jargon-heavy text.  And anyone that has done data analysis on text-heavy data sets can tell you that aggregate analysis and trending of text data is almost impossible using traditional analytical techniques.

“During performance of IST AF43-9PT4F, AFP A tripped on an overspeed signal.  This resulted in an unplanned entry into LCO 1.54, and mandatory entry into Mode 2 within 2 hours.  SRO cleared the alarm, and AFP A was declared OPERATIONAL after performing re-test.”

Share a few thousand of these with an analyst from a different industry and see how well they can trend (they won’t get very far)

Domestic Nuclear sites have recognized this challenge, and combat it by having professionals with extensive Nuclear experience bucket their Condition Reports by tagging them with one or more “codes”.  By tagging Condition Reports with one or more of these codes, plants find it much easier to trend and analyze their issues, understand which performance areas they may be doing poorly in, and benchmark themselves against the rest of the industry.  Many sites have developed their own coding schemes, but recently many sites have begun tagging their issues with one or more of the INPO Performance Objectives & Criteria (PO&C).  The INPO PO&Cs are site-agnositc list of nearly 1,000 performance criteria that plants must meet to be considered high-performing.  

Trending Isn’t Cheap…

While this process of “coding” issues is an important part of a healthy CAP program, it comes with a significant cost and several drawbacks.  The “coding” process is very time consuming and requires someone with an extensive background in many different areas of Nuclear Power, as well as an intricate knowledge of their coding scheme.  Our experience with “coding” processes is that they take three to five minutes per Condition Report.  When we combine this with the typical inflow of issues at a domestic Nuclear reactor, and consider a reasonable loaded hourly rate for a nuclear professional…

… we reach the staggering number of $40,000 per year per reactor spent just coding Condition Reports.  If we take into account that this role will typically require at least one dedicated individual at a multi-unit site, and that that individual will need to spend a significant amount of time in training, PTO, and activities required of a nuclear professional, we can reach an even higher number.

In addition to costs, there are other challenges with manual coding practices.  There are significant differences in the “styles” used by different individuals when coding, which can result in unpredictable labeling practices.  When the people performing the coding change, or when additional individuals help with coding during an outage, it is common to see drastic changes in how issues are coded and what codes are used.  This can result in “trends” and “spikes” in the data that have nothing to do with underlying plant issues, but are instead the byproduct of a manual coding process.  Additionally, it is not uncommon for coding schemes to change.  The INPO PO&C codes for example have received several revisions over the years. When faced with a revision, plants are faced with either losing their historical data, paying tens of thousands of dollars to “re-code” historical data, or use inexact “translating” processes to transform their historical data into the new schemes.  None of these options are particularly appealing.

Artificial Intelligence Can Help

What if plants could get the majority of the benefit of coding, eliminate some of the drawbacks, and save costs while doing so?  Recent advances in Artificial Intelligence techniques for text and “natural language processing” can allow them to do just that.  Using a subset of AI called “Machine Learning”, it is now possible for a computer algorithm to process historical CAP data to learn the correlations between Condition Reports and the codes that have been assigned to them.  After training is complete, new Condition Reports can be fed through the model and codes can be applied automatically using the learned historical patterns.

This chart shows how an AI agent “Aristo” scored on the New York Regents eighth-grade science test over time.

This approach has numerous advantages over manual coding.  A machine learning algorithm will always code the same Condition Report the same way, removing the inherent variability in manual coding.  Additionally, a machine learning algorithm can code 24×7, allowing sites to do real-time trending around the clock (or just not have to wait until a “coder” is back in the office from PTO).  When new coding schemes come out, sitescan train a model on a relatively small number of new examples and apply it to historical data – allowing them to retain historical trends by re-coding data efficiently.  

So why haven’t plants adopted AI-drive automated coding en masse?  Well, Machine Learning isn’t easy, and it especially isn’t easy to do on text-heavy datasets with complex domain-specific vocabularies (looking at you Nuclear).  Sites that have looked to traditional Nuclear software vendors have found that they lack the technical capabilities to build and train Machine Learning models to a reasonable accuracy level.  Others that have looked to larger tech firms have found price tags well into the hundreds of thousands of dollars, with no guarantee that the provided solutions will actually work.

To have any chance of adopting AI-driven automation of Condition Reports, sites need an affordable solution built by people that know State of the Art Artificial Intelligence techniques and know Nuclear.

Nuclearn.ai to the Rescue

We at Nuclearn have built just that: an affordable Machine Learning algorithm that applies PO&C codes to Condition Reports.  As part of our free “Cloud Basic” subscription, anyone can start automatically coding their Condition Reports.  The process to get started is exceedingly simple – register an account and download the “Condition Report Trending Dashboard” excel sheet from the CAP AI module, and follow the instructions to input your Condition Report texts!  The pre-built excel sheet will send Condition Reports through our PO&C algorithm, predict the PO&C codes, and automatically trend them in just a few minutes.

*Note: As of v1.4, the CAP Report Trending Dashboard is deprecated in favor of our new analytics module. Learn more by watching this video: https://youtu.be/rxsq5CF72ss

Now what about cost?  As discussed earlier, any AI-driven solution needs to be competitive with the $40k a year per unit price tag for manual coding.  Nuclearn’s free Cloud Basic subscription offers 1,000 predictions free per month at no cost.  For most Nuclear sites in the U.S., that means that adopting Artificial Intelligence based coding can be done for free, right now.

Nuclear CAP Coding AI Dashboard
Example of the visualizations included in the “Condition Report Trending Dashboard”

Some sites may find that Nuclearn’s PO&C algorithm performs well, but that they either have unique Condition Reports unlike the rest of the industry, or desire a different coding scheme from the standard PO&Cs.  In that case, sites can contact us directly, and we can discuss our on-premise platform that can train directly on your data and meet your needs.  And we can guarantee for PO&C coding, Nuclearn’s solution will be substantially cheaper than $40k per reactor per year.

Nuclearn is here to help Nuclear Power Plants adopt AI-driven automation of Condition Report coding, and save money in the process. Sites need to automate key processes, such as Condition Report trending, if they want to remain economically viable. Read Introducing Nuclearn to learn about why we started Nuclearn, and visit https://nuclearn.ai to learn about other processes that Nuclearn can help automate.