Model Monitoring with WeightWatcher: Data-Free DEEP LEARNING Diagnostics

Weighwatcher is an open-source Model Monitoring tool that provides Data-Free Diagnostics for production-quality Deep Neural Networks (DNNs). It. can tell you if your model is over-trained or over-parameterized. And it can it tell you which layers are over-trained or under-trained (over-parameterized). And all without needing training or test data.

Let’s see how to do this. First, install the tool.

pip install weightwatcher

Second, pick a model and get a basic description of it

import weightwatcher as ww
watcher = ww.WeightWatcher(model=my_model)
details = watcher.analyze()

WeightWatcher produces a pandas dataframe, details, with layer metrics describing your model. In particular, the details dataframe contains layer names, ids, types, and warnings.

Here is an example, an analysis of the OpenAI GPT model (discussed in our recent Nature paper)

import transformers
from transformers import OpenAIGPTModel,GPT2Model

gpt_model = OpenAIGPTModel.from_pretrained('openai-gpt')
gpt_model.eval();

watcher = ww.WeightWatcher(model=gpt_model)
details = watcher.analyze()

The details dataframe now includes a wide range of information, for each layer, including a specific warnings columns

details[details.warning!=""][['layer_id','name','warning']]

For comparison, below we show the same details deatframe, but for GPT2. GPT2 is the same model as GPT, but trained with more and better data, Being a much better model, GPT2 has far fewer warnings than GPT.

That’s all there is to it. WeightWatcher provides simple layer metrics for pre-trained Deep Neural Networks, indicating simple warnings for which layers are over-trained and which layers are under-trained.

Give it a try. We are looking for early adopters needing better, faster, and cheaper AI monitoring. if it is useful to you , please let me know.