PSCs need the ability to dump rectifier black box contents in production firmware

[isobering]

In customer-support #1440, a PSC in a customer data center keeps disappearing. The current working hypothesis - based on the fact that the PSC says one of the power drops is off - is that one power drop is off, and the other power drop is flopping.

To confirm this, it would be really useful to dump the rectifier black boxes over PMBus to see if/when the power drops are flopping. We've only had to do this a handful of times, and never in production, so we just do humility pmbus to poke the black boxes manually. However, the PSC production firmware doesn't have hiffy so we can't dump the black boxen in the customer rack.

We should come up with a way to let us dump the black box contents that doesn't use hiffy and talk about how to tie that into our fault management system.

[hawkw]

We should probably look at the similar work done for dumping Renesas blackboxen in #1215. I imagine there's stuff in there that it might make sense to borrow for the design of the MGS interface.

[evan-oxide]

We have documentation about how the black box works on the MWOCP68, but not the MWOCP67. So if we want to do this for the Observer PSC as well, we'll need additional information from Murata.

But also, note that the documentation says the MWOCP68 does not save data to the black box when there's an input undervoltage fault. So it might not actually be helpful for debugging a flopping power drop.

[evan-oxide]

Actually we do have some documentation about the MWOCP67's black box as well: section 12 of acan-157.pdf. It looks quite different than the MWOCP68 - if I understand correctly, it only stores data from the most recent fault instead of the last 5 faults, and it does store data on an input undervoltage fault.