Non-determinism with OpenROAD

Hey there, there might a non-determinism happening with yosys. The following screenshots are from OpenROAD at the Yosys stage, both logs should have no differences among them. They come from a pair of runs with a no-op branch.

I would hope for some guidance on how to investigate further. We seem to be using a different instance master at message 23.10 for Mapping module

With sky130hd/microwatt:

I also created an issue in OpenROAD GH for this at: Yosys non-determinism · Issue #9943 · The-OpenROAD-Project/OpenROAD · GitHub

1 Like

Are these both on the same system or two different systems? And is it the same input for both (not equivalent, but identical)? Both of those have existing issues

Yes, the same system and the same input. I was able to reproduce locally. My colleague also observed the behavior, he created the issue: [Synthesis] nangate45/swerv_wrapper QoR fluctuation due to Yosys abc_new nondeterminism · Issue #4056 · The-OpenROAD-Project/OpenROAD-flow-scripts · GitHub

And he was able to make a workaround at: synth: Add sort before ABC to avoid nondeterminism (workaround) by openroad-ci · Pull Request #4058 · The-OpenROAD-Project/OpenROAD-flow-scripts · GitHub.

He mentioned: “abc_new visits modules in dict insertion order which can vary between runs”

Looking at the synth.tcl, this is (potentially) the second abc run. Do you know if this is being produced with or without the SYNTH_RETIME_MODULES enabled? (more specifically, is this the first or second run of ABC). AFAIK if you’re using the same inputs, ABC should be the only thing that could change the insertion order (since it could be running in parallel and the order could depend on the order that ABC processes return).

@widlarizer you’re probably more familiar with topological sorting than I am. What happens when Toposort::sort() is called with topologically equivalent cells? Do they maintain their original sorting? Is there a benefit to (optionally?) introducing a fallback sort so that topologically equivalent modules have a canonical sort (e.g. by ID)?

This issue does seem to disappear when calling design->sort() before the call to order_modules() in abc_new.cc, though I’m unclear what is causing the modules to be stored in a different order (as far as I can tell the Yosys runs are equivalent up to the iterating through the modules, at least as far as yosys -X output is concerned) so as much as I would prefer to identify and resolve the source of the change in order, just canonicalizing the order instead may be the easier approach.

I checked and abc only gets called once in the report. Something is wrong. Sorting harder is not a solution, everything prior to abc should be deterministic so we have figure out what isn’t and fix it rather than hiding it. The canonicalized inputs are also equal, no src attribute polution or anything. I’ll see, could be related to opt parallelization, but not necessarily, we fuzzed all that I think

1 Like

I believe Jaehyun / Codex has traced this down to using a std::set of pointers which gives iterations of modules in a non-deterministic order.