Numerical imprecision in loom_infer

[fritzo]

$ python -m loom.benchmark test mixed-10000-100-0.5
# python /home/fobermeyer/sf/loom/loom/benchmark.py test mixed-10000-100-0.5
/home/fobermeyer/.virtualenvs/sf/bin/loom_tare_debug \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/schema.pb.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/rows.pbs.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/tare/ingest/tares.pbs.gz
/home/fobermeyer/.virtualenvs/sf/bin/loom_sparsify_debug \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/schema.pb.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/tares.pbs.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/rows.pbs.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/sparsify/ingest/diffs.pbs.gz
/home/fobermeyer/.virtualenvs/sf/bin/loom_shuffle_debug \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/diffs.pbs.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/shuffle/samples/sample.0/shuffled.pbs.gz \
  0 \
  4000000000.0
/home/fobermeyer/.virtualenvs/sf/bin/loom_infer_debug \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/config.pb.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/samples/sample.0/shuffled.pbs.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/tares.pbs.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/samples/sample.0/init.pb.gz \
  --none \
  --none \
  --none \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/model.pb.gz \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/groups \
  --none \
  --none \
  /home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/infer_log.pbs
ERROR expected super_group close to sub_group; actual {count: 2 mean: 6913.3125 count_times_variance: 5.7614212} vs {count: 2 mean: 6913.3125 count_times_variance: 4}
    /home/fobermeyer/sf/loom/src/product_mixture.cc : 895
    void loom::ProductMixture_::validate_subset_fun::operator()(T*) [with T = loom::NormalInverseChiSq, bool other_cached = true, bool cached = false]
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/fobermeyer/sf/loom/loom/benchmark.py", line 422, in 
    parsable.dispatch()
  File "/home/fobermeyer/.virtualenvs/sf/local/lib/python2.7/site-packages/parsable.py", line 114, in dispatch
    dispatch(args)
  File "/home/fobermeyer/.virtualenvs/sf/local/lib/python2.7/site-packages/parsable.py", line 98, in dispatch
    parser(*args, **kwargs)
  File "/home/fobermeyer/.virtualenvs/sf/local/lib/python2.7/site-packages/parsable.py", line 52, in parser
    fun(*typed_args, **typed_kwargs)
  File "/home/fobermeyer/sf/loom/loom/benchmark.py", line 417, in test
    infer(name, **options)
  File "/home/fobermeyer/sf/loom/loom/benchmark.py", line 257, in infer
    profile=profile)
  File "loom/runner.py", line 248, in infer
    outfiles=[model_out, groups_out, assign_out, checkpoint_out, log_out])
  File "loom/runner.py", line 90, in check_call_files
    check_call(command, debug, profile)
  File "loom/runner.py", line 84, in check_call
    subprocess.check_call(command, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/fobermeyer/.virtualenvs/sf/bin/loom_infer_debug', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/config.pb.gz', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/samples/sample.0/shuffled.pbs.gz', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/ingest/tares.pbs.gz', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/samples/sample.0/init.pb.gz', '--none', '--none', '--none', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/model.pb.gz', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/groups', '--none', '--none', '/home/fobermeyer/sf/loom/data/mixed-10000-100-0.5/benchmark/infer/samples/sample.0/infer_log.pbs']' returned non-zero exit status -6

[fritzo]

Another occurrence:

$ python -m loom.benchmark test mixed-500-500-0.5
# python /home/fritz/sf/loom/loom/benchmark.py test mixed-500-500-0.5
/home/fritz/.virtualenvs/sf/bin/loom_tare_debug \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/schema.pb.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/rows.pbs.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/tare/ingest/tares.pbs.gz
/home/fritz/.virtualenvs/sf/bin/loom_sparsify_debug \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/schema.pb.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/tares.pbs.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/rows.pbs.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/sparsify/ingest/diffs.pbs.gz
/home/fritz/.virtualenvs/sf/bin/loom_shuffle_debug \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/diffs.pbs.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/shuffle/samples/sample.0/shuffled.pbs.gz \
  0 \
  4000000000.0
/home/fritz/.virtualenvs/sf/bin/loom_infer_debug \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/config.pb.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/samples/sample.0/shuffled.pbs.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/tares.pbs.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/samples/sample.0/init.pb.gz \
  --none \
  --none \
  --none \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/model.pb.gz \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/groups \
  --none \
  --none \
  /home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/infer_log.pbs
ERROR expected super_group close to sub_group; actual {count: 2 mean: 1204.1025 count_times_variance: 0.37267995} vs {count: 2 mean: 1204.1008 count_times_variance: 1.4642334}
    /home/fritz/sf/loom/src/product_mixture.cc : 895
    void loom::ProductMixture_::validate_subset_fun::operator()(T*) [with T = loom::NormalInverseChiSq; bool other_cached = true; bool cached = false]
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/fritz/sf/loom/loom/benchmark.py", line 422, in 
    parsable.dispatch()
  File "/home/fritz/.virtualenvs/sf/local/lib/python2.7/site-packages/parsable.py", line 114, in dispatch
    dispatch(args)
  File "/home/fritz/.virtualenvs/sf/local/lib/python2.7/site-packages/parsable.py", line 98, in dispatch
    parser(*args, **kwargs)
  File "/home/fritz/.virtualenvs/sf/local/lib/python2.7/site-packages/parsable.py", line 52, in parser
    fun(*typed_args, **typed_kwargs)
  File "/home/fritz/sf/loom/loom/benchmark.py", line 417, in test
    infer(name, **options)
  File "/home/fritz/sf/loom/loom/benchmark.py", line 256, in infer
    profile=profile)
  File "loom/runner.py", line 248, in infer
    outfiles=[model_out, groups_out, assign_out, checkpoint_out, log_out])
  File "loom/runner.py", line 90, in check_call_files
    check_call(command, debug, profile)
  File "loom/runner.py", line 84, in check_call
    subprocess.check_call(command, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/fritz/.virtualenvs/sf/bin/loom_infer_debug', '/home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/config.pb.gz', '/home/fritz/sf/loom/data/mixed-500-500-0.5/samples/sample.0/shuffled.pbs.gz', '/home/fritz/sf/loom/data/mixed-500-500-0.5/ingest/tares.pbs.gz', '/home/fritz/sf/loom/data/mixed-500-500-0.5/samples/sample.0/init.pb.gz', '--none', '--none', '--none', '/home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/model.pb.gz', '/home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/groups', '--none', '--none', '/home/fritz/sf/loom/data/mixed-500-500-0.5/benchmark/infer/samples/sample.0/infer_log.pbs']' returned non-zero exit status -6