SambaTune logs
SambaTune provides detailed logs at different levels for checking the high-level status, debugging errors, and reproducing substeps.
Understand the logs
| Log | Description |
|---|---|
|
High-level status information for processing, compilation, execution, verification, and post-processing. |
|
Debug-level information for reproducing specific steps using compilation or execution subcommands. |
|
Fatal failure information for debugging purposes. |
|
Recording of running messages. |
|
Logs the elapsed time spent on different steps. |
Examples
status_summary.log
SUCCESS : Overall Run Status indicates it is a successful SambaTune run. SUCCESS : Has baseline pefs and SUCCESS : Execute compile instrumentation pefs tells users both baseline and instrumentation compilations are successful.
SUCCESS : Overall Run Status
SUCCESS : Save configuration file
SUCCESS : Save Configuration File (0.007 sec)
SUCCESS : Process user input
SUCCESS : Process User Input (-1.000 sec)
SUCCESS : Execute compile baseline pefs
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net (-1.000 sec)
SUCCESS : Has baseline pefs
SUCCESS : linear_net.py PEF (baseline) (-1.000 sec)
SUCCESS : Execute compile instrumentation pefs
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net___stage_instrumentation_metapipe_buf_cycles (-1.000 sec)
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles___include_read_stall_cycles (-1.000 sec)
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles (-1.000 sec)
SUCCESS : Has instrumentation pefs
SUCCESS : linear_net.py PEF (-1.000 sec)
SUCCESS : Runner
SUCCESS : Compile App Baseline (-1.000 sec)
SUCCESS : Compile App (-1.000 sec)
SUCCESS : Run test lists
SUCCESS : Run snprof by lists (-1.000 sec)
SUCCESS : Run benchmarking by lists (-1.000 sec)
SUCCESS : Execute
SUCCESS : Execute Snconfig (4.668 sec)
SUCCESS : Execute Snprof (-1.000 sec)
...
SUCCESS : Gather instr output
SUCCESS : Gather instrumentation output for --stage-instrumentation=all_stage_cycles:5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles found at /<ARTIFACT_ROOT>/perf_data/instr-count (None (-1.000 sec)
SUCCESS : Gather instrumentation output for --stage-instrumentation=all_stage_cycles --include-read-stall-cycles:5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles___include_read_stall_cycles found at /<ARTIFACT_ROOT>/perf_data/instr-count (stage latency (-1.000 sec)
...
SUCCESS : Gather instr data
SUCCESS : Gather stage latencies (-1.000 sec)
SUCCESS : Gather TLIR (-1.000 sec)
...
...
status_debug.log
If you want to reproduce compilation, you can just rerun the commend under SUCCESS : Execute Compile …. Similarly, if you would like to reproduce execution, you can rerun commdans under for example SUCCESS : Execute Snprof or SUCCESS : Execute pyinstrument.
SUCCESS : Save Configuration File (0.007 sec)
Message: Save configuration file for this run at: /<ARTIFACT_ROOT>/sambatune_gen/benchmark_instrument_linear_net_5_11_14_4_16
SUCCESS : Process User Input (-1.000 sec)
SUCCESS : Validate Arch (0.003 sec)
SUCCESS : Execute Snconfig (4.668 sec)
Command: /usr/local/bin/python3 /opt/sambaflow/bin/snconfig show Node all -j /<ARTIFACT_ROOT>/sambatune_gen/benchmark_instrument_linear_net_5_11_14_4_16/reports/snconfig
SUCCESS : Copy Compile Gen Folder (0.538 sec)
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net (-1.000 sec)
Message: /usr/local/bin/python3 /opt/sambaflow/apps/micros/linear_net.py compile --plot --debug -v -b 1024 -mb 64 --in-features 5120 --out-features 512 --compiler-configs-file /<ARTIFACT_ROOT>/sambatune_gen/benchmark_instrument_linear_net_5_11_14_4_16/test_lists/compiler_configs_5_11_14_4_16_linear_net.json --output-folder /<ARTIFACT_ROOT>/apps/sn20__5_11_14_4_16_linear_net/compile --pef-name sn20__5_11_14_4_16_linear_net --arch sn20
SUCCESS : linear_net.py PEF (baseline) (-1.000 sec)
SUCCESS : Compile App Baseline (-1.000 sec)
SUCCESS : Copy Compile Gen Folder DUPLICATE (1.059 sec)
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net___stage_instrumentation_metapipe_buf_cycles (-1.000 sec)
Message: /usr/local/bin/python3 /opt/sambaflow/apps/micros/linear_net.py compile --plot --debug -v -b 1024 -mb 64 --in-features 5120 --out-features 512 --stage-instrumentation=metapipe_buf_cycles --gen-tensorboard --compiler-configs-file /<ARTIFACT_ROOT>/sambatune_gen/benchmark_instrument_linear_net_5_11_14_4_16/test_lists/compiler_configs_5_11_14_4_16_linear_net___stage_instrumentation_metapipe_buf_cycles.json --output-folder /<ARTIFACT_ROOT>/apps/sn20__5_11_14_4_16_linear_net___stage_instrumentation_metapipe_buf_cycles/compile --pef-name sn20__5_11_14_4_16_linear_net___stage_instrumentation_metapipe_buf_cycles --arch sn20
SUCCESS : Copy Compile Gen Folder DUPLICATE (1.256 sec)
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles___include_read_stall_cycles (-1.000 sec)
Message: /usr/local/bin/python3 /opt/sambaflow/apps/micros/linear_net.py compile --plot --debug -v -b 1024 -mb 64 --in-features 5120 --out-features 512 --stage-instrumentation=all_stage_cycles --include-read-stall-cycles --gen-tensorboard --compiler-configs-file /<ARTIFACT_ROOT>/sambatune_gen/benchmark_instrument_linear_net_5_11_14_4_16/test_lists/compiler_configs_5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles___include_read_stall_cycles.json --output-folder /<ARTIFACT_ROOT>/apps/sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles___include_read_stall_cycles/compile --pef-name sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles___include_read_stall_cycles --arch sn20
SUCCESS : Copy Compile Gen Folder DUPLICATE (1.223 sec)
SUCCESS : Execute Compile sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles (-1.000 sec)
Message: /usr/local/bin/python3 /opt/sambaflow/apps/micros/linear_net.py compile --plot --debug -v -b 1024 -mb 64 --in-features 5120 --out-features 512 --stage-instrumentation=all_stage_cycles --gen-tensorboard --compiler-configs-file /<ARTIFACT_ROOT>/sambatune_gen/benchmark_instrument_linear_net_5_11_14_4_16/test_lists/compiler_configs_5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles.json --output-folder /<ARTIFACT_ROOT>/apps/sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles/compile --pef-name sn20__5_11_14_4_16_linear_net___stage_instrumentation_all_stage_cycles --arch sn20
SUCCESS : Execute pyinstrument (-1.000 sec)
Message: Run Cmd: /usr/local/bin/python3 -m pyinstrument --show-all --renderer=html --outfile /<ARTIFACT_ROOT>/perf_data/pyinstrument/sn20__5_11_14_4_16_linear_net_measure_performance_pyinstrument.html /opt/sambaflow/apps/micros/linear_net.py measure-performance --num-iterations 100 -b 1024 -mb 64 --in-features 5120 --out-features 512 --pef=/<ARTIFACT_ROOT>/apps/sn20__5_11_14_4_16_linear_net/compile/sn20__5_11_14_4_16_linear_net/sn20__5_11_14_4_16_linear_net.pef
...
failures_debug.log
If an error occurred during compilation, you can find the command and stack trace in failures_debug.log.
FAILURE : Run linear.py 'baseline_perf' jobs (-1.000 sec)
Command: python /opt/software/infra/python/run_idc_tests.py --skip-pef-gen --arch=ch --snrdu-opts="--perf" --artifact-root /<ARTIFACT_ROOT> --full-timeout /<ARTIFACT_ROOT>/test_lists/linear_baseline_perf.list --scandump
Message: Error while running linear.py 'baseline_perf' jobs.
Stack Trace:
...
run.log
11-May 14:04:20 - INFO - Starting Sambatune Run 11-May 14:04:20 - INFO - Saved file linear_net.yaml for app index 1 11-May 14:04:20 - INFO - SUCCESS: Save Configuration File 11-May 14:04:24 - INFO - SUCCESS: Process User Input 11-May 14:04:29 - INFO - SUCCESS: Validate Arch 11-May 14:04:29 - INFO - SUCCESS: Execute Snconfig ...
elapsed_time.log
*** Elapsed Time of Events (Highest to Lowest) *** 4.668 sec - Execute Snconfig 1.256 sec - Copy Compile Gen Folder DUPLICATE 1.223 sec - Copy Compile Gen Folder DUPLICATE 1.059 sec - Copy Compile Gen Folder DUPLICATE 0.916 sec - Post Process 0.538 sec - Copy Compile Gen Folder 0.509 sec - Post Process - Instrumentation 0.217 sec - Post Process - DDR Counter 0.101 sec - Copy Instrumentation Data 0.042 sec - Build Overview Layer ...