{"prompt":"Match each driving condition to its primary-strength sensor and the weak-link sensor that fails there","map":[{"condition":"Pitch-dark rural road","primaryStrength":"Radar","weakLink":"Camera","correct":true},{"condition":"Dense fog on the motorway","primaryStrength":"Radar","weakLink":"Camera","correct":true},{"condition":"Child behind a parked van","primaryStrength":"Lidar","weakLink":"Occlusion","correct":true},{"condition":"Clear sunny highway","primaryStrength":"Camera","weakLink":"Radar","correct":true}],"passed":true,"attempt":1}

{"checkpointId":"example_iou_demo","code":"# WORKED EXAMPLE: Intersection-over-Union for one clean box pair.\n# Boxes are [x1, y1, x2, y2] in pixels. (Next slide: you do this on a tougher pair.)\n\ngt   = [60, 50, 200, 150]   # ground-truth box: where the object really is\npred = [68, 56, 206, 156]   # the detector's predicted box (a near-perfect fit)\n\ndef iou(a, b):\n    # 1) the SHARED rectangle (intersection): the overlap of the two boxes\n    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])\n    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])\n    inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)\n\n    # 2) the COMBINED area (union): both boxes together, counted once\n    area_a = (a[2] - a[0]) * (a[3] - a[1])\n    area_b = (b[2] - b[0]) * (b[3] - b[1])\n    union = area_a + area_b - inter\n\n    return inter / union, inter, union\n\nscore, inter, union = iou(gt, pred)\nTHRESHOLD = 0.5\nverdict = \"HIT  (counts as a detection)\" if score >= THRESHOLD else \"MISS (no match)\"\n\nprint(f\"shared area    = {inter:6.0f} px\")\nprint(f\"combined area  = {union:6.0f} px\")\nprint(f\"IoU            = {score:6.3f}   (shared / combined)\")\nprint(f\"threshold      = {THRESHOLD}\")\nprint(f\"VERDICT        = {verdict}\")","stdout":"shared area    =  12408 px\ncombined area  =  15392 px\nIoU            =  0.806   (shared / combined)\nthreshold      = 0.5\nVERDICT        = HIT  (counts as a detection)\n","stderr":"","error":null,"imageCount":0,"durationMs":12,"ranAt":"2026-07-26T11:22:09.767Z"}

{"prompt":"Confirm the IoU threshold at 0.5, compute IoU for the borderline pair, and classify it as hit / miss / phantom; define a false negative as an unmatched real object","iouThreshold":0.5,"pairIoU":0.111,"classification":"miss","falseNegativeDefinition":"a real (ground-truth) object with no predicted box matched at IoU >= 0.5","cellRun":false,"passed":true,"attempt":1}

{"checkpointId":"example_metrics_demo","code":"# WORKED EXAMPLE: score a tiny 8-row detections-vs-ground-truth split.\n# (Next slide: you do this on the detector's full table.)\nimport pandas as pd\n\n# Each row = one scored object. outcome is:\n#   \"TP\" a real object the detector caught,\n#   \"FN\" a real object it MISSED (a false negative),\n#   \"FP\" a phantom box over empty road (a false positive).\ndemo = pd.DataFrame([\n    (\"Pedestrian\", \"TP\"),   # 3 real pedestrians, 2 caught, 1 missed\n    (\"Pedestrian\", \"TP\"),\n    (\"Pedestrian\", \"FN\"),\n    (\"Cyclist\",    \"TP\"),   # 2 real cyclists, 1 caught, 1 missed\n    (\"Cyclist\",    \"FN\"),\n    (\"Car\",        \"TP\"),   # 2 real cars, both caught\n    (\"Car\",        \"TP\"),\n    (\"Car\",        \"FP\"),   # 1 phantom car (braked for nothing)\n], columns=[\"obj_class\", \"outcome\"])\n\n# 1) Confusion counts over the whole split.\nTP = (demo.outcome == \"TP\").sum()\nFP = (demo.outcome == \"FP\").sum()\nFN = (demo.outcome == \"FN\").sum()\nprint(f\"TP = {TP}   FP = {FP}   FN = {FN}\")\n\n# 2) Precision = of everything flagged, how much was real.\nprecision = TP / (TP + FP)\n# 3) Recall = of everything really there, how much we caught.\nrecall = TP / (TP + FN)\nprint(f\"precision = TP/(TP+FP) = {TP}/{TP+FP} = {precision:.2f}\")\nprint(f\"recall    = TP/(TP+FN) = {TP}/{TP+FN} = {recall:.2f}\")\n\n# 4) The safety number: recall on vulnerable road users (pedestrians + cyclists),\n#    then the false-negative RATE = the share of real VRUs we MISSED.\nvru = demo[demo.obj_class.isin([\"Pedestrian\", \"Cyclist\"])]\nvru_TP = (vru.outcome == \"TP\").sum()\nvru_FN = (vru.outcome == \"FN\").sum()\nvru_recall  = vru_TP / (vru_TP + vru_FN)\nvru_fn_rate = 1 - vru_recall\nprint(f\"VRU recall  = {vru_TP}/{vru_TP+vru_FN} = {vru_recall:.2f}\")\nprint(f\"VRU FN-rate = 1 - VRU recall = {vru_fn_rate:.2f}   <- the safety number\")","stdout":"TP = 5   FP = 1   FN = 2\nprecision = TP/(TP+FP) = 5/6 = 0.83\nrecall    = TP/(TP+FN) = 5/7 = 0.71\nVRU recall  = 3/5 = 0.60\nVRU FN-rate = 1 - VRU recall = 0.40   <- the safety number\n","stderr":"","error":null,"imageCount":0,"durationMs":2526,"ranAt":"2026-07-26T11:25:08.891Z"}

{"checkpointId":"ttcs_scored_metrics","code":"# Section 2: score the Aurora-7 detector on its detections table.\n# SYNTHETIC, illustrative detections-vs-ground-truth (no raw imagery, no model runs here;\n# proportions grounded in the published KITTI/nuScenes benchmarks).\n# Each row = one scored object, tagged with class + condition, with an outcome:\n#   \"TP\" caught,  \"FN\" a real object MISSED (false negative),  \"FP\" a phantom box.\nimport pandas as pd\n\n# Read the SAME scored detections table Act 3 slices by condition, so Section 2 and\n# Section 3 can never disagree. (Normalise two column names so the scoring below reads\n# cleanly: class -> obj_class, flag -> outcome.)\ndf = pd.read_csv(\"aurora7_detections_kitti.csv\").rename(\n    columns={\"class\": \"obj_class\", \"flag\": \"outcome\"})\nprint(f\"scored objects in table: {len(df)}\")\n\ndef score(d, label):\n    TP = (d.outcome == \"TP\").sum()\n    FP = (d.outcome == \"FP\").sum()\n    FN = (d.outcome == \"FN\").sum()\n    precision = TP / (TP + FP)\n    recall    = TP / (TP + FN)\n    print(f\"{label:18s} TP={TP:4d} FP={FP:3d} FN={FN:3d}  \"\n          f\"precision={precision:.2f}  recall={recall:.2f}\")\n    return recall\n\n# 1) The whole table, the headline (aggregate) numbers.\nagg_recall = score(df, \"ALL (aggregate)\")\n\n# 2) Vulnerable road users only = pedestrians + cyclists. THIS is the safety subset.\nvru = df[df.obj_class.isin([\"Pedestrian\", \"Cyclist\"])]\nvru_recall = score(vru, \"VRU (Ped+Cyc)\")\n\n# 3) The safety number: VRU false-negative RATE = share of real VRUs MISSED.\nvru_fn_rate = 1 - vru_recall\nprint(\"-\" * 60)\nprint(f\"AGGREGATE recall (all classes) : {agg_recall:.2f}   <- looks reassuring\")\nprint(f\"VRU false-negative rate        : {vru_fn_rate:.2f}   <- the safety number\")","stdout":"scored objects in table: 991\nALL (aggregate)    TP= 822 FP= 41 FN=128  precision=0.95  recall=0.87\nVRU (Ped+Cyc)      TP= 392 FP= 35 FN=122  precision=0.92  recall=0.76\n------------------------------------------------------------\nAGGREGATE recall (all classes) : 0.87   <- looks reassuring\nVRU false-negative rate        : 0.24   <- the safety number\n","stderr":"","error":null,"imageCount":0,"durationMs":59,"ranAt":"2026-07-26T11:26:30.040Z"}

{"prompt":"Score the detector: read off precision, aggregate recall, and the VRU false-negative rate (the safety number)","aggregatePrecision":0.95,"aggregateRecall":0.87,"vruRecall":0.76,"vruFalseNegativeRate":0.236,"safetyNumber":"VRU false-negative rate","readOffBand":"20–30%","readOffCorrect":true,"scoringCellRan":true,"passed":true,"attempt":1}

{"checkpointId":"ttcs_example_slice","code":"# Worked example: slice VRU recall by ONE condition (day vs night).\n# Same move you'll run for every condition next, just for one cut here.\nimport pandas as pd\n\ndf = pd.read_csv(\"aurora7_detections_kitti.csv\")\n\n# Vulnerable road users = pedestrians + cyclists. Keep REAL objects only\n# (a TP was caught, an FN was a real person missed; phantoms are false positives).\nvru = df[df[\"class\"].isin([\"Pedestrian\", \"Cyclist\"])]\nreal = vru[vru[\"flag\"].isin([\"TP\", \"FN\"])]\n\ndef recall(rows):\n    tp = (rows[\"flag\"] == \"TP\").sum()\n    fn = (rows[\"flag\"] == \"FN\").sum()\n    return tp / (tp + fn)            # share of real VRUs the detector actually caught\n\nfor light in [\"day\", \"night\"]:\n    slice_ = real[real[\"lighting\"] == light]\n    r = recall(slice_)\n    print(f\"{light:>5}:  recall = {r:.2f}   ->  {(1-r)*100:4.0f}% of real VRUs missed\")\n\nprint()\nprint(f\"Both together:  recall = {recall(real):.2f}   (the reassuring average)\")","stdout":"  day:  recall = 0.85   ->    15% of real VRUs missed\nnight:  recall = 0.39   ->    61% of real VRUs missed\n\nBoth together:  recall = 0.76   (the reassuring average)\n","stderr":"","error":null,"imageCount":0,"durationMs":35,"ranAt":"2026-07-26T11:29:18.427Z"}

{"checkpointId":"ttcs_triage_slice","code":"# Section 3: slice VRU recall over EVERY condition in the table.\nimport pandas as pd\n\ndf = pd.read_csv(\"aurora7_detections_kitti.csv\")\nreal = df[(df[\"class\"].isin([\"Pedestrian\", \"Cyclist\"])) & (df[\"flag\"].isin([\"TP\", \"FN\"]))]\n\ndef recall(rows):\n    tp = (rows[\"flag\"] == \"TP\").sum(); fn = (rows[\"flag\"] == \"FN\").sum()\n    return tp / (tp + fn) if (tp + fn) else float(\"nan\")\n\nprint(f\"All conditions (the average):  {recall(real):.2f}\\n\")\nprint(\"By condition  ->  recall   (% of real VRUs missed)\")\nfor col, vals in [(\"lighting\", [\"day\", \"night\"]),\n                  (\"visibility\", [\"clear\", \"occluded\"]),\n                  (\"rarity\", [\"common\", \"rare\"])]:\n    for v in vals:\n        s = real[real[col] == v]\n        r = recall(s)\n        print(f\"  {v:<10} {r:.2f}     {(1-r)*100:4.0f}% missed\")\n\nworst = real[(real[\"lighting\"] == \"night\") & (real[\"visibility\"] == \"occluded\")]\nprint(f\"\\nWorst slice (night + occluded): {recall(worst):.2f}   \"\n      f\"{(1-recall(worst))*100:.0f}% of those people missed\")","stdout":"All conditions (the average):  0.76\n\nBy condition  ->  recall   (% of real VRUs missed)\n  day        0.85       15% missed\n  night      0.39       61% missed\n  clear      0.83       17% missed\n  occluded   0.45       55% missed\n  common     0.79       21% missed\n  rare       0.66       34% missed\n\nWorst slice (night + occluded): 0.27   73% of those people missed\n","stderr":"","error":null,"imageCount":0,"durationMs":26,"ranAt":"2026-07-26T11:29:21.357Z"}

{"prompt":"Slice VRU recall per condition, then triage the failure modes by (condition likelihood x miss impact); do not bury a rare-but-catastrophic VRU miss in green","aggregateVruRecall":0.76,"perConditionRecall":[{"condition":"Daylight","recall":0.85},{"condition":"Clear","recall":0.83},{"condition":"Common class","recall":0.79},{"condition":"Rare class (cyclist)","recall":0.66},{"condition":"Occluded","recall":0.45},{"condition":"Night","recall":0.39},{"condition":"Night + occluded","recall":0.27}],"worstCondition":"Night + occluded","worstConditionRecall":0.27,"triage":[{"failureMode":"Night-time pedestrian miss","recallSlice":"night 0.39","likelihood":"Common","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Occluded child at night","recallSlice":"night+occluded 0.27","likelihood":"Rare","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Rare-class cyclist miss","recallSlice":"rare 0.66","likelihood":"Rare","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Occluded pedestrian (daytime)","recallSlice":"occluded 0.45","likelihood":"Occasional","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Night glare phantom brake","recallSlice":"precision dip","likelihood":"Occasional","impact":"Serious","severityBand":"amber"},{"failureMode":"Clear-daylight car miss","recallSlice":"day 0.99","likelihood":"Rare","impact":"Minor","severityBand":"green"},{"failureMode":"Dusk / low-sun pedestrian","recallSlice":"transition","likelihood":"Occasional","impact":"Catastrophic","severityBand":"red"}],"topBandFailureModes":["Night-time pedestrian miss","Occluded child at night","Rare-class cyclist miss","Occluded pedestrian (daytime)","Night glare phantom brake","Dusk / low-sun pedestrian"],"passed":true,"attempt":2}

{"prompt":"For each top failure mode write one ISO 21448 SOTIF chain (triggering condition -> functional insufficiency -> hazardous behaviour -> concrete mitigation); a vague mitigation or a SOTIF gap framed as a fault fails","rows":[{"failureMode":"Occluded child at night","triggeringCondition":"A child steps out from behind a parked vehicle at night, combining severe occlusion with low light conditions.","functionalInsufficiency":"The perception system has inadequate VRU detection under combined night-time and occluded conditions, with recall falling to 0.27.","hazardousBehaviour":"The vehicle detects the child too late or fails to detect the child, causing absent or delayed emergency braking and creating a collision risk.","mitigation":"Restrict autonomous operation in severely occluded night-time environments unless sensor fusion fallback is available to maintain reliable VRU detection.","mitigationIsConcrete":true,"framedAsFault":false},{"failureMode":"Rare-class cyclist miss","triggeringCondition":"A cyclist from a rare or underrepresented class appears in the vehicle's operating environment, especially under challenging conditions.","functionalInsufficiency":"The perception system has insufficient detection performance for rare VRU classes; rare condition recall is only 0.66.","hazardousBehaviour":"The vehicle may fail to detect the cyclist or detect them too late, resulting in delayed or absent braking and a potential collision.","mitigation":"The vehicle may fail to detect the cyclist or detect them too late, resulting in delayed or absent braking and a potential collision.","mitigationIsConcrete":true,"framedAsFault":false}],"mitigations":["Restrict autonomous operation in severely occluded night-time environments unless sensor fusion fallback is available to maintain reliable VRU detection.","The vehicle may fail to detect the cyclist or detect them too late, resulting in delayed or absent braking and a potential collision."],"passed":true,"attempt":1}

{"prompt":"Write the GO / GO-WITH-CONDITIONS / NO-GO ship verdict, each of five fields citing its own dossier section number, internally consistent (no unconditional GO over an unmitigated night miss; no NO-GO if a daylight-restricted release is viable)","verdict":"CONDITIONS","fields":{"definition":"A detection counts as a hit when IoU ≥ 0.5; a real object without a matching detection is counted as a false negative.","metrics":"Aggregate recall is 0.87, VRU recall is 0.76, and the VRU false negative rate is 0.24 (24%).","worst":"The worst condition is an occluded child at night: night + occluded VRU recall falls to 0.27, meaning 73% of VRUs in this slice are missed.","mitigation":"Restrict autonomous operation in severely occluded night time conditions unless sensor-fusion fallback is available to maintain reliable VRU detection.","ask":"Approve deployment with conditions: permit operation only within the restricted ODD and do not allow unrestricted night-time operation until the night + occluded VRU detection gap is mitigated and validated."},"citedSections":["Detection Definition","Metrics","Worst Failure Mode","Mitigation","The Ask"],"internallyConsistent":true,"passed":true,"attempt":1}

{"prompt":"In your own words, write what YOUR dossier would have caught about the real Uber ATG Tempe 2018 crash: which of your own rows, which real NTSB figure, and the mechanism that links them.","analysis":"My night time pedestrian miss row would have caught this risk: night VRU recall was only 0.39, meaning 61% were missed. In the real Tempe crash, the NTSB case reports that the pedestrian was detected about 5.6 seconds before impact, but the system did not resolve the classification and automatic emergency braking was suppressed. This connects directly to the SOTIF chain: poor night time perception and classification created a functional insufficiency that led to absent braking despite a real pedestrian being in the vehicle’s path.","signalsDetected":["row","figure","mechanism"],"passed":true,"attempt":1}