Selbstgesteuertes Projekt
Machine Learning for Automotive Safety
Perception Validation Engineer (ADAS Safety Sign-off) · KI / ML · Juli 2026
Built a perception validation dossier end to end for a synthetic Aurora-7 detector: defined detection correctly with an IoU >= 0.5 match rule, computed precision/recall and the VRU false-negative rate, and then sliced recall by lighting, visibility, and class to expose the dangerous night and occluded VRU failures. The final safety case framed these as ISO 21448 functional insufficiencies and closed with an evidence-based GO-WITH-CONDITIONS recommendation tied to the dossier’s own numbers.
Bewertet nach
- Detection metrics correctness (precision, recall, VRU false-negative rate)30%
- Failure-mode analysis & per-condition rigour25%
- SOTIF safety case quality (ISO 21448 reasoning)25%
- Ship verdict — evidence-based & internally consistent20%
Bestanden · Bestehensgrenze 60/100
Was herausstach6
- Computed and carried forward the correct IoU threshold rule: a match only counts at IoU >= 0.5, with unmatched real objects treated as false negatives.
- Used VRU recall and VRU false-negative rate as the safety metric, explicitly showing why aggregate accuracy is misleading.
- Identified the worst slice as night + occluded with recall 0.27 and kept catastrophic VRU misses in the red band.
- Wrote concrete mitigations such as daylight-only restriction, sensor-fusion fallback, targeted night/rare-class data, and driver-attention requirements.
- Leakage-Free Detection Definition
- Safety-Critical Metric Selection
Meine eingereichte Arbeit12 Aufgaben
Sensor Match
{"prompt":"Match each driving condition to its primary-strength sensor and the weak-link sensor that fails there","map":[{"condition":"Pitch-dark rural road","primaryStrength":"Lidar","weakLink":"Camera","correct":true},{"condition":"Dense fog on the motorway","primaryStrength":"Radar","weakLink":"Camera","correct":true},{"condition":"Child behind a parked van","primaryStrength":"Lidar","weakLink":"Occlusion","correct":true},{"condition":"Clear sunny highway","primaryStrength":"Camera","weakLink":"Radar","correct":true}],"passed":true,"attempt":1}Example Iou Demo
{"checkpointId":"example_iou_demo","code":"# WORKED EXAMPLE: Intersection-over-Union for one clean box pair.\n# Boxes are [x1, y1, x2, y2] in pixels. (Next slide: you do this on a tougher pair.)\n\ngt = [60, 50, 200, 150] # ground-truth box: where the object really is\npred = [68, 56, 206, 156] # the detector's predicted box (a near-perfect fit)\n\ndef iou(a, b):\n # 1) the SHARED rectangle (intersection): the overlap of the two boxes\n ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])\n ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])\n inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)\n\n # 2) the COMBINED area (union): both boxes together, counted once\n area_a = (a[2] - a[0]) * (a[3] - a[1])\n area_b = (b[2] - b[0]) * (b[3] - b[1])\n union = area_a + area_b - inter\n\n return inter / union, inter, union\n\nscore, inter, union = iou(gt, pred)\nTHRESHOLD = 0.5\nverdict = \"HIT (counts as a detection)\" if score >= THRESHOLD else \"MISS (no match)\"\n\nprint(f\"shared area = {inter:6.0f} px\")\nprint(f\"combined area = {union:6.0f} px\")\nprint(f\"IoU = {score:6.3f} (shared / combined)\")\nprint(f\"threshold = {THRESHOLD}\")\nprint(f\"VERDICT = {verdict}\")","stdout":"shared area = 12408 px\ncombined area = 15392 px\nIoU = 0.806 (shared / combined)\nthreshold = 0.5\nVERDICT = HIT (counts as a detection)\n","stderr":"","error":null,"imageCount":0,"durationMs":1,"ranAt":"2026-07-21T01:44:53.321Z"}Task Definition Code
{"checkpointId":"ttcs_iou_classify_run","code":"# Run IoU on this pair, read the number, then classify it (Section 1).\n# Boxes are [x1, y1, x2, y2] in pixels. The object is REAL, there is a ground-truth box.\n\ngt = [60, 50, 200, 150] # ground truth: a real object is here\npred = [150, 96, 280, 196] # the detector's box for this frame\n\ndef iou(a, b):\n ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])\n ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])\n inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)\n union = (a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter\n return inter / union\n\nscore = iou(gt, pred)\nprint(f\"IoU = {score:.3f}\")\nprint(f\"threshold = 0.5 -> {'matches' if score >= 0.5 else 'does NOT match'} a ground-truth box\")\nprint(\"So: a real object, no matching box. Your call on the next panel.\")","stdout":"IoU = 0.111\nthreshold = 0.5 -> does NOT match a ground-truth box\nSo: a real object, no matching box. Your call on the next panel.\n","stderr":"","error":null,"imageCount":0,"durationMs":1,"ranAt":"2026-07-21T01:46:55.989Z"}Task Definition
{"prompt":"Confirm the IoU threshold at 0.5, compute IoU for the borderline pair, and classify it as hit / miss / phantom; define a false negative as an unmatched real object","iouThreshold":0.5,"pairIoU":0.111,"classification":"miss","falseNegativeDefinition":"a real (ground-truth) object with no predicted box matched at IoU >= 0.5","cellRun":true,"passed":true,"attempt":1}Example Metrics Demo
{"checkpointId":"example_metrics_demo","code":"# WORKED EXAMPLE: score a tiny 8-row detections-vs-ground-truth split.\n# (Next slide: you do this on the detector's full table.)\nimport pandas as pd\n\n# Each row = one scored object. outcome is:\n# \"TP\" a real object the detector caught,\n# \"FN\" a real object it MISSED (a false negative),\n# \"FP\" a phantom box over empty road (a false positive).\ndemo = pd.DataFrame([\n (\"Pedestrian\", \"TP\"), # 3 real pedestrians, 2 caught, 1 missed\n (\"Pedestrian\", \"TP\"),\n (\"Pedestrian\", \"FN\"),\n (\"Cyclist\", \"TP\"), # 2 real cyclists, 1 caught, 1 missed\n (\"Cyclist\", \"FN\"),\n (\"Car\", \"TP\"), # 2 real cars, both caught\n (\"Car\", \"TP\"),\n (\"Car\", \"FP\"), # 1 phantom car (braked for nothing)\n], columns=[\"obj_class\", \"outcome\"])\n\n# 1) Confusion counts over the whole split.\nTP = (demo.outcome == \"TP\").sum()\nFP = (demo.outcome == \"FP\").sum()\nFN = (demo.outcome == \"FN\").sum()\nprint(f\"TP = {TP} FP = {FP} FN = {FN}\")\n\n# 2) Precision = of everything flagged, how much was real.\nprecision = TP / (TP + FP)\n# 3) Recall = of everything really there, how much we caught.\nrecall = TP / (TP + FN)\nprint(f\"precision = TP/(TP+FP) = {TP}/{TP+FP} = {precision:.2f}\")\nprint(f\"recall = TP/(TP+FN) = {TP}/{TP+FN} = {recall:.2f}\")\n\n# 4) The safety number: recall on vulnerable road users (pedestrians + cyclists),\n# then the false-negative RATE = the share of real VRUs we MISSED.\nvru = demo[demo.obj_class.isin([\"Pedestrian\", \"Cyclist\"])]\nvru_TP = (vru.outcome == \"TP\").sum()\nvru_FN = (vru.outcome == \"FN\").sum()\nvru_recall = vru_TP / (vru_TP + vru_FN)\nvru_fn_rate = 1 - vru_recall\nprint(f\"VRU recall = {vru_TP}/{vru_TP+vru_FN} = {vru_recall:.2f}\")\nprint(f\"VRU FN-rate = 1 - VRU recall = {vru_fn_rate:.2f} <- the safety number\")","stdout":"TP = 5 FP = 1 FN = 2\nprecision = TP/(TP+FP) = 5/6 = 0.83\nrecall = TP/(TP+FN) = 5/7 = 0.71\nVRU recall = 3/5 = 0.60\nVRU FN-rate = 1 - VRU recall = 0.40 <- the safety number\n","stderr":"","error":null,"imageCount":0,"durationMs":5,"ranAt":"2026-07-21T01:50:48.959Z"}Scored Metrics Run
{"checkpointId":"ttcs_scored_metrics","code":"# Section 2: score the Aurora-7 detector on its detections table.\n# SYNTHETIC, illustrative detections-vs-ground-truth (no raw imagery, no model runs here;\n# proportions grounded in the published KITTI/nuScenes benchmarks).\n# Each row = one scored object, tagged with class + condition, with an outcome:\n# \"TP\" caught, \"FN\" a real object MISSED (false negative), \"FP\" a phantom box.\nimport pandas as pd\n\n# Read the SAME scored detections table Act 3 slices by condition, so Section 2 and\n# Section 3 can never disagree. (Normalise two column names so the scoring below reads\n# cleanly: class -> obj_class, flag -> outcome.)\ndf = pd.read_csv(\"aurora7_detections_kitti.csv\").rename(\n columns={\"class\": \"obj_class\", \"flag\": \"outcome\"})\nprint(f\"scored objects in table: {len(df)}\")\n\ndef score(d, label):\n TP = (d.outcome == \"TP\").sum()\n FP = (d.outcome == \"FP\").sum()\n FN = (d.outcome == \"FN\").sum()\n precision = TP / (TP + FP)\n recall = TP / (TP + FN)\n print(f\"{label:18s} TP={TP:4d} FP={FP:3d} FN={FN:3d} \"\n f\"precision={precision:.2f} recall={recall:.2f}\")\n return recall\n\n# 1) The whole table, the headline (aggregate) numbers.\nagg_recall = score(df, \"ALL (aggregate)\")\n\n# 2) Vulnerable road users only = pedestrians + cyclists. THIS is the safety subset.\nvru = df[df.obj_class.isin([\"Pedestrian\", \"Cyclist\"])]\nvru_recall = score(vru, \"VRU (Ped+Cyc)\")\n\n# 3) The safety number: VRU false-negative RATE = share of real VRUs MISSED.\nvru_fn_rate = 1 - vru_recall\nprint(\"-\" * 60)\nprint(f\"AGGREGATE recall (all classes) : {agg_recall:.2f} <- looks reassuring\")\nprint(f\"VRU false-negative rate : {vru_fn_rate:.2f} <- the safety number\")","stdout":"scored objects in table: 991\nALL (aggregate) TP= 822 FP= 41 FN=128 precision=0.95 recall=0.87\nVRU (Ped+Cyc) TP= 392 FP= 35 FN=122 precision=0.92 recall=0.76\n------------------------------------------------------------\nAGGREGATE recall (all classes) : 0.87 <- looks reassuring\nVRU false-negative rate : 0.24 <- the safety number\n","stderr":"","error":null,"imageCount":0,"durationMs":10,"ranAt":"2026-07-21T01:51:08.790Z"}Scored Metrics
{"prompt":"Score the detector: read off precision, aggregate recall, and the VRU false-negative rate (the safety number)","aggregatePrecision":0.95,"aggregateRecall":0.87,"vruRecall":0.76,"vruFalseNegativeRate":0.236,"safetyNumber":"VRU false-negative rate","readOffBand":"20–30%","readOffCorrect":true,"scoringCellRan":true,"passed":true,"attempt":1}Ttcs Example Slice
{"checkpointId":"ttcs_example_slice","code":"# Worked example: slice VRU recall by ONE condition (day vs night).\n# Same move you'll run for every condition next, just for one cut here.\nimport pandas as pd\n\ndf = pd.read_csv(\"aurora7_detections_kitti.csv\")\n\n# Vulnerable road users = pedestrians + cyclists. Keep REAL objects only\n# (a TP was caught, an FN was a real person missed; phantoms are false positives).\nvru = df[df[\"class\"].isin([\"Pedestrian\", \"Cyclist\"])]\nreal = vru[vru[\"flag\"].isin([\"TP\", \"FN\"])]\n\ndef recall(rows):\n tp = (rows[\"flag\"] == \"TP\").sum()\n fn = (rows[\"flag\"] == \"FN\").sum()\n return tp / (tp + fn) # share of real VRUs the detector actually caught\n\nfor light in [\"day\", \"night\"]:\n slice_ = real[real[\"lighting\"] == light]\n r = recall(slice_)\n print(f\"{light:>5}: recall = {r:.2f} -> {(1-r)*100:4.0f}% of real VRUs missed\")\n\nprint()\nprint(f\"Both together: recall = {recall(real):.2f} (the reassuring average)\")","stdout":" day: recall = 0.85 -> 15% of real VRUs missed\nnight: recall = 0.39 -> 61% of real VRUs missed\n\nBoth together: recall = 0.76 (the reassuring average)\n","stderr":"","error":null,"imageCount":0,"durationMs":10,"ranAt":"2026-07-21T01:55:33.833Z"}Failure Triage
{"prompt":"Slice VRU recall per condition, then triage the failure modes by (condition likelihood x miss impact); do not bury a rare-but-catastrophic VRU miss in green","aggregateVruRecall":0.76,"perConditionRecall":[{"condition":"Daylight","recall":0.85},{"condition":"Clear","recall":0.83},{"condition":"Common class","recall":0.79},{"condition":"Rare class (cyclist)","recall":0.66},{"condition":"Occluded","recall":0.45},{"condition":"Night","recall":0.39},{"condition":"Night + occluded","recall":0.27}],"worstCondition":"Night + occluded","worstConditionRecall":0.27,"triage":[{"failureMode":"Night-time pedestrian miss","recallSlice":"night 0.39","likelihood":"Common","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Occluded child at night","recallSlice":"night+occluded 0.27","likelihood":"Occasional","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Rare-class cyclist miss","recallSlice":"rare 0.66","likelihood":"Rare","impact":"Catastrophic","severityBand":"red"},{"failureMode":"Occluded pedestrian (daytime)","recallSlice":"occluded 0.45","likelihood":"Occasional","impact":"Serious","severityBand":"amber"},{"failureMode":"Night glare phantom brake","recallSlice":"precision dip","likelihood":"Common","impact":"Minor","severityBand":"amber"},{"failureMode":"Clear-daylight car miss","recallSlice":"day 0.99","likelihood":"Rare","impact":"Minor","severityBand":"green"},{"failureMode":"Dusk / low-sun pedestrian","recallSlice":"transition","likelihood":"Occasional","impact":"Catastrophic","severityBand":"red"}],"topBandFailureModes":["Night-time pedestrian miss","Occluded child at night","Rare-class cyclist miss","Occluded pedestrian (daytime)","Night glare phantom brake","Dusk / low-sun pedestrian"],"passed":true,"attempt":2}Ttcs Triage Slice
{"checkpointId":"ttcs_triage_slice","code":"# Section 3: slice VRU recall over EVERY condition in the table.\nimport pandas as pd\n\ndf = pd.read_csv(\"aurora7_detections_kitti.csv\")\nreal = df[(df[\"class\"].isin([\"Pedestrian\", \"Cyclist\"])) & (df[\"flag\"].isin([\"TP\", \"FN\"]))]\n\ndef recall(rows):\n tp = (rows[\"flag\"] == \"TP\").sum(); fn = (rows[\"flag\"] == \"FN\").sum()\n return tp / (tp + fn) if (tp + fn) else float(\"nan\")\n\nprint(f\"All conditions (the average): {recall(real):.2f}\\n\")\nprint(\"By condition -> recall (% of real VRUs missed)\")\nfor col, vals in [(\"lighting\", [\"day\", \"night\"]),\n (\"visibility\", [\"clear\", \"occluded\"]),\n (\"rarity\", [\"common\", \"rare\"])]:\n for v in vals:\n s = real[real[col] == v]\n r = recall(s)\n print(f\" {v:<10} {r:.2f} {(1-r)*100:4.0f}% missed\")\n\nworst = real[(real[\"lighting\"] == \"night\") & (real[\"visibility\"] == \"occluded\")]\nprint(f\"\\nWorst slice (night + occluded): {recall(worst):.2f} \"\n f\"{(1-recall(worst))*100:.0f}% of those people missed\")","stdout":"All conditions (the average): 0.76\n\nBy condition -> recall (% of real VRUs missed)\n day 0.85 15% missed\n night 0.39 61% missed\n clear 0.83 17% missed\n occluded 0.45 55% missed\n common 0.79 21% missed\n rare 0.66 34% missed\n\nWorst slice (night + occluded): 0.27 73% of those people missed\n","stderr":"","error":null,"imageCount":0,"durationMs":25,"ranAt":"2026-07-21T02:20:41.814Z"}Sotif Case
{"prompt":"For each top failure mode write one ISO 21448 SOTIF chain (triggering condition -> functional insufficiency -> hazardous behaviour -> concrete mitigation); a vague mitigation or a SOTIF gap framed as a fault fails","rows":[{"failureMode":"Occluded child at night","triggeringCondition":"zero ambient light (nighttime darkness) combined with total physical line of sight occlusion","functionalInsufficiency":"The camera fails due to darkness, and the LiDAR/Radar are physically blocked by the van. The system works as designed but cannot infer the hidden pedestrian, resulting in a catastrophic perception gap with a measured recall of just 0.27.","hazardousBehaviour":"The perception system fails to pass a target to the control system, resulting in an absent or critically late Automatic Emergency Braking command and a direct collision with the pedestrian.","mitigation":"restriction limiting the system to a daylight only cap. Alternatively, enforce a mandatory driver attention rule requiring the human driver to take over in urban/residential environments at night.","mitigationIsConcrete":true,"framedAsFault":false},{"failureMode":"Rare-class cyclist miss","triggeringCondition":"the detector wasn’t exposed to enough cyclist examples, so it may recognize cyclists less reliably than common objects like cars.","functionalInsufficiency":"The system hardware works exactly as designed, but the software's training distribution lacks sufficient cyclist examples.","hazardousBehaviour":"The perception system fails to classify the cyclist as an obstacle, meaning no target is passed to the control system. The vehicle fails to engage the Automatic Emergency Braking.","mitigation":"Enforce a targeted data collection and retraining cyclist data before deployment, and implement a mandatory sensor-fusion fallback relying on Radar (which detects moving metallic objects like bikes regardless of visual classification).","mitigationIsConcrete":true,"framedAsFault":false}],"mitigations":["restriction limiting the system to a daylight only cap. Alternatively, enforce a mandatory driver attention rule requiring the human driver to take over in urban/residential environments at night.","Enforce a targeted data collection and retraining cyclist data before deployment, and implement a mandatory sensor-fusion fallback relying on Radar (which detects moving metallic objects like bikes regardless of visual classification)."],"passed":true,"attempt":1}Ship Verdict
{"prompt":"Write the GO / GO-WITH-CONDITIONS / NO-GO ship verdict, each of five fields citing its own dossier section number, internally consistent (no unconditional GO over an unmitigated night miss; no NO-GO if a daylight-restricted release is viable)","verdict":"CONDITIONS","fields":{"definition":"A valid detection is registered when the predicted bounding box overlaps the ground truth with an IoU ≥ 0.5. Anything below this geometric threshold is classified as a missed object (a false negative).","metrics":"While the aggregate numbers look fine, the model struggles with vulnerable road users. The VRU recall is 0.76, which leaves us with a catastrophic VRU false-negative rate of 24%.","worst":"The system completely collapses in compound edge cases. In the worst-case scenario—an occluded child at night—the sensor stack is blinded, and the worst slice drops to 0.27 recall.","mitigation":"Because the system cannot currently be trusted to identify humans in the dark, we must implement a strict restriction limiting the system to a daylight-only cap.","ask":"I am asking the board to approve a GO WITH CONDITIONS. The Aurora-7 detector is approved for deployment strictly under a daylight-only cap, and cannot be authorized for nighttime operation until the radar-fusion fallback is thoroughly retrained and re-validated."},"citedSections":["Detection Definition","Metrics","Worst Failure Mode","Mitigation","The Ask"],"internallyConsistent":true,"passed":true,"attempt":1}