Hello, I'm Harshit Kashyap.

Worked example: Ontario Drug Utilization Review

In the saved example, the app identifies checks for interactions, duplicate prescriptions, multiple prescribers or pharmacies, and early or late refills. The answer points to Ontario Drug Benefit Formulary/CDI Edition 43, page 13, section 2. This is a summary of a document passage, not a recommendation about a patient.

Data foundation

I normalized official Health Canada and Ontario sources into a single evidence base before retrieval.

Source coverage

13,343 current Health Canada products and 8,639 Ontario formulary DIN records

Normalized union

15,562 DINs across 12 PDFs, 866 pages, and 30 safety alerts

Reproducible ingestion and retrieval

I record input and output hashes, counts, configuration, and provenance at every stage. I use section-aware chunks to keep regulatory meaning intact, then combine semantic, lexical, and metadata signals for ranking.

Indexed evidence

1,327 chunks, including 1,099 document chunks and 228 alert chunks

Ranking weights

40% dense similarity, 45% BM25, and 15% metadata signals

Chunk policy

750-token target, 900-token maximum, approximately 100-token overlap, no section crossing

Guarded runtime

I keep routing, retrieval, and generation separate. Missing fields trigger clarification or abstention before retrieval, and answers use only selected passages with citations.

• I return official-source evidence with page, section, and URL context.
• My Streamlit interface surfaces the evidence behind each response.
• I built it for document review and education, not diagnosis or treatment advice.

Evaluation and failure analysis

I separate retrieval relevance, answer relevance, groundedness, and correctness instead of hiding them in one score.

Evaluation set

24 development questions, 18 validation questions, and a 50-question sealed holdout

Holdout result

3.95 / 4 correctness and 100% citation validity and presence

Behavior result

95% behavior accuracy, with failures recorded rather than relabelled or tuned away

• Two monograph questions placed the strict gold chunk below the top three results.
• One fallback response included a detail unsupported by retrieved sources.
• One safety request was clarified instead of abstained from.

From disconnected exports to one warehouse

I started with five wide backtest CSVs where discovery was slow and it was easy to confuse market bars, trades, units, and metric coverage. I replaced them with a validated DuckDB and MotherDuck warehouse that I could inspect directly.

Warehouse

4,095,134 market-bar rows, 499 columns, and 5.16 GiB locally

Evaluation partitions

Chronological train, validation, and test splits with symbol, event time, split, and source-file metadata

Governed architecture

I separated tool selection from calculation. I let the model choose what to ask for, while Python and SQL calculate the result through 21 read-only MCP tools and a Streamlit answer and chart.

Tool surface

7 data-discovery tools, 10 performance and risk tools, and 4 feature and safe-SQL tools

User flow

Classify the request, inspect metric coverage, calculate against a split, then explain with caveats

Safety before and during access

I made safety controls part of the product flow, not a post-processing layer.

• I block scope and prompt-injection requests before MCP tools run.
• I accept only SELECT or WITH queries, with four allowlisted tables, EXPLAIN checks, a 20-second timeout, and a 200-row cap.
• I keep database tokens and connection code away from the model.
• I use structured outputs and a ten-prompt session limit to keep the demo bounded.

Worked example: ES cumulative P&L

Saved case gold_019 contains 21 dated points from $0 on December 5, 2022 to $22,233.84 on July 5, 2024, using one ES contract. The grader matched its numeric series but flagged the ES (S&P 500) label. These sampled points do not measure drawdown between observations. The downloadable example contains the original response and dates; I have not presented it as a fresh calculation.

Rubric review, separate from the original score

I preserve the original 46% result. A documented review distinguishes numeric errors, incomplete answers, unsupported unit assumptions, protocol failures, and potential grading artifacts. An alias flag alone does not make an answer correct: one affected case omitted uncertainty, and another used dollar labels for native-unit values. The evaluation report links the case-by-case review and proposed rubric changes, without publishing an adjusted success rate.

Saved evaluation: task quality and tool reliability

I evaluated 50 gold and challenge cases on August 30, 2026 and regraded the saved responses on August 31. This is evaluation v2 with grader contract v2, using gpt-5.4-mini for the agent and grader. The report does not record the code commit, and it does not establish the current live app performance.

End-to-end task success

23/50 cases passed (46%); the stricter overall-agent-success score was 40%

Tool reliability

100% tool execution and response-schema validity, with zero recorded tool hallucinations, call errors, timeouts, or permission violations

Failure breakdown

12 cases failed deterministic checks; 15 more passed those checks but failed answer grading

Answer quality

95.0% claim groundedness and 96.9% claim faithfulness

Next constraint

Reduce the 37.9% redundant-call rate, 7,886 tokens per task, and 7.61-second mean latency

• Task success requires route, schema, chart and expected-tool checks plus a passing answer grade. Successful execution alone does not establish a correct answer.
• Failures included missing expected tools, chart-policy mismatches, incorrect row counts, drawdown sign changes and unsupported assumptions.
• The strict evidence grader also penalized familiar symbol names and one-contract assumptions absent from its supplied evidence. I retain these failures under the saved rules and disclose that limitation.
• The evaluation link on the project card includes all 50 case outcomes, aggregate metrics, grading notes and a downloadable summary.