How Engineers Use AI

Hey, I'm Sam.

Intro

Disclaimer

AI facts are perishable. Model behavior, vendor terms, court expectations, and bar guidance can change faster than the USPTO can change it's trademark search tool.

Raise of hands

  • Who has used ChatGPT or Claude?
  • Who has used Claude Code or Codex?
  • Who has used Cursor or VSCode for AI?

Agents

An Agent is any AI model that takes action, not just reads and analyzes.

Agents can

  • Review and send a contract
  • Handle a client intake conversation

Agents are not

  • Plain chatbots that answer legal questions
  • Research mode on ChatGPT or Claude

How Engineers Use AI

The basics

  • Verifications and algebra
  • Play dumb
  • Code smells
  • Prompt smells
  • Lean into existing summarizing frameworks
  • Smart Brevity Core 4

Verifications and algebra

Play dumb

Code smells

Prompt smells

Summaries are you friend

  • Smart Brevity Core 4
  • BLUF - Bottom Line Up Front
  • SCQA / Pyramid Principle

How an Engineer mitigates Hallucination

Mitigations

  • Always check the sourced materials - RTFS
  • Give it clear the consequences of faulty data. The more serious the better
  • Always check the sourced materials
  • RTFS

The burger example

Consequences of faulty data

  • I will lose my legal license
  • I could be disbarred

AI will still lie to you

Live demo: OSCN on request

How AI has changed Software work

Comprehension Debt

The gap between what AI can generate and what a human can explain, verify, and maintain.

https://addyosmani.com/blog/comprehension-debt/

Cognitive Surrender

Letting the Agent's judgment replace your own judgment, especially when the work still needs review.

https://addyosmani.com/blog/cognitive-surrender/

The Context Window

Context Window Habits

  • Start with fresh threads as often as possible
  • Keep files, documents, and context you need to reuse in a shared file or project space
  • Don't trust Agents to remember and use what you talked about in that thread

Benchmark Rankings

Benchmark Rankings

  • Don't use benchmarks as God's word.
  • They're just signal help you use your own discernment.

Model Benchmarks

Tool / Harness Benchmarks

Questions

oba-2026.samcarlton.com

QR code linking to the AI for Lawyers article

Article + slides

samcarlton.com/ai-for-lawyers-engineering-world

Resources

Extra

Bleeding Edge to watch

  • Subagents
  • Adversarial Harnesses
  • Claude Dynamic Workflows
  • AI Fatigue

The power of a Hard Drive

Welcome to 1995

Open with the practical frame: this is not an AI hype talk, it is a workflow talk. The question is how to use AI while keeping professional judgment, source review, and accountability intact.

- Sam Carlton - Tulsa native - Been building and consulting on software for over 15 years - VP of Techlahoma w/ Tim - Built products used by NFL, Virgin Brands, Samsung, Aston Martin - Maintain many Open Source projects for the community helping thousands of developers. - Love Grilling, Wagyu, and GrandMA Lighting Consoles.

- Today we're going to explore how AI has taken over the Software Industry - PAUSE - There might be lawyers in the room or online - Under advisement of my legal team I have to put of this disclaimer

- Case & Point: Claude Fable 5. Release on Tue, pulled yesterday at 7:51pm - Backstory: Kate I'm not lawyer

- AI is taking over Software - Finance and legal are next - Kate paid my speaking fee - Finance will just have to be a year yall

- My best effort definition - The lines can be fuzzy

The basics

- Hold back things you already know - Anchor with actual numbers such as: counts - The AI 6 problem

There's so much power

Explain code smells

The more you use these tools you get a feel for when something is off. Lean into that. Interogate the Agent until you feel confident. Mark Manson's video AI Brain video.

Lean into existing summarizing frameworks. - Summaries can give you AI smells early - Worth reading or throw away

- AI will always lie to you eventually - There are tools

- Treat AI as a conduit and never and oracle - Showing you what sources to read is it's superpower

- I will be poisoned

- There are real stakes give them to the AI

Do not let any techniques lull you

- Explain Live Demos and that they always fail - Get 3 OSCN case details from the audience to look up - Review the accuracy with the lawyers

- We can do way more way faster - It's easy to accidentally create useless extra work - It's easy to take longer to get to the same result - Spidering - Lean into one thing at a time

- Lot's of code(or docs) that know human has read much less understands -

- The complete surrender to the Agent's judgment - Tia touched on this with her students - You stop verifying it's outputs - You stop challenging smells

Show of hands - AI has a senior moment

Tie this back to the hard-drive point: reusable context belongs somewhere outside the chat transcript.

Who's familiar with AI Benchmarks? Who's familiar with Legal AI Benchmarks?

- Don't use benchmarks as God's word. - They're just signal help you use your own discernment. - Balance against practitioner's opinions - Sometime it's better to keep using the old tool for a few months

Group these as model or retrieval-model signals. Vals LegalBench is useful for broad model signal. MLEB matters when the workflow depends on retrieval, search, or embeddings rather than chat output. LawBench is jurisdiction-specific, so do not overgeneralize it to U.S. legal work.

Group these as tools, harnesses, workflows, and product-style comparisons. Use them as references, not as a shopping list. Remind the audience to ask what task was measured, what data was used, and whether the benchmark resembles their real work.

Invite questions on workflow, verification, and risk boundaries. If questions are slow, ask: "Where would AI output be hardest for you to verify in your current practice?" Point everyone back to the deck URL for links.

Frame these as starting points, not guarantees. The practical rule: discover through public directories, but inspect source, pin versions, and sandbox before trusting a skill in a real client workspace.

The joke is that local files still matter. Having the right source materials organized and available can be more valuable than chasing a new model. Tie this back to RTFS: AI gets safer when the source set is concrete, available, and reviewable.