How To Use AI To Track Your Home's Cost Basis (and Catch Tax Mistakes)

I've been doing my own taxes with TurboTax for a while. It works fine one year at a time, but it doesn't really keep track of anything across years.

That turns out to matter if you own a house. Your cost basis is what you paid for the house plus permanent improvements (a remodel, a new roof, windows, a fence). When you sell, your gain is figured against that number, so every receipt you can't find is money you might pay tax on later. If you have a rental, it gets worse: depreciation every year, rental days, carried-over losses, and sometimes another state's return.

I had some houses and years of paperwork scattered across my NAS, Google Drive, and email. So I had an AI agent dig through all of it and build a ledger for me. Here's how I set it up and what happened.

Give it read-only access

I didn't want anything getting changed or deleted, especially on the NAS since that's my backup. So everything was read-only:

• Gmail, Calendar, and Drive through gog, a command-line client for Google, using my own OAuth client
• Monarch (my budgeting app) through its MCP server, to match up bank payments. If you want to try it, that's my referral link.
• The NAS, copy only

All the documents got copied into one folder on my laptop. The agent pulled the text out of every PDF and ran OCR on scans and phone pictures of receipts.

Put it in a SQLite database

I wanted something I could back up easily and that kept a history of changes. A single SQLite file does both. It has tables for each property, how it was used over time (lived in, rented, vacant), documents, improvements, and depreciation.

The part I like: every change has to include a reason, or the database rejects it. Nothing ever gets deleted. If something's wrong it gets marked as replaced, so you can always see what changed and why.

The other rule is that the agent can only propose things. Nothing counts until I confirm it.

Answer a lot of questions

The agent split the work into a few sub-agents, one per group of tax years plus one searching email for closing statements and contractor invoices. Then it came back with questions like:

• "There are a few checks to a contractor in 2020. What were they for?" (Bathroom remodel. It should've been depreciated and wasn't.)
• "There's a quote for a deck from a few years back. Did that happen?" (Nope.)
• "The contractor billed in stages and I only found two of the invoices. Was the rest paid?" (Yes. I just don't have receipts.)

For stuff where I don't have a receipt, it records my answer as the evidence and notes it. Not perfect, but better than nothing, and it's written down with a date.

What it found

A few mistakes on my old returns:

• Depreciation that got calculated on Form 4562 but never made it onto Schedule E
• A big improvement on a rental depreciated over 5 years instead of 27.5
• Vacant months between tenants counted as personal use days
• Improvements made right before renting the house out that never got included

The annoying thing is your basis goes down by the depreciation you were allowed to take, even if you didn't take it. So these go on a list for a CPA.

It's not perfect

It told me I hadn't filed one of my state returns. I had. It was buried deep in the return PDF, but that page's text was garbled, so searching it came up empty. I pointed at the page, and it fixed the text extraction to OCR pages like that.

It also grabbed a purchase price out of an old spreadsheet of mine. That was a projection, not what I actually paid. The closing disclosure had the right number.

Both got caught because every number points back to a document and I'm the one confirming. I wouldn't do this any other way.

Keep it going

Now there's a job that runs once a month. It:

• Checks my email for anything that looks like tax or property paperwork
• Lists deadlines coming up (property tax, filing, extensions)
• Writes an updated backup to the NAS
• Sends me a notification

It doesn't decide anything. When I sit down to deal with it, the open questions are already sorted into stuff for me, stuff for the CPA, and stuff the agent can handle.

If you try this

• Keep it read-only and keep your documents local
• Make it point to a document for every number
• Quotes and spreadsheets aren't receipts
• Expect to answer a lot of questions
• It's not a CPA, but you'll show up to one way more organized

Not tax advice, just what worked for me.

Good luck!