So, I’ve been playing with using Hermes Agent attached to Grok or ChatGPT (the two services I use) to catalogue my zillion of downloaded articles.\u00a0 This has been a problem for me for years.\u00a0 I have, I think, thousands of downloaded PDFs of various academic articles and books.\u00a0 Most of them are sitting idle in old backup files going back over 10 years.\u00a0 Every once in a blue moon, I think, “Hey, I ought to grab all those and put them in a database.”\u00a0 \u00a0But it is *so* much work to go to each directory, find the pdfs, read them, enter the data into a database or citation software, etc.\u00a0 It’s easier just to download the file again.<\/p>\n
Life is short.\u00a0 Eat dessert first.\u00a0 Plus, I’m old.\u00a0 I don’t have that much time left, and I don’t think God wants me to spend it on that kind of make work.<\/p>\n
Don’t get me wrong, God believes in work.\u00a0 \u00a0In his second letter to the church at Thessalonika, Paul excoriates church members who were lazy and took advantage of the charity of their brethren when they didn’t need to.\u00a0 Paul writes:<\/p>\n
Brothers and sisters, we command you in the name of our Lord Jesus Christ to stay away from every brother or sister who lives an undisciplined life that is not in line with the traditions that you received from us. You yourselves know how you need to imitate us because we were not undisciplined when we were with you. We didn\u2019t eat anyone\u2019s food without paying for it. Instead, we worked night and day with effort and hard work so that we would not impose on you. We did this to give you an example to imitate, not because we didn\u2019t have a right to insist on financial support. Even when we were with you we were giving you this command: \u201cIf anyone doesn\u2019t want to work, they shouldn\u2019t eat.\u201d We hear that some of you are living an undisciplined life. They aren\u2019t working, but they are meddling in other people\u2019s business. By the Lord Jesus Christ, we command and encourage such people to work quietly and put their own food on the table.<\/em><\/strong><\/p>\n So there’s that.\u00a0 But I don’t think that God means us to be doing busywork.\u00a0 We should work, and we should be good craftsmen. The issue of charity involving people who could work but feel entitled is an issue for another day.\u00a0 But regardless, work isn’t everything.\u00a0 We shouldn’t make it the center of our lives (a sin I was subject to at points in my life).\u00a0 Solomon noted in Psalm 112:<\/p>\n I<\/em>f<\/em> God doesn\u2019t build the house,the builders only build shacks.<\/em><\/strong> The old saying that we should work smarter, not harder is not scripture –though I think it should be.\u00a0 Solomon does say in Ecclesiastes 10:<\/p>\n Remember: The duller the ax the harder the work;<\/strong><\/em> That’s a paraphrase from “The Message” Bible, not a “real” translation, but I like it ,so I’ll run with it.<\/p>\n I thought, you know, if I can get Grok to do this while I go eat a burrito, it’s a win-win.<\/p>\n And it works!\u00a0 Kinda.\u00a0 I downloaded Hermes Agent and hooked it up to my Grok account (you don’t have to use the API for either of them for this kind of thing).\u00a0 I turned both Grok and ChatGPT loose and both worked well.\u00a0 Again, Kinda.\u00a0 ChatGPT offers a whole bunch of models to use and, not surprisingly, the newer and better models do better.\u00a0 I have a subscription to “SuperGrok” so I don’t know what the free or cheaper version would do.<\/p>\n The only problem is that I *also* discovered that I had some limits on how much I can do with the subscription service.\u00a0 ChatGPT has limits it sets in 5 hour blocks and week blocks, and Grok has week blocks.\u00a0 It turns outthat looking at 21 pdfs and\u00a0 making an Excel spreadsheet of the results takes about 1-2 percent of my 5 hour allocation in ChatGPT and about 1% of my Grok allocation. I have to either spread out the agent’s work, or pony up a couple bucks now and then.\u00a0 For Grok, I think a reset costs five bucks.<\/p>\n One way to decrease usage is to load a lot of the stuff onto local resources.\u00a0 In other words, instead of having Grok read the pdf file, have Hermes use pdftotext (a linux tool to extract text from pdf files) and have Grok look at the text file.\u00a0 Same thing for optical character recognition tools.\u00a0 \u00a0I’m putting my Hermes agent instructions at the end of this.\u00a0 Unfortunately, the linux tools are not the best in the world.\u00a0 Fortunately, I can do a second pass using Grok or ChatGPT on the failures and they both do well.\u00a0 So fsr, I’ve just been playing with a few files at time, but I am pleased.<\/p>\n Has anybody done this with Grok or ChatGPT at the subscription, not API level?\u00a0 What did you use as instructions, and how successful were you, particularly about using resources sparingly?<\/p>\n I estimate that I have about 8000 documents to scan….<\/p>\n Drop me a line.\u00a0 Or leave a comment.<\/p>\n <\/p>\n Here’s the prompt for Hermes Agent.\u00a0 Grok wrote most of it.\u00a0 The spreadsheet it produced is at the bottom.\u00a0 Took about 4 minutes.\u00a0 No work on my part other than setting it up.<\/p>\n You are running a bounded local PDF inventory on a Linux laptop. Maximize local compute. Minimize Grok tokens, context, and tool-gateway calls.<\/em><\/p>\n Do not upload PDFs. Do not paste full PDF text into the chat. Do not attach files to the model. Do not call web search, DOI resolvers, PubMed, browser, image generation, or TTS. If a local command can do it, use the terminal.<\/em><\/p>\n GOAL<\/em> ROOT<\/em> If the path is missing or unreadable, stop. Do not search $HOME, \/mnt, or other disks.<\/em><\/p>\n HARD LIMITS<\/em> LOCAL-FIRST POLICY<\/em> Preferred local tools, in this order:<\/em><\/p>\n Discovery \/ file facts<\/em> PDF metadata<\/em> Fonts \/ scan detection<\/em> Text extraction<\/em> OCR only if native text is inadequate<\/em> Spreadsheet<\/em> WORK DIRECTORY<\/em> BOOTSTRAP<\/em> PHASE 1 \u2014 LOCAL INVENTORY (no model reasoning)<\/em> Fields per line:<\/em> Selection:<\/em> PHASE 2 \u2014 LOCAL EXTRACTION (no model reasoning)<\/em> For each file write:<\/em> JSON schema:<\/em> Extraction rules:<\/em> Then write a compact model card file:<\/em> PHASE 3 \u2014 MODEL CLASSIFICATION (Grok, cards only)<\/em> Assign exactly one primary_type:<\/em> Optional secondary_type, else blank.<\/em><\/p>\n Cues:<\/em> Do not invent DOI, ISBN, case number, patient name, publisher, or dates. Blank is required when unknown.<\/em><\/p>\n Type-specific fields to fill when present:<\/em><\/p>\n scientific_article \/ preprint \/ conference_paper:<\/em> book \/ book_chapter:<\/em> medical_record:<\/em> trial_transcript:<\/em> deposition:<\/em> trial_report_or_expert_report:<\/em> Always also fill:<\/em> If a PDF is a mixed packet, classify the first dominant document and note multi-document packet.<\/em><\/p>\n Write classifications to:<\/em> If there are 50 cards, classify in batches of 10 locally-prepared groups. Do not reload earlier full cards.<\/em><\/p>\n PHASE 4 \u2014 LOCAL SPREADSHEET<\/em> Output workbook:<\/em> Sheet “inventory” \u2014 one row per PDF, columns in this order:<\/em>
\nIf God doesn\u2019t guard the city, the night watchman might as well nap.<\/em><\/strong>
\nIt\u2019s useless to rise early and go to bed late, and work your worried fingers to the bone.<\/em><\/strong>
\nDon\u2019t you know he enjoys giving rest to those he loves?<\/em><\/strong><\/p>\n
\nUse your head: The more brains, the less muscle.<\/strong><\/em><\/p>\n
\nRecursively scan one root directory, find PDFs, classify each file, extract type-specific fields, and write an Excel workbook. TEST RUN: at most 50 PDFs.<\/em><\/p>\n
\nREPLACE_WITH_PATH<\/em><\/p>\n
\n– Max 50 PDFs processed.<\/em>
\n– If more than 50 exist, take the 50 most recently modified. Tie-break by full path.<\/em>
\n– Only *.pdf \/ *.PDF files.<\/em>
\n– Do not modify, move, rename, decrypt-in-place, OCR-overwrite, or delete source PDFs.<\/em>
\n– Do not follow symlinks that escape ROOT.<\/em>
\n– Skip zero-byte files.<\/em>
\n– Never send raw PDF bytes anywhere.<\/em>
\n– For the model, each file may contribute at most ~1200 words of extracted text, already truncated locally.<\/em><\/p>\n
\nGrok is only a classifier\/normalizer over compact JSON records. All discovery, metadata, text extraction, scan detection, OCR, truncation, and spreadsheet writing happen in the shell or local Python.<\/em><\/p>\n
\n– find, stat, python3<\/em><\/p>\n
\n– pdfinfo (poppler-utils)<\/em>
\n– pdfinfo -meta<\/em>
\n– exiftool if present<\/em>
\n– python3 + pypdf or pikepdf if present<\/em><\/p>\n
\n– pdffonts<\/em>
\n– pdfimages -list<\/em><\/p>\n
\n– pdftotext -layout -enc UTF-8<\/em>
\n– First pass: pages 1-5<\/em>
\n– Second pass if needed: last 2 pages<\/em>
\n– Fallback: python3 + pypdf or pymupdf<\/em><\/p>\n
\n– pdftoppm (png, 200 dpi, first 3 pages only) + tesseract<\/em>
\n– Do not OCR more than 3 pages per file<\/em>
\n– Do not run ocrmypdf against the original file<\/em>
\n– Write OCR text to the work dir only<\/em><\/p>\n
\n– python3 + openpyxl (preferred)<\/em>
\n– fallback: write CSV, then:<\/em>
\nsoffice –headless –convert-to xlsx<\/em>
\n– fallback: xlsxwriter<\/em>
\nDo not hand-build the workbook in prose.<\/em><\/p>\n
\nCreate:<\/em>
\n\/tmp\/pdf_inventory_$$\/<\/em>
\nUse it for text dumps, OCR images, JSONL, and logs. Do not write into the PDF folders except the final .xlsx if ROOT is writable.<\/em><\/p>\n
\n1. Check tools:<\/em>
\ncommand -v find pdfinfo pdftotext pdffonts pdftoppm tesseract python3 exiftool soffice<\/em>
\npython3 -c “import openpyxl” and “import pypdf” or “import fitz”<\/em>
\n2. If poppler tools are missing, try: python3 extractors only. If those are also missing, stop and print the install lines:<\/em>
\nsudo apt install poppler-utils tesseract-ocr tesseract-ocr-eng python3-openpyxl<\/em>
\nDo not invent results.<\/em>
\n3. Print the tool matrix once, then continue.<\/em><\/p>\n
\nIn the terminal, produce:<\/em>
\n\/tmp\/pdf_inventory_$$\/candidates.jsonl<\/em><\/p>\n
\n– full_path, filename, parent_folder, file_size_bytes, last_modified_iso, pages, encrypted, text_source<\/em><\/p>\n
\n– List every PDF under ROOT.<\/em>
\n– Sort by mtime descending, path ascending.<\/em>
\n– Keep first 50.<\/em><\/p>\n
\nFor each of the 50 files, run a local extractor (a single python3 script is best; shell is fine).<\/em><\/p>\n
\n\/tmp\/pdf_inventory_$$\/extracts\/<file_id>.json<\/em><\/p>\n
\n{<\/em>
\n“file_id”: 1,<\/em>
\n“full_path”: “”,<\/em>
\n“filename”: “”,<\/em>
\n“parent_folder”: “”,<\/em>
\n“file_size_bytes”: 0,<\/em>
\n“last_modified”: “”,<\/em>
\n“pages”: null,<\/em>
\n“encrypted”: false,<\/em>
\n“embedded_meta”: {<\/em>
\n“title”: “”, “author”: “”, “subject”: “”, “keywords”: “”,<\/em>
\n“creator”: “”, “producer”: “”, “creation_date”: “”, “mod_date”: “”<\/em>
\n},<\/em>
\n“has_fonts”: null,<\/em>
\n“image_only_likely”: false,<\/em>
\n“text_source”: “pdf_text|ocr|metadata_only|unreadable”,<\/em>
\n“first_pages_text”: “”,<\/em>
\n“last_pages_text”: “”,<\/em>
\n“ocr_pages”: 0,<\/em>
\n“extract_error”: “”<\/em>
\n}<\/em><\/p>\n
\n– pdfinfo for pages, encryption, title\/author\/dates.<\/em>
\n– pdffonts: if no fonts and pdfimages shows large images, mark image_only_likely=true.<\/em>
\n– If encrypted and no password, text_source=unreadable and skip text.<\/em>
\n– pdftotext pages 1-5 into first_pages_text.<\/em>
\n– If that text, stripped, has fewer than 400 characters AND image_only_likely or no fonts, OCR pages 1-3 only.<\/em>
\n– Also extract last 2 pages only when native text from the start is short but the file has many pages (possible back-matter \/ copyright page for books).<\/em>
\n– Truncate first_pages_text to 9000 characters.<\/em>
\n– Truncate last_pages_text to 3000 characters.<\/em>
\n– Collapse runs of whitespace. Keep line breaks.<\/em>
\n– Do not include binary, hex dumps, or embedded XML streams.<\/em><\/p>\n
\n\/tmp\/pdf_inventory_$$\/cards.jsonl<\/em>
\nEach card must be small:<\/em>
\n– file_id, filename, parent_folder, pages, embedded_meta<\/em>
\n– text_source, image_only_likely<\/em>
\n– first_120_lines of extracted text OR first 1500 characters, whichever is shorter<\/em>
\n– last_40_lines only if last_pages_text was extracted<\/em>
\nNo full-text cards.<\/em><\/p>\n
\nRead cards.jsonl. Do not reopen the PDFs. Do not cat extracts\/*.json in full unless a single card is ambiguous, and then only that one file.<\/em><\/p>\n
\n– scientific_article<\/em>
\n– book<\/em>
\n– book_chapter<\/em>
\n– thesis_dissertation<\/em>
\n– conference_paper<\/em>
\n– preprint<\/em>
\n– medical_record<\/em>
\n– trial_transcript<\/em>
\n– deposition<\/em>
\n– trial_report_or_expert_report<\/em>
\n– court_order_or_opinion<\/em>
\n– legal_pleading<\/em>
\n– government_or_agency_report<\/em>
\n– other<\/em><\/p>\n
\n– article\/preprint\/conference: Abstract, Keywords, DOI, journal, volume(issue):pages, received\/accepted, arXiv\/bioRxiv, IMRaD, references<\/em>
\n– book\/chapter: title page, publisher, ISBN, edition, CIP\/copyright verso, chapter list<\/em>
\n– medical_record: patient, MRN, DOB, hospital\/clinic, provider, progress note, discharge summary, labs; confidential<\/em>
\n– trial_transcript: caption, case number, court, official transcript, line numbers, direct\/cross, witness<\/em>
\n– deposition: Deposition of, under oath, noticing attorney, reporter certificate<\/em>
\n– trial\/expert report: expert credentials, retaining party, opinions, Rule 26, case caption<\/em>
\n– order\/opinion: judge, IT IS ORDERED, memorandum<\/em>
\n– pleading: complaint\/motion\/brief, counsel block, certificate of service<\/em><\/p>\n
\ntitle, authors (semicolon-separated), year, journal_or_venue, volume, issue, pages_citation, doi (bare 10.x\/…), pmid_pmcid, keywords, abstract_present<\/em><\/p>\n
\ntitle, authors_or_editors, year, publisher, place_of_publication, edition, isbn, chapter_title<\/em><\/p>\n
\npatient_name, patient_identifiers_found, facility_or_hospital, provider_or_caretaker, record_type, encounter_or_record_date<\/em><\/p>\n
\ncase_name, case_number, court, hearing_or_trial_date, witness_or_speaker, proceeding_type, volume_or_pages<\/em><\/p>\n
\ncase_name, case_number, deponent, deposition_date, taking_attorney_or_firm, location<\/em><\/p>\n
\ncase_name, case_number, report_author, author_role, report_date, subject_of_report<\/em><\/p>\n
\nclassification_confidence (high|medium|low)<\/em>
\nextraction_notes<\/em>
\nshort_evidence (<=240 chars, copied from the card)<\/em><\/p>\n
\n\/tmp\/pdf_inventory_$$\/classified.jsonl<\/em>
\nOne JSON object per file, same file_id.<\/em><\/p>\n
\nJoin candidates + extracts + classified with python3\/openpyxl.<\/em><\/p>\n
\nREPLACE_WITH_PATH\/_pdf_inventory_test.xlsx<\/em>
\nIf ROOT is not writable:<\/em>
\n$HOME\/Desktop\/pdf_inventory_test.xlsx<\/em><\/p>\n
\nfile_id<\/em>
\nfull_path<\/em>
\nfilename<\/em>
\nparent_folder<\/em>
\nfile_size_bytes<\/em>
\nlast_modified<\/em>
\npages_if_known<\/em>
\ntext_source<\/em>
\nprimary_type<\/em>
\nsecondary_type<\/em>
\nclassification_confidence<\/em>
\ntitle<\/em>
\nauthors_or_editors<\/em>
\nyear<\/em>
\njournal_or_venue<\/em>
\nvolume<\/em>
\nissue<\/em>
\npages_citation<\/em>
\ndoi<\/em>
\npmid_pmcid<\/em>
\nkeywords<\/em>
\npublisher<\/em>
\nisbn<\/em>
\nedition<\/em>