Messy archive
Migrated folders lost their context
After fileshares move and merge, the old folder path becomes a weak memory of how work actually happened.
old-share/
migration-export/
final/
final-v2-copy.docx
staff-handover/
unknown-owner.xlsx
Work projectApache SolrLegacy data
Solr as a 2nd brain for messy fileshares
Search first. Ask AI second. Keep the sources visible.
Legacy folders were becoming harder to trust: migrated shares, duplicate packs, hidden sensitive files, and metadata trapped inside documents. The project reframed that mess as a searchable index, so teams could find evidence, inspect sources, and act.
Before Solr
folder archaeologyremediation/old-share/export/final/final-v2.docx
archive/handover/silo-2/final-final.docx
planning/permissions/access-list.xlsx
remote-handling/civil/reports/stack-access.pdf
Problem
The folder path is the only map, but the map is out of date.
Solr shift
Each file becomes a searchable document with fields, text, facets, and a source link.
The problem
Fileshares were storing knowledge, but blocking recall.
The OSD and write-up frame the core issue as data entropy: unmanaged shares become less accessible over time, especially after migrations, staff turnover, and weak retention practices.
Solr model
Turn the archive into fields
Solr keeps the original path, then adds searchable fields so discovery is not limited to folder browsing.
fieldPath
fieldowner
fieldmodified date
fieldfile type
fieldextracted text
Interactive Solr lab
Choose a business question and watch Solr narrow the mess.
This is a simplified model of the trial: files become indexed documents, queries retrieve evidence, and facets make the result set explainable.
Query intent
duplicate files in remediation work packs
area:Remediation AND (duplicate_group:* OR title:"work pack")
Solr narrows the collection to remediation records, then promotes documents with duplicate keys and matching work-pack language.
Ranked results
4 shown
Remediation Work Pack - Silo 2 - Final
\\remediation\migrated\area-a\workpacks\silo-2\final.docx
96%
Three files share a duplicate key. This copy has the latest version field and most recent modified date.
RemediationDOCXv3Official
Remediation Work Pack - Silo 2 - FINAL final copy
\\archive\handover\mizan-export\silo-2\final-final.docx
91%
Same duplicate key as the current work pack, but older version metadata and an archive source path.
RemediationDOCXv2Official
BEPPS DIFF File Share Review - Metadata Export
\\bepps-diff\data-review\exports\metadata-review.csv
44%
This record matched through indexed text and metadata fields.
BEPPS DIFFCSVv1Official
Legacy Folder Inventory - Author And Version Gaps
\\remediation\governance\inventories\author-version-gaps.xlsx
38%
This record matched through indexed text and metadata fields.
RemediationXLSXv1Official
Why the top result wins
Remediation Work Pack - Silo 2 - Final
Solr can rank this record because the evidence is structured: title terms, extracted text, source area, document type, classification, version, and modified date are all available as fields.
Source stays visible
\\remediation\migrated\area-a\workpacks\silo-2\final.docx
How Solr works
A second brain needs retrieval mechanics, not just storage.
The useful move is to separate discovery from synthesis. Solr retrieves inspectable evidence first. LLMs can then ask better questions downstream because the context is grounded.
step 01
Crawl the fileshares
\\remediation\migrated\area-a\workpacks
Solr starts by receiving documents from folders, exports, or ingestion scripts. The path is preserved so every result can point back to a source.
User stories from the OSD deck
The point was not search for its own sake. It was faster answers to real operational questions.
Remediation Data Manager
I need to tidy up and discover duplicate and redundant data within our fileshares.
Run Solr queries over remediation shares to identify duplicate files, compare versions, and keep the most current evidence accessible.
Less folder archaeology, fewer redundant copies, and a cleaner evidence base for work planning.
Remote Handling Engineer
I need to discover civil inspection reports that can be replaced with UAV flights by searching for specific key terms.
Search inspection reports for UAV, ROV, drone, route, and access terms, then review the ranked shortlist.
A faster path from inspection archive to remote-handling opportunity.
Classification and compliance stay in scope
The OSD also treats classification as a core constraint: Official-Sensitive files may be indexed, so security settings, access controls, and standards alignment matter from the start.
ISO/IEC 27001GDPRData Protection Act 2018ISO 9001
Impact
The solution turns a passive archive into an active retrieval layer.
The largest change is practical: teams can ask sharper questions of legacy data while still keeping the source record, classification, and governance trail visible.
Discovery
Before
Staff browse inherited folder trees and hope naming conventions hold.
After
Users search by intent, then narrow with fields and facets.
Deduplication
Before
Current, copied, and archived documents compete for attention.
After
Duplicate groups become reviewable candidates with version and date context.
Security
Before
Sensitive material may sit unnoticed inside unmanaged exports.
After
Specific queries surface likely risks for classification and access review.
AI readiness
Before
An LLM would be asked to reason over whatever context someone pasted in.
After
Solr retrieves inspectable sources first; LLM inquiry can happen downstream.
Back to portfolio