I know you've said you've looked at off-the shelf tools, but in that did you consider https://www.transkribus.org/? It's a tool designed for reading historical, hand-written documentation—gets used a lot in archives and historical studies. Might be worth an evaluation to see if your handwriting is not great in similar ways to Dutch bankers from the 18th century.
My father worked with him at CMU and the story he always told (while possibly apocryphal) was that the reason that the ASCII bell character sequence was CRTL-G was because of Gordon.
If you're interested in playing around more with this, it's using a standard called IIIF (https://iiif.io) for image access via API. It's used by many museums, libraries, and archives.
One of the benefits of this is that you can get access to a metadata file that lets you know technical information about the image: It's at https://iiif.micr.io/TZCqF/info.json.
Museums in general have recognized that digital images don't replace the museum experience—and they'd rather provide good images of the art than have people rip off poor-quality ones. Not every institution, and artist copyright remains a huge barrier, particularly for contemporary art, but a LOT of artwork photography is out there in the public domain.
My grandfather spent the last part of his career doing this sort of work—often on classified projects, I'm told. Usually there was an assumption made on the project about the useful working life of any part, and the expected duration of the program, and they made the "right" number of spares.
Things change, and the factory/tooling/people are long gone. Also, often the underlying tech isn't available. But the part needs to weigh the same amount, meet the same guidelines, and fit in the same hole.
It's cheaper to pay someone to spend the time to figure out how to make the replacement then to mothball the entire airplane—almost regardless of how expensive it might be.
This, more than anything else, might be the reason that I stop using Uber. I've loved the idea that I can pay a fixed fee, and that I don't have to think about money at all—having to tip, or think about tips, is so much of what I hated about taxis. Uber was great because it was a fixed amount.
If you want the driver to get more money, raise the price. Don't raise the price by making me feel guilty.
Some context for the picture—my father, Paul, was the business manager for the computer science department at Carnegie Mellon at the time, and is a fabulous record-keeper, so he kept all sorts of interesting things from that era.
He was also one of the founders of Three Rivers Computer as well as PERQ, which were tech transfer spin-outs from the CS dept in the seventies.
Look into http://www.highcharts.com. Much more high level than D3, does nice charts and graphs. Free for non-commercial, and reasonably cheap for commercial work.
The other really useful tool is Tableau Public, if your data isn't confidential. GUI interface, but makes visualizing information really easy.
The other tool I use a lot is http://p5js.org. It's more a general-purpose drawing tool, but if all you want to do is put some boxes at (x,y) positions, it's a really good prototyping tool.
We found the same—using OSM data that just had points was problematic, both because of accuracy and because it tended to often be the front door, not the centroid of the object.
I believe we limited our model generation to selecting places that had outlines, and computed the centroid of that outline. One of the benefits of our technique is that we didn't need to be comprehensive—we can throw out lots of places and still have enough to be useful for the model.
On the front end, it's easy——Add a couple lines to a configuration file, make a couple images, and it's done.
The difficult part is that the search is not optimized to be RAM-efficient; each city takes between two and ten gigabytes of RAM. You also need to have several hundred thousand tile images, which are commercially available, but not free.
If you've got a hefty server and the images, then it takes about a computer-day to compute the features and create the search index.
Not Kyle, but another of the team involved in the project. When we built the training model, we used OpenStreetmap data to find locations of ~1,000,000 "things". A thousand churches, a thousand water towers, a thousand playgrounds, and so on. For each of those locations, we downloaded a satellite image.
The neural net was then trained to look for the things that make each one of those things distinct; what makes a playground different than a church? It could be patterns, it could be colors, it could be any number of things. (For more precise details, you'll need to talk to Aman or Kyle.) It compares lots of things to lots of things, makes some guesses, and then sees whether those guesses help it correctly determine what we told it was in each tile.
Once the model is trained, it's identified the 1024 "features" that are most significant in correctly distinguishing types of things from each other. We then run every tile of a geographical region through that feature determiner, which converts each tile into a point in our 1024 dimensional space. The search function then identifies a tile, looks up its location, and finds the 100 things closest to it within the 1024 dimensional space.
So, TL;dr: It's not looking for colors, it's looking for computable features, which may or may not be color-specific. (Actually, they're highly non-color-specific: the training model randomly "wiggles" the color to makes sure that it doesn't get too tied to a very precise color.)
It might be interesting to look into creating a service for form-letters for various real-estate complaints. I know that previously, I've had to deal with landlords who were withholding my security deposit, or refusing to make repairs, and a way to quickly generate official, legally-correct letters for requesting what I am legally entitled to could be useful.
What you missed was the twenty years where Pittsburgh turned into Detroit. Losing all of your industry followed by 60% of your population does that do a city. As a Pittsburgher, it's a good city with ambitions to be great, but it took a generation to get there.