I got much inspiration from this. I like the Nietzche quote at the end which is quite different from the usual übermench stuff everybody uses:
"We should consider every day lost on which we have not danced at least once."
I found the other part to be equally true:
"And we should call every truth false which was not accompanied by at least one laugh.""
Our body/brain needs its learning time. You can't rush things. But you can't force things either. If repeating stuff every day feels like a chore, you should try to find other way to do it. Like learning computer stuff: from a book, a blog post, experimenting with random code from Github.
When the thing you are trying to learn becomes something you can't feel like not doing, learning will happen without consious effort. Variety and interest are hard to conjure if you fail all the time or your resistance to change (be it in strength or memory) is great, but finding a different tack on things can make things easy again.
Sometimes the best thing is to do nothing and let things lie for a bit. When you have a goal, you move closer to it just by it being there. Measuring your progress externally (like this video, bugging friends to read your stuff, pull requests to open source projects) helps you learn from mistakes. Mistakes are better way to progress, since I can easily analyze them as opposed to success: what I did do to succeed?
I don't see that web platform's developers chose to reinvent anything. I would put the blame to Microsoft and other closed system makers of 80's and 90's which prevented development of cross platform applications for open content production and consuming. Thus everything OS's can do (and did even in 60's and 70's), needs to be re-implemented in the browsers, in aggreement. This too has taken a long time for precisely same reasons: companies waging war over platforms to lock users in, embracing and extending fledgling standards.
Web development tools are the best we can use these days if we want to reach the broadest audience, benefit from each others experience and further enchance the platform. Other tools make us choose one walled garden where development is rosy and ideas old, all the while hoping it stays alive long enough. I hope web can do what Engelbart's system did in 1968, but this time as open source, documented, findable and usable by every user with every device connected to the same truly global network.
What about document oriented editing? Forms and word processor kind of functionality?
I'm researching browser development frameworks and languages to develop an XML editor. Editing should work more like Google Docs than CodeMirror, and with rudimentary form support (think key-value metadata, not complex forms).
So far I have settled on using contenteditable, but structural editing of XML in document oriented fashion is still an open question. One user action can result in many XML tree changes, like pushing enter twice inside a paragraph should split it in two (and thus close and create any number of tags). Also, there is no schema to describe XML document's editing workflow. Your post on IDE as a value gave me insight for a possible implementation.
Also, big thank you for LT. I'm enjoying using it immensely.
I just finished with it. It's actually quite good to get you started with Postgres. Besides installing and command line usage it introduces many Postgres features and how they differ from MySQL, Oracle and the like.
I am very interested about seeing D3 with Angular. DangleJS is a new one. I have to look through it at a better time.
I'm currently in the process of moving our hand coded D3 visualizations to Angular and trying to decide how to handle graph's parameters (axis, colors, scale etc.), data bindings and interactivity. One option I see is to adopt the Vega D3 library and create directives for its components.
If open sourcing your work wont be possible, perhaps you could tell more about the implementation?
I have yet to test this, since it looks like early days yet.
Wakari is a cloud service and somewhat expensive for my needs. Though its hard to say how big an instance my data would actually require. If my Pandas experimentations don't pan out, I might have to give it a whirl.
The stuff you can do with current IPython notebook is already amazing and immensely useful, as the examples mentioned in the posts demonstrate. Browser is not a good place to manipulate gigabytes of data (like Python), but it is the easieast to use, most versatile rendering engine for visualizations.
I keep envisioning a marriage of Light Table with IPython. If this post is any indication of Light Table's prowess, it could very well work with IPython backend: www.chris-granger.com/2012/05/21/the-future-is-specific/
I mourn the lack of language to build these editors. Schema languages only validate document's structure and content. Browser based editors either abstract editing away to wiki like syntax or hide it inside Word processor emulating widgets. Both come with a set of assumptions that makes them hard to customize to suit the needs of the data.
It should be possible to create a declarative editing and visualization language (a Domain Specific Language) that would drive this browser/data structure server combination. You could describe the editing environment for one particular domain (like exploring and visualizing a statistical dataset). This should be data driven, ie. it builds the environment according to chosen dataset. Language should allow macros or modules in suitable languages for additional functionality, like complex validation, animations and transitions. You can export your view of the data, complete with your code and command history.
Tools for data visualization and editing need to look like they are build for this data, for these particular visualizations. And they should be customizable, right there in the same environment so the link between the data and your knowledge of its domain stays in view. I think these tools hold the promise of becoming toolmakers tools, too.
There was a WebGL 3D flying simulator on HN few weeks back which used the sky and light changes to great effect. Perhaps something like morning rush hour traffic against evening or rainy days againts sunny.
I'll definetely have to delve more into PostGIS. Three.js should follow when I get a better graps of D3. I think there was an example on how to generate a satellite 3D like map from GeoJSON at Mike Bostocks example site.
How did you implement your data path from getting GIS data (shapefiles I presume) to PostGIS then GeoJSON and finally into ThreeJS?
What about combining the statistical data with the location?
And the heatmap/choropleth, is that something Three.js can do?
I have no knowledge of developing 3D stuff, but I have been looking into using D3 with canvas rendering just for this kind of visualizations. I'm playing with multidimensional data as stacked 3D cubes. So, getting something like this would really bootstrap my project to another level. Any thoughts about open sourcing?
In all, this looks very useful and expendable. The sky is the limit. Literally, when is sky coming? ;)
There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).
I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.
Could you perhaps elaborate on the implementation of Sting's in memory database?
"We should consider every day lost on which we have not danced at least once."
I found the other part to be equally true:
"And we should call every truth false which was not accompanied by at least one laugh.""
Our body/brain needs its learning time. You can't rush things. But you can't force things either. If repeating stuff every day feels like a chore, you should try to find other way to do it. Like learning computer stuff: from a book, a blog post, experimenting with random code from Github.
When the thing you are trying to learn becomes something you can't feel like not doing, learning will happen without consious effort. Variety and interest are hard to conjure if you fail all the time or your resistance to change (be it in strength or memory) is great, but finding a different tack on things can make things easy again.
Sometimes the best thing is to do nothing and let things lie for a bit. When you have a goal, you move closer to it just by it being there. Measuring your progress externally (like this video, bugging friends to read your stuff, pull requests to open source projects) helps you learn from mistakes. Mistakes are better way to progress, since I can easily analyze them as opposed to success: what I did do to succeed?