Ask HN: Which industry sector should I target?
20 comments
What do you want to optimise around? Stable job, 9-17, decent salary? Lost of money, caution to the wind? Be something [for one or more of several values of 'something'] to a lot of people? Fiddle with deeply intellectual CS problems, shielded from pedestrian worries about revenue and users?
I personally enjoy the knowledge of having solved a real (business) problem for real people, and being appreciated both financially and personally for this, even when the actual tech going into this isn't very complicated, and especially, doesn't tick off a lot of "hip" boxes.
I have also met good people who do not thrive in these circumstances, so it's hard to give generic advice as to what might make you happy.
I personally enjoy the knowledge of having solved a real (business) problem for real people, and being appreciated both financially and personally for this, even when the actual tech going into this isn't very complicated, and especially, doesn't tick off a lot of "hip" boxes.
I have also met good people who do not thrive in these circumstances, so it's hard to give generic advice as to what might make you happy.
Is it possible to go the "work on deeply intellectual problems while shielded from pedestrian worries about revenue and users" route without academia? I suppose just do it on the side until you invent something which is valuable to industry.
I know of departments in some of the investment banks that are essentially CS R&D. There are some hedge funds (HFT and otherwise) that are basically playgrounds for PhD.
Of course, in both cases, your work is ultimately directed by revenue, the shield I'm talking about is more against the tight, client/user responsive iterative process of most modern software development which can make it difficult to carve out the time and space to work on something deeper.
Also, I'm not convinced academia is much of a panacea here: I hear there is a nontrivial number of years of pretty severe drudgery before you arrive in the sort of position with a lot of freedom to work on whatever you like.
Of course, in both cases, your work is ultimately directed by revenue, the shield I'm talking about is more against the tight, client/user responsive iterative process of most modern software development which can make it difficult to carve out the time and space to work on something deeper.
Also, I'm not convinced academia is much of a panacea here: I hear there is a nontrivial number of years of pretty severe drudgery before you arrive in the sort of position with a lot of freedom to work on whatever you like.
I also enjoy this, but not for my current employer (for several reasons I don't feel like going into).
How do you do it, as a consultant or as a 9-5 job? Any advice on getting started as someone who graduated 2014?
How do you do it, as a consultant or as a 9-5 job? Any advice on getting started as someone who graduated 2014?
I do it as an employee of a company, but it's a pretty sweet fit for consultants/freelancers (I used to be one).
I don't have any particular advice, beyond the generic stuff: You'll need a network of people who might (know someone who might) want to pay you to help them achieve business outcomes. Remember that you're selling business results, make sure every line on your invoice is part of a narrative that makes (or reduces risk of losing) money for the business.
I don't have any particular advice, beyond the generic stuff: You'll need a network of people who might (know someone who might) want to pay you to help them achieve business outcomes. Remember that you're selling business results, make sure every line on your invoice is part of a narrative that makes (or reduces risk of losing) money for the business.
If you like web-scraping, and you're not an idiot, you immediately have an information advantage for anything you decide to do. Do not underestimate this advantage.
I would suggest you leverage your interest in web-scraping and use it to customers in what ever niche you believe you can sell them something.
Collect your "beta" users through web-scraping, figure out a way to reach them at scale. Build/sell a product they want (you can leverage your information advantage to figure out what this is)
This is personally what I do and have created/worked on many bootstrapped companies over my career. Niche for me is fairly irrelevant, as long as I have a pool of interested customers ahead of time. I primarily use my information advantage to figure these out.
The additional skills you will need is:
* learn to think as a "user/customer"
* minor copy-writing skills (or at least understand what shitty copy is and how to improve it)
* UX
* Data mining & Analytics, Analytics, Analytics
* A/B Testing, Iterations, Incremental Improvements (see point above)
* Hypothesis Driven Development (https://www.thoughtworks.com/insights/blog/how-implement-hyp...)
My one suggestion is to avoid any projects which do not have a clear monetization strategy. If you're following my blue-print companies that don't make money from day will only incur costs as you reach your pool of customers at scale.
I would suggest you leverage your interest in web-scraping and use it to customers in what ever niche you believe you can sell them something.
Collect your "beta" users through web-scraping, figure out a way to reach them at scale. Build/sell a product they want (you can leverage your information advantage to figure out what this is)
This is personally what I do and have created/worked on many bootstrapped companies over my career. Niche for me is fairly irrelevant, as long as I have a pool of interested customers ahead of time. I primarily use my information advantage to figure these out.
The additional skills you will need is:
* learn to think as a "user/customer"
* minor copy-writing skills (or at least understand what shitty copy is and how to improve it)
* UX
* Data mining & Analytics, Analytics, Analytics
* A/B Testing, Iterations, Incremental Improvements (see point above)
* Hypothesis Driven Development (https://www.thoughtworks.com/insights/blog/how-implement-hyp...)
My one suggestion is to avoid any projects which do not have a clear monetization strategy. If you're following my blue-print companies that don't make money from day will only incur costs as you reach your pool of customers at scale.
Can you provide an example or two of a project that involved web scraping? (without divulging client info) I can see how it could be used for price scraping in web retail, but am drawing a blank in other industries.
Address parsing from a large web corpus
https://github.com/openvenues/libpostal
The author uses http://commoncrawl.org/ data, 250TB/3.6b URL and a 100 machine cluster of 8-core machines.
Text analytics, for one.
I've worked 5-6 programming jobs in 5-6 different industries. Honestly the job is pretty similar in any industry. I'd recommend working for whoever pays best and/or a team you get on with.
YMMV. A cognitive load and inherent complexity can vary dramatically between industries. Compare web frontend development with, say, a mission critical CPU design.
You state this as if the answer was obvious... I recently switched back to front end development after months of developing a Bitcoin node from scratch as well as a custom BitTorrent server. I'd say among those three, modern front end is what I find most complex. The amount of technologies, protocols, tools and leaky abstractions to know about is just staggering and by the time you finally caught up, you are already behind.
Of course, if you're doing things the wrong way, you can create any amount of complexity out of nothing. I'm not considering this case. Why would anyone care about the amount of "technologies, protocols, tools and leaky abstractions" when the smart thing to do is to avoid them altogether and just pick the lowest common denominator?
And what would that be for web development? Or are you just saying "don't develop for the web"?
I've seen people who are great at diving deeply into a narrow field struggle greatly with the broad, changing knowledge needed for web dev, especially at scale. It's not clear to me that one is intellectually more challenging than the other. They often seem like different types of intelligence.
I've seen people who are great at diving deeply into a narrow field struggle greatly with the broad, changing knowledge needed for web dev, especially at scale. It's not clear to me that one is intellectually more challenging than the other. They often seem like different types of intelligence.
Same as with any other crappy set of APIs and standards - hide the complexity underneath a thin layer of DSLs.
I do not consider any area where this is possible to do (i.e., any area where complexity is artificial, not inherent) to be really complex and challenging, unlike the domains where complexity cannot be eliminated by simply removing the stupidity.
I do not consider any area where this is possible to do (i.e., any area where complexity is artificial, not inherent) to be really complex and challenging, unlike the domains where complexity cannot be eliminated by simply removing the stupidity.
Are you really saying that the internet and web applications (and all their various components) are artificially complex? Or is it just whatever reification of "web dev" that you have in mind?
Yes, I'm really saying that a majority of this web crap is artificially overcomplicated for no reason at all, and it is mostly a product of stupidity rather than any sane design.
And stupidity got this funny property - it attracts even more stupidity, they stick together, they grow like a snowball, consuming everything around. Ill designed web standards, exceptionally, incomprehensibly shitty language (javascript) - of course it should have created tiers upon tiers of ill designed frameworks, ladders of leaky abstractions and a broken culture of tolerance towards all this shit.
The only sane way to deal with this kind of crap is to shield from it entirely, introduce a simple and clean design which should interact with a broken system, without allowing any of the underlying abstractions to leak to your level. It always works, with all kinds of broken APIs, and I cannot see why web should be any different.
Btw., it's amusing that web devs got so offended by quite an obvious fact that their work is nowhere near in complexity with the hardcore realtime stuff.
And stupidity got this funny property - it attracts even more stupidity, they stick together, they grow like a snowball, consuming everything around. Ill designed web standards, exceptionally, incomprehensibly shitty language (javascript) - of course it should have created tiers upon tiers of ill designed frameworks, ladders of leaky abstractions and a broken culture of tolerance towards all this shit.
The only sane way to deal with this kind of crap is to shield from it entirely, introduce a simple and clean design which should interact with a broken system, without allowing any of the underlying abstractions to leak to your level. It always works, with all kinds of broken APIs, and I cannot see why web should be any different.
Btw., it's amusing that web devs got so offended by quite an obvious fact that their work is nowhere near in complexity with the hardcore realtime stuff.
It's hard to argue your "fact", especially against such an insufferable amount of smugness and causal reductionism.
I find it not amusing that people can assume things they don't participate in are simple, stupid, and trivial. It's the same type of intellectual closemindedness that leads to major problems in human existence. It's a mindset that deliberately ignores complexity in order to justify preexisting belief and prop up belief as truth while protecting the ego.
Personally, I enjoy working on hardware and working on distrubted systems. I also like building things that look beautiful. But to each their own, I guess.
I find it not amusing that people can assume things they don't participate in are simple, stupid, and trivial. It's the same type of intellectual closemindedness that leads to major problems in human existence. It's a mindset that deliberately ignores complexity in order to justify preexisting belief and prop up belief as truth while protecting the ego.
Personally, I enjoy working on hardware and working on distrubted systems. I also like building things that look beautiful. But to each their own, I guess.
> people can assume things they don't participate in are simple, stupid, and trivial
I do not "assume" that web technologies are stupid. I know they're stupid and I can prove they're stupid. Where shall I start? Javascript? All of the W3C issued mess? All the damage left from the browser wars?
This guy said it all much better than I can: http://beauty-of-imagination.blogspot.co.uk/2016/01/tcltk-vs...
When you have something to compare with, something as simple and beautiful as Tk, all the complexity of the modern web looks like a steaming pile of stupidity.
I do not "assume" that web technologies are stupid. I know they're stupid and I can prove they're stupid. Where shall I start? Javascript? All of the W3C issued mess? All the damage left from the browser wars?
This guy said it all much better than I can: http://beauty-of-imagination.blogspot.co.uk/2016/01/tcltk-vs...
When you have something to compare with, something as simple and beautiful as Tk, all the complexity of the modern web looks like a steaming pile of stupidity.
That was an interesting read. I'm not familiar with Tcl/Tk. To help me understand, are you saying that the "web" should remain as purely a hypertext platform and that any application should be built using something like Tcl/Tk? E.g. a program provided by your bank to interact with your accounts? If these are pure applications, how are they discoverable in the way that search engines make the internet so useful?
Do you find anything useful in the compile to js (or even html/css) world like Elm, PureScript, etc.? Do these provide clean wrappers to broken systems?
I certainly find the entire front end world (html/css/js and various compilation of those) painful, and I would definitely enjoy a better paradigm.
Do you find anything useful in the compile to js (or even html/css) world like Elm, PureScript, etc.? Do these provide clean wrappers to broken systems?
I certainly find the entire front end world (html/css/js and various compilation of those) painful, and I would definitely enjoy a better paradigm.
It must be possible to keep the current structure (some form of a DOM plus a scripting language with a full access to it), and still have a dead simple Tk-like event model around it.
It's all about overengineering. Tk demonstrated that all this stuff is fundamentally easy. But the modern web technologies are overengineered to such an extend that anyone touching them draw back in fear.
So, yes, now the only way to put all that complexity under control is exactly this - things that compile to js (+ css + html), that can hide the overengineered parts of the underlying ecosystem and only expose simple, obvious interfaces. PureScript is a right thing.
It's all about overengineering. Tk demonstrated that all this stuff is fundamentally easy. But the modern web technologies are overengineered to such an extend that anyone touching them draw back in fear.
So, yes, now the only way to put all that complexity under control is exactly this - things that compile to js (+ css + html), that can hide the overengineered parts of the underlying ecosystem and only expose simple, obvious interfaces. PureScript is a right thing.
There was an article on HN recently which I believe applies equally well to technical stacks: http://techcrunch.com/2016/01/18/why-big-companies-keep-fail...
Which sources nobel laureate Philip Anderson in the paper "More Is Different":
"The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe...The behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of a simple extrapolation of the properties of a few particles."
https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Diff...
"The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe...The behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of a simple extrapolation of the properties of a few particles."
https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Diff...
Kolmogorov and Chaitin got a different view on this matter.
This is actually what science is all about - finding an information in data, or, in other words, measuring the complexity. If you can reduce a complex set of data to a simple algorithm, then this algorithm is all the knowledge that is there, and the rest is just a noise. By definition.
What Phil Anderson is talking about is a different kind of complexity, which is sort of irrelevant in this context. Yes, knowing the fundamental laws won't help you to explain, say, a protein folding, not without hundreds of years of a CPU time. But it does not matter. It does not make this science more "complex".
This is actually what science is all about - finding an information in data, or, in other words, measuring the complexity. If you can reduce a complex set of data to a simple algorithm, then this algorithm is all the knowledge that is there, and the rest is just a noise. By definition.
What Phil Anderson is talking about is a different kind of complexity, which is sort of irrelevant in this context. Yes, knowing the fundamental laws won't help you to explain, say, a protein folding, not without hundreds of years of a CPU time. But it does not matter. It does not make this science more "complex".
I'm always curious when I see comments like this: have you done both things professionally at scale, or are you guessing based on how things seem to you from the outside?
I have to admit that I did not do any web development, but I have a pretty good idea of what kind of complexity and challenges it does involve. I also know about it from interaction with the frontend developers themselves and from toying around it.
I know what kind of tools are available, I know how dead easy debugging is. There is nothing nearly comparable in complexity with a hardware design.
I know what kind of tools are available, I know how dead easy debugging is. There is nothing nearly comparable in complexity with a hardware design.
It seems like you're being a little biased.
Writing the architecture for a new Intel CPU is much more difficult than writing a calculator on an FPGA or designing a circuit board.
Similarly, designing and maintaining the UX of Amazon.com is a lot more difficult than a personal website.
Is Physics 'harder' than Psychology just because it's 'more hardcore' or 'more pure'?
Apples to oranges. You cannot really make comparisons like that.
Writing the architecture for a new Intel CPU is much more difficult than writing a calculator on an FPGA or designing a circuit board.
Similarly, designing and maintaining the UX of Amazon.com is a lot more difficult than a personal website.
Is Physics 'harder' than Psychology just because it's 'more hardcore' or 'more pure'?
Apples to oranges. You cannot really make comparisons like that.
> designing and maintaining the UX
I did not say a word about design, I was talking about development. UX design is probably among the most complex problems out there, since it is not solved yet and there is even no glimpse of a distant hope.
> Amazon.com
Well, this one was obviously designed by a drunk cobbler, and he could not have spent more than half an hour overall, judging by how miserable the user experience is.
> Is Physics 'harder' than Psychology just because it's 'more hardcore' or 'more pure'?
Physics is solvable, Psychology is not, the latter is in an early phenomenological state of development. So, on this scale, it is harder to apply it (because one have to operate with a much bigger amount of information), but it is much easier in terms of models.
> You cannot really make comparisons like that.
Of course you can. Complexity is a very well-defined and objective thing.
I did not say a word about design, I was talking about development. UX design is probably among the most complex problems out there, since it is not solved yet and there is even no glimpse of a distant hope.
> Amazon.com
Well, this one was obviously designed by a drunk cobbler, and he could not have spent more than half an hour overall, judging by how miserable the user experience is.
> Is Physics 'harder' than Psychology just because it's 'more hardcore' or 'more pure'?
Physics is solvable, Psychology is not, the latter is in an early phenomenological state of development. So, on this scale, it is harder to apply it (because one have to operate with a much bigger amount of information), but it is much easier in terms of models.
> You cannot really make comparisons like that.
Of course you can. Complexity is a very well-defined and objective thing.
Even then I'd say there's not a lot of difference - you naturally adjust your defect rate to what you're working on (even within a single codebase). I mean sure, when doing something like CPU design it's easy to feel like you're not using your full intellectual capacity the way you would in web frontend development, but you just need to step it up a level - the safety requirements justify putting a fair amount of time and effort into techniques like automatically generated testcases, maybe even formal methods.
One thing that can be harder to overcome is being trapped by processes that you're not allowed to change (and the kind of institutions that make CPUs are particularly prone to that kind of failure, IME). But even then, it just becomes a case of adding as much value as you can within those restrictions, and that can be fulfilling in its own way. But yeah, some organizations (in any industry) are really badly managed and if you can't live with that then your only recourse is to leave.
One thing that can be harder to overcome is being trapped by processes that you're not allowed to change (and the kind of institutions that make CPUs are particularly prone to that kind of failure, IME). But even then, it just becomes a case of adding as much value as you can within those restrictions, and that can be fulfilling in its own way. But yeah, some organizations (in any industry) are really badly managed and if you can't live with that then your only recourse is to leave.
Well, formal verification is exactly what I find the most intellectually challenging and enjoyable. There is no way you can apply any advanced math in something like frontend development, and you're very unlikely to be able to do anything anywhere near any formal methods outside of the hardware design.
You absolutely can. PureScript is one of the most interesting things going on in programming at the moment. In web dev the cost of confirming things empirically is orders of magnitude lower than in hardware, but the complexity of what you're working on can eat that margin up surprisingly quickly; on large codebases a bit of formality is well worth it.
And where do you think this "complexity" is coming from? There are ill designed standards - but you can leverage that. There are tons of frameworks - you can ignore them. There are broken protocols - wrap them into something sane.
PureScript is the right thing indeed, I like the approach, but what exactly makes PureScript so special? I quickly skimmed through, expecting something cool and innovative like, say, gradual typing, but it looks like something fairly standard, I could not find any particularly new ideas there. What did I miss?
PureScript is the right thing indeed, I like the approach, but what exactly makes PureScript so special? I quickly skimmed through, expecting something cool and innovative like, say, gradual typing, but it looks like something fairly standard, I could not find any particularly new ideas there. What did I miss?
UI design is very complex, because humans are complex. Often you're automating business processes that work based on implicit judgements that are so obvious to the humans performing them that they're not even aware they're making them, making it very difficult for them to explain the rules they follow.
What I like about PureScript is the careful typeclass hierarchy - it feels like a rethinking of the Haskell standard library from the ground up. They often end up with a hierarchy of single-method typeclasses, which feels like a first step to something along the lines of http://degoes.net/articles/principled-typeclasses/ . I'd like to combine that with Idris-style full dependent typing but you can't have everything and there's a tension between what's state of the art and what's designed for practical use (I feel like I'd be happier using PureScript in production than Idris). Gradual typing I don't find particularly interesting; YMMV.
What I like about PureScript is the careful typeclass hierarchy - it feels like a rethinking of the Haskell standard library from the ground up. They often end up with a hierarchy of single-method typeclasses, which feels like a first step to something along the lines of http://degoes.net/articles/principled-typeclasses/ . I'd like to combine that with Idris-style full dependent typing but you can't have everything and there's a tension between what's state of the art and what's designed for practical use (I feel like I'd be happier using PureScript in production than Idris). Gradual typing I don't find particularly interesting; YMMV.
As I answered elsewhere in this thread, I did not say a word about design. I was talking about programming solely.
Design is not only a complex problem, it is an unsolved problem. Is there any passable design out there? I have not seen it yet. It is either disgusting, incomprehensibly disgusting, or, in the best case, barely tolerable. And the same thing is going on in any kind of an industrial design, it is not limited to web.
As for the type class hierarchy, yes, I did not notice this bit, thanks. I'll take a closer look now.
Design is not only a complex problem, it is an unsolved problem. Is there any passable design out there? I have not seen it yet. It is either disgusting, incomprehensibly disgusting, or, in the best case, barely tolerable. And the same thing is going on in any kind of an industrial design, it is not limited to web.
As for the type class hierarchy, yes, I did not notice this bit, thanks. I'll take a closer look now.
Programming is nothing but design - a full spec of what needs to happen is a program.
Like any other kind of programming, if parts are trivial you automate them, until you're at a state where what you have to do to accomplish a given task is precisely the amount of thought/uniqueness that that task involves; if you're a programmer then you never need be doing mindless drudge work (you've got a computer for that). When you've automated most of what used to be your job, reaching up or down the stack is a good way to find more things to automate (or you can simply increase throughput, but often that just means other parts of the business become the bottleneck). IME it's if anything easier to get closer to the "full brain all the time" state with web programming, because there's so little external latency, whereas in hardware there's only so much you can do before you're spending most of your time waiting for slow test cycles or actual manufacturing (even if you try to reduce the importance of those by introducing things like formal methods, others will often want to see a few test cycles to prove the effectiveness of those).
Like any other kind of programming, if parts are trivial you automate them, until you're at a state where what you have to do to accomplish a given task is precisely the amount of thought/uniqueness that that task involves; if you're a programmer then you never need be doing mindless drudge work (you've got a computer for that). When you've automated most of what used to be your job, reaching up or down the stack is a good way to find more things to automate (or you can simply increase throughput, but often that just means other parts of the business become the bottleneck). IME it's if anything easier to get closer to the "full brain all the time" state with web programming, because there's so little external latency, whereas in hardware there's only so much you can do before you're spending most of your time waiting for slow test cycles or actual manufacturing (even if you try to reduce the importance of those by introducing things like formal methods, others will often want to see a few test cycles to prove the effectiveness of those).
> Programming is nothing but design
No, it's a different kind of a "design". It's an engineering. You've got a set of constraints and need to optimise a solution until all the constraints are met. Purely mechanical.
Designing something for a human interaction is different. It involves extorting information from a not necessarily cooperative and not very sane human, reducing it to a reasonably sane subset and then finding out the workflows that would make sense for a living human being. Unsolvable. Impossible.
Got nothing in common with something as simple as "here's a high level formal description of an algorithm, turn in into a code".
> whereas in hardware there's only so much you can do before you're spending most of your time waiting for slow test cycles or actual manufacturing
In my book, it's a blessing. Time to slow down and reflect on what a mess you've just created.
No, it's a different kind of a "design". It's an engineering. You've got a set of constraints and need to optimise a solution until all the constraints are met. Purely mechanical.
Designing something for a human interaction is different. It involves extorting information from a not necessarily cooperative and not very sane human, reducing it to a reasonably sane subset and then finding out the workflows that would make sense for a living human being. Unsolvable. Impossible.
Got nothing in common with something as simple as "here's a high level formal description of an algorithm, turn in into a code".
> whereas in hardware there's only so much you can do before you're spending most of your time waiting for slow test cycles or actual manufacturing
In my book, it's a blessing. Time to slow down and reflect on what a mess you've just created.
I want to suggest something that I would not normally suggest which is you should pair up with a non-technical founder. What you want to be looking for is someone with deep business experience (15+ years) in a narrow business field. Team up with them and work towards creating something amazing, but tightly focused.
I don't have any specific industry for you to get into, but I have a suggestion on how you might be able to find it. Attend as many trade shows as you can and talk to the folks to get an idea of what they are doing, although some of them may be difficult to get to due distance, cost, invites etc.
For example, Embedded World in Nürnberg, Germany provides a good cross section of industry that has embedded engineering involvement in it. There will be a similar one for telecommunications, Automotive etc.
Good luck on searching for your calling.
For example, Embedded World in Nürnberg, Germany provides a good cross section of industry that has embedded engineering involvement in it. There will be a similar one for telecommunications, Automotive etc.
Good luck on searching for your calling.
IMO your question isn't really answerable: you've listed tools and techniques you enjoy but "sectors" in the industry are primarily about producing results using those. A similar realspace question would be, "I really enjoy building cantilevers and arches. What city should I look for construction work in?"
My personal recommendation would be to find people who are doing interesting things and talking about it in public - blog posts, conference talks, etc etc and reach out to them.
My personal recommendation would be to find people who are doing interesting things and talking about it in public - blog posts, conference talks, etc etc and reach out to them.
Have a look at the legal industry. There are quite a few legal tech startups by now. Even web-scraping applications are rare in legal. The sector is famous for being profitable (high margins), rusty and inefficient.
Having worked at a fintech startup specializing in decision systems for regulatory compliance, I second this. There are interesting theoretical aspects stemming from the fact that domain is very tricky to model.
* There is considerable structural sharing inside finance regulations, but it is not hierarchical
* Regulations may be changed at any time, and these changes may be future-dated or back-dated
* Users insist on arbitrarily complex exceptions and overrides
* Nontechnical users (particularly legal) will insist on being able to vet and introspect
In my experience, anyone can do this badly, but to make it scalable and robust requires subtlety and a good understanding of formal logic, ontology and DSL design. It's a good fit for FP. Oh, and there's plenty of room for machine learning: Automated discovery, natural language processing, noncompliance detection, etc.
The problem of turning regulations into a expert system that is robust against future changes has not been solved, and there's demand for such systems from even the most sophisticated financial companies, even though everyone expects they'd be able to build such systems in-house.
* There is considerable structural sharing inside finance regulations, but it is not hierarchical
* Regulations may be changed at any time, and these changes may be future-dated or back-dated
* Users insist on arbitrarily complex exceptions and overrides
* Nontechnical users (particularly legal) will insist on being able to vet and introspect
In my experience, anyone can do this badly, but to make it scalable and robust requires subtlety and a good understanding of formal logic, ontology and DSL design. It's a good fit for FP. Oh, and there's plenty of room for machine learning: Automated discovery, natural language processing, noncompliance detection, etc.
The problem of turning regulations into a expert system that is robust against future changes has not been solved, and there's demand for such systems from even the most sophisticated financial companies, even though everyone expects they'd be able to build such systems in-house.
That's interesting, may I ask which startup is that? Is it still alive?
Still alive, last I heard!
Droit Financial Technologies http://www.droitfintech.com
Droit Financial Technologies http://www.droitfintech.com
I second that. regulatory compliance needs some better solutions
The technology to cost-effectively store massive amounts of raw metrics / data and quickly derive insight from it is still a relatively recent development, and so far we've seen turn-key tools to leverage that in just a few areas:
1. Where developers themselves are the likely product users (think Google Analytics, New Relic, Optimizely, Appdynamics)
2. Where there is a direct and immediate connection to $$$ (finance / quant stuff)
3. Generic data science platforms (Cloudera, Hortonworks, Databricks)
I think there is still tons of opportunity for industry-specific turn-key data science tools. Especially for verticals where there is a low coincidence of skilled developers with industry domain expertise.
1. Where developers themselves are the likely product users (think Google Analytics, New Relic, Optimizely, Appdynamics)
2. Where there is a direct and immediate connection to $$$ (finance / quant stuff)
3. Generic data science platforms (Cloudera, Hortonworks, Databricks)
I think there is still tons of opportunity for industry-specific turn-key data science tools. Especially for verticals where there is a low coincidence of skilled developers with industry domain expertise.
Learn the art of writing gpu compilers. Software defined radio is another.
If you find web-scraping dull don't pursue it. Life is short.
If you find web-scraping dull don't pursue it. Life is short.
If you are interested in huge bonuses up to 400k-1M a year for being a developer, then I suggest to you high frequency trading development. The problem is you'll have to move close to an exchange, but the finance industry rewards you handsomely and takes care of it's employees.
I'd just suggest looking into something up and coming that interests you, and start learning whatever you can about that.
Industries that will have specific uses for machine-vision: autonomous driving cars, drones, robotics, augmented reality, internet of things
Others: crypto-currencies (specifically digital ledgers/blockchains), virtual reality, commercial space exploration, online education, voice recognition, (more as I think of them)
Tech that's already big and probably getting bigger: streaming video, instant stock trading based on algorithms,
Certainly one of those must be interesting to you. They're all pretty interesting to me.
Industries that will have specific uses for machine-vision: autonomous driving cars, drones, robotics, augmented reality, internet of things
Others: crypto-currencies (specifically digital ledgers/blockchains), virtual reality, commercial space exploration, online education, voice recognition, (more as I think of them)
Tech that's already big and probably getting bigger: streaming video, instant stock trading based on algorithms,
Certainly one of those must be interesting to you. They're all pretty interesting to me.
you should look at marketplaces. possibly even ones run by crypto currencies. There are alot of ways to configure a marketplace and it would be nice to abstract that from CRUD operations and such.
[deleted]
Whereabouts are you? If you're interested in PLT, DSLs and compilers, there are some small enclaves in finance that might interest you.
edit spelling
edit spelling
What small enclaves in finance are you talking about? I know that banks have small scripting languages that enable semi-technical users to describe a derivative product ("if condition1 then payoff1 etc.") but those are old and I doubt they are actively maintained and that banks are hiring people for that.
I was talking about banks. I work for one where we build stuff on top of essentially this paper:
http://research.microsoft.com/en-us/um/people/simonpj/Papers...
We're even hiring: https://donsbot.wordpress.com/2016/01/18/haskell-developer-r...
http://research.microsoft.com/en-us/um/people/simonpj/Papers...
We're even hiring: https://donsbot.wordpress.com/2016/01/18/haskell-developer-r...
Best of luck to you, I am not in the industry and cannot recommend from that perspective.
As a consumer though, augmented reality ( eg Hololens and Magic Leap) seems like it is about to become very big, and probably utilizes machine vision.
Can you recommend some resources for learning about web scraping?
As a consumer though, augmented reality ( eg Hololens and Magic Leap) seems like it is about to become very big, and probably utilizes machine vision.
Can you recommend some resources for learning about web scraping?
you're right, web scraping is dull ( it was a big, but important part of my job for a number of years ). The interesting, and non-trivial, part comes on how to use the data you have scraped.
[deleted]
I'm very much into DSLs, and I've been building and using them extensively throughout a very wide spectrum: database engines, CADs, hardware design and verification, compilers, device drivers, HPC (GPGPU).
Which one are you talking about:
Damn Small Linux, a very small Linux distribution
Definitive software library
Domain-specific language, a computer language designed for a specific problem domain
Damn Small Linux, a very small Linux distribution
Definitive software library
Domain-specific language, a computer language designed for a specific problem domain
DSL= Domain Specific Languages
sorry I left the non-technical terms in there I didn't mean to piss everybody off. Thanks for answering.
My interests within the field is not what one would call narrow. Primarily though, I have a fondness for programming language theory, compilers, parsing and writing my own DSLs so that I can make powerful compile-time assertions regarding my code as well as generate tests automatically from the DSL itself.
My StackOverflow cv has an additional set of tasks that pertain to my interests, http://stackoverflow.com/cv/filipallberg.
Regarding my primary interests, the only relevant "thing" I have come up with is web-scraping, but I find it is a dull task as it is a trivial matter to me.
Machine vision is also something that I think a lot about, however most of my ideas are consumer-facing utilities and not something to build a business around. If I get a good idea I'd be willing to go into business myself.