Hah, I literally just fought this for the past month. We run a large esports league that relies on player ranked data. They have the data, and as mentioned above, they send it down to the browser in beautiful JSON objects.
But they're sitting behind Cloudflare and aggressively blocking attempts to fetch data programmatically, which is a huge problem for us with 6000+ players worth of data to fetch multiple times every 3 months.
So... I built a Chrome Extension to grab the data at a speed that is usually under their detection rate. Basically created a distributed scraper and passed it out to as many people in the league as I could.
For big jobs when we want to do giant batches, it was a simple matter of doing the pulls and when we start getting 429 errors (rate limit blocking code they use), switch to a new IP on the VPN.
The only way they can block us now is if they stop having a website.
Give one of the commercial VPN providers a try. They're usually pretty cheap and have tons of IPs all over the place. Adding a "VPN Disconnect / Reconnect" step to the process only added about 10 seconds per request every so often.
Holy Moly! The author of this paper makes a very scary argument further shifting the burden of proof onto the employers.
The problem with hiring is that there is a hugely disproportionate supply/demand ratio. We might have 1 position to fill and 100+ applicants. How can any company correctly protect themselves against allegations of illegal hiring criteria in this type of environment? Right now the best protection is to say "we hired the most qualified candidate", but even that argument is shaky. What if the most "technically qualified" candidate performed the worst in the interview process?
Hiring is scary enough already without bringing the law into the picture. :/
I've noticed the same thing and have been actively working over the past 5 years trying to make introductory robotics approachable.
I've been a software developer for a long time, but robotics always felt like a mystery to me until I had someone show me the basics. I recently started a small company trying to teach the basics of robotics. The biggest challenge is that robotics have a real price attached that normal software development doesn't have associated.
With Javascript, I just need a text editor and a browser, but with Robotics I have to make the leap and purchase a kit and trust that the kit will do enough to teach me what I need to know to get started.
We designed and built a low-cost kit and have been using it in middle-schools throughout the southeast teaching kids the introduction, and so far that's been working really well.
Unfortunately it seems like it's really hit or miss depending on which worker handles the order.
Some days I get exactly what I ask for really quickly, when other days I'll have my shopper switching 25% of my requested items with alternates (some of which are wildly off base).
My biggest frustration is that it seems like the shoppers aren't intimately familiar with my local grocery stores. I'll request an item that I know exists, they'll swap it out for something completely different and not what I want, yet if I personally go to the store, I'll easily find the item I requested.
One cool approach to a solution would be to use data to save the day. Millions of people fly through the carriers each year. The airlines could analyze the statistical heights of their average airline passenger and design the seats accordingly. Then assign seats on the flight based on passenger height. You could have a percentage of your seats configured for short people, a bigger percentage configured for average height, and several set up for tall outliers. Then everyone will be statistically more likely to get a comfortable experience.
Twitter has caused serious harm to myself and my family. I'm a big fan of the platform, but I've been "doxxed", harassed, and was even SWATed once by different people trying to get me to give them my username.
I'd release it, except that'd be almost as big of a pain. I'd have to release a new edition of my book, change all my websites. Ugh. People are dicks. I wish I'd picked a less desirable handle.
We used them to design and build a custom board for our Sumo robots that had our V1 built from an Arduino board and a custom board that plugged into the Arduino. We couldn't really afford to pay them a big chunk up front, but we negotiated a pretty pleasant royalty deal that comes out to a few dollars per board (which we get manufactured for less than $10/board).
They did excellent work and I highly recommend them for that type of "prototype -> production" path.
Doesn't this use case eventually screw over any future Twilio customer that selects that number? I see several inbound calls to my Twilio number from random numbers that I have to pay for, even though I'm only using Twilio in a development capacity right now. Do you guys "expire" numbers from your pool that receive aggressive amounts of Inbound traffic (voice, fax, or SMS)?
At some point a few years back, my VW dealership sold my phone number to people for vehicle service contracts. Even though I sold that car 4 years ago, I STILL get at least 2-3 calls per week from different companies offering to sell me an extended warranty. How does Twilio mitigate that for us?
(Off topic, but seriously, thanks for Twilio. Ping me if you want to know how we're using it in our new startup [email in profile]. It's pretty rad.)
Anecdote of one: I was diagnosed with Stage 4 Hodgkins Lymphoma two years ago. My _only_ symptom at the time was pretty severe back pain, which is unusual for lymphoma.
After treatment and remission a year later, I still had the severe back pain. The cancer pain management doctor I was seeing was perfectly happy to keep prescribing godlike levels of opiods. (50+ mg of oxycodone on top of 30+ mg of slow-release morphine pills).
After a few months, I lamented that I was tired of taking pills and dealing with all the side effects. He offered to send me to Physical Therapy, but explained that most of his patients preferred to stay on the pills and that PT rarely worked.
After 3 weeks of twice-weekly 30 minute PT sessions, my back felt amazing. No more pain and I was able to quickly wean off the drugs.
At some point, my cancer had caused back pain, which I'd continued to worsen by avoiding exercise and sitting at weird angles to alleviate the pressure.
As a humorous aside, I had to pay for several weeks of PT out of pocket because my insurance did not consider it important. They would prefer to shell out $500+ dollars every month to see a pain management doctor and continue taking pills.
Dude, you absolutely should be trying to be a manager in 3 years, if you want to go down the management track.
Depending on company size, you should probably be a team-lead or something by then.
Junior developers that remain "solo" workers after 3+ years in the industry is a bad sign for value. If you've been developing for a while, at the very least you should be put in a leadership position from a project perspective. Management/leadership experience doesn't just pop up out of nowhere. It's a skill you have to work really, really hard at. Take every leadership opportunity you can grab. You don't have to be "the boss" in the formal sense of the word, but you absolutely should be working towards a leadership role.
I think you may be missing the brilliance of this product from a sales perspective.
Even if this product isn't a massive success, the units they do move will have a huge potential impact to their retail clothing lines.
Now, instead of suggesting clothing items in the store based on other items you've looked at, they can now suggest clothing items in their sales material based on what you already own and look good wearing.
If I had this product and Amazon started sending me curated outfits that they knew would look great on me and matched my style, I'd probably start using Amazon exclusively to fill out my wardrobe.
As it stands now, I buy almost everything EXCEPT clothes through Amazon and prefer to let my wife buy me things she wants me to wear.
This product, on it's own, is a game changer in the fashion industry. Now Amazon has a compelling and unique product that is going to make it increasingly difficult for other retail outlets to compete in the same space.
And once we start seeing "apps" for this camera that auto-instagram/snapchat your pictures, I think we're going to see a huge demand in the teen/millennial female market for what is kind of a geeky niche product.
I think it's your third paragraph here. You claim not to have insulted anyone, but you literally claimed that GitLab employees are worse than Hitler.
That's:
A) Super inflammatory
B) Extremely disrespectful
C) Adds nothing of value to the discussion
You're absolutely allowed to have a difference of opinion. Even saying something like, "I really hate the GitLab interface and it feels super janky" would have probably gone through this unscathed.
Don't look for jobs when you're desperate. Look for jobs while you're happy and comfortable in your current position.
It's like investing. Don't buy while the market is going up. Buy after a crash. Buy when the deck is stacked in your favor.
I (mostly) only ever actively look for positions when I'm comfortable in my current position. If a new employer really wants me, they'll bend over backwards to get me. I recommend everyone do the same.
Sometimes you may not have that leverage. In those situations, absolutely take whatever job you have to take to keep your house and keep the electricity flowing. But don't stop looking for a better job.
You are literally the only person on the planet that is going to look out for your best interests. The difference beteween a 100,000 and a 110,000 salary over 30 years is freaking $300,000 (not even counting for investment potential). Maximize your career for yourself.
Either they're bluffing completely or you're extremely junior.
We often spend $10k-$30k on each new hire. If you make it through that pipeline and they try to strong-arm you on the first day... they're going to have a rough time.
I had to trick my brain into just enjoying it as a strange engineering discipline. I like to go through life imagining everything as a system to be programmed, so to me, business politics was just a different type of system in a programming language I had yet to learn.
Now I like to think that I can bend those systems to my will as well. :)
This is just my perspective. I've been in (increasingly) challenging engineering leadership roles over the past 5 or so years.
When I first started, I felt exactly like you do. That my job was to enable and empower my engineering team to build awesome product and that all the "management" stuff was a huge waste of time.
I received some advice from my boss that I was missing 50% of the job. You're being paid and expected to do the engineering leadership side on top of being the interface with the company for progress and business objectives. Any talented engineer with a modicum of people skills can usually do a halfway decent job of leading a small team to build a product, but it takes some business acumen to understand how your team fits into the bigger operating environment of your business.
That was the skill they expected you to bring to the table, and you not only despised it, you blew up at them in a meeting.
Granted, that company may very well have had a terrible meeting process and been looking at all the wrong KPIs for how things should work. But knowing how to positively influence that process inside the environment and doing so in a political way that preserves culture and existing team dynamics is crucial.
I hope you give that type of role another shot. The world needs better engineering leaders that can help businesses understand the value of a strong engineering culture.
ISC Consulting Group | Software Test Engineer | Augusta, GA | REMOTE, SECRET CLEARANCE REQUIRED
We're looking for a full-time software test engineer for a major military contract. Any type of software engineer will do, but this position is going to involve writing and building automated tests for our existing software and system development teams. This individual will officially be an engineering member of the QA department and will be lightly disassociated from the actual engineering team.
This individual must have experience with writing automated tests (functional, regression, etc) and experience with any of the AWS stack is a plus.
Unfortunately, we absolutely must have someone with either an active permanent or interim SECRET clearance.
If you're interested, email (me at ericharrison dot info) with your resume and I'll pass it along to my QA lead. Thanks. :D
But they're sitting behind Cloudflare and aggressively blocking attempts to fetch data programmatically, which is a huge problem for us with 6000+ players worth of data to fetch multiple times every 3 months.
So... I built a Chrome Extension to grab the data at a speed that is usually under their detection rate. Basically created a distributed scraper and passed it out to as many people in the league as I could.
For big jobs when we want to do giant batches, it was a simple matter of doing the pulls and when we start getting 429 errors (rate limit blocking code they use), switch to a new IP on the VPN.
The only way they can block us now is if they stop having a website.
Give one of the commercial VPN providers a try. They're usually pretty cheap and have tons of IPs all over the place. Adding a "VPN Disconnect / Reconnect" step to the process only added about 10 seconds per request every so often.