> When VLMs make errors, they don't make random mistakes. Instead, 75.70% of all errors are "bias-aligned" - meaning they give the expected answer based on prior knowledge rather than what they actually see in the image.
Yeah, that's exactly what our paper said 5 years ago!
Datasets need to stop shipping with any training sets at all! And they should forbid anyone from using the test set to update the parameters of any model through their license.
We did this with ObjectNet (https://objectnet.dev/) years ago. It's only a test set, no training set provided at all. Back then it was very controversial and we were given a hard time for it initially. Now it's more accepted. Time to make this idea mainstream.
No more training sets. Everything should be out of domain.
I'm the lead author of the paper you cited. Glad you enjoyed our work :)
Sure, you can break systems. That doesn't mean that they aren't useful! In many cases a system will see the same boring input many times over. People are often willing to be a bit flexible and help out when it happens to misread something. The fact that you can intentionally break systems like that, and that you can break them in a particular direction, like making them always think there's no danger in an image, is really worrisome.
Our work shows that your autonomous car won't always work well; that its vision system has some systematic error which we can characterize now. Adversarial attacks show that someone can intentionally make your car see a lane, whenever and wherever they feel like it, and drive you off the road. It's a whole different ballgame, and the language of attack and defense really fits well.
> merely a student or professor, but can get a job at or create their own a startup or big company or meaningful open source project of their own.
Hey! Professors do meaningful things too :)
> The real downfall of icfp is that in the past 10 years, anyone talented enough to participate
I don't think the optimization is the biggest problem. If you look at it, Endo in 2007 was "optimization", you had to find the shortest prefix to save our alien friend. But the problem had more meat to it and it was different from the problems that came before.
People are no different today than 10-20 years ago, and more good young people are created all the time who can participate. If anything, there are way more capable people that can participate today than 20 years ago.
The problems are now all the same. Why would you waste your time doing this year's problem when the problem from last year is nearly identical but with a different two-paragraph lead-in story.
Regardless of our diagnosis, the numbers bear out that the ICFP contest is slowly dying. The number of teams submitting from 2014 to 2018: 275, 230, 203, 120, 107.
We have a few more years before it's scrapped at this rate.
Years ago we had a lot of good problems. Not just 2006, but 2007 was great too where you had secret messages to save Endo. 2008 at least had you doing some fun things with the rover. 2014 you programmed a fun machine. One of the years was an optimization problem, I forgot which, but it involved orbital mechanics, so it was a lot deeper than the current setup.
In the past 5 or so years the contest has settled into a pretty boring rut. The problems are all the same. "We came up with a system that has an agent. It can do 5-15 things. Get it to solve this simple to define problem efficiently."
It's not that these problems aren't fun, it's that they're all the same :(
What's the point in doing the 10th contest in exactly the same format? It's gotten to the point where I can just reuse code from previous years.
HIPAA doesn't talk about US citizens or distinguish different types of records based on any properties of the people that those records cover. The words citizen do not appear in HIPAA or HITECH. HIPAA applies to any records by covered entities, which is what it discusses, regardless of who those records refer to.
There is no requirement in HIPAA that PII must be stored in the US. This is such basic info it's in the HIPAA FAQ from HHS https://www.hhs.gov/hipaa/for-professionals/special-topics/c.... Question 9 is unequivocal, you can store data outside of the US, but you need to think about any dangers or risks associated with this. Which is totally logical.
There are lots of reasons to have issues with the US. But not what you're talking about.
If you use lens as just a way to access records like you do in other languages, then there is absolutely nothing hard about it. Literally all you need to know is:
Name your records like "data Prefix = Prefix { prefixFieldName :: ... }" call "makeFields ''Prefix" once at the bottom of your file and use "obj ^. fieldName" to access and "obj & fieldName .~ value" to set.
That's it. You now have 100% of the capabilities of record update in any other language. This doesn't get any simpler in any other language. It even pretty much looks like what you would do in other languages.
I'll grant you, Haskell and lens do a terrible job of explaining subsets of functionality that are simple and let you get the job done before jumping in the deep end.
There's a control group. Both groups were watched by the experimenters. Not only that, the same teachers had students in both groups in different classes.
This is a huge problem throughout science, not just ML. As scientists, we're rewarded for publishing cool new things that work, not for pointing out things that don't or for pointing out flaws in existing papers. If the point is to get people to not read one bad paper, it's just a waste of my time. Most papers are false and a lot of them should never have passed review.
If the authors actually wanted to do good ML research, they could always have reached out to a decent ML researcher who could have told them all of this. There's no shortage of us. The journal could have reached out to an ML reviewer. Why wouldn't they? But no one did, because the results look good and so they send it off to press and it's good for both the authors and the journal to have something that is hype-worthy. It's just the sad reality of modern science.
The imbalance is totally artificial and objectionable though. Where's the evidence that doctors see a 80/20 split in real life? If there is going to be an imbalance they should make it reflect the actual statistics of the task that the doctors perform not some artificial number. It doesn't even reflect the statistics of the dataset they started with (which is 90/10 unblanaced).
Admittedly, the correct analysis for when the data is unbalanced is more annoying and ROC curves are easier to interpret. That's why in something like ImageNet even though the training set is imbalanced, the test set is is balanced.
Comparisons against humans are also harder when the data is imbalanced in a way that reflects the training set, not the task. Humans don't know they are supposed to say "no" 80% of the time. That rewards the machine and that isn't easy to correct (you can correct what you think about the machine results with respect to a baseline, but not what biases the humans had).
I do research in computer vision and this paper is so bad it's beyond words.
* They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis.
* They measure the wrong things that reward the network. Because the dataset is imbalanced you can't use an ROC curve, sensitivity, or specificity. You need to use precision and recall and make a PR curve. This is machine learning and stats 101.
* They measure the wrong thing about humans. What a doctor does is they decide how confident they are and then they refer you to a biopsy. They don't eyeball it and go "looks fine" or "it's bad". They should measure how often this leads to a referral, and they'll see totally different results. There's a long history in papers like this of defining a bad task and then saying that humans can't do it.
* They have a biased sample of doctors that is highly skewed toward people with no experience. Look at figure 1. A lot of those doctors have about as much experience to detect melanoma as you do. They just don't do this task.
* "Electronic questionnaire"s are a junk way of gathering data for this task. Doctors are busy. What tells the authors that they're going to be as careful for this task as with a real patient? Real patients also have histories, etc.
I could go on. The number of problems with this paper is just interminable (54% of their images were non-cancer because a bunch of people looked at them. If people are so wrong, why are they trusting these images? I would only trust biopsies).
This isn't coming to a doctor's office anywhere near you. It's just a publicity stunt by clueless people. Please collaborate with some ML folks before publishing work like this! There are so many of us!
That's a common misconception. She did not do jail time for insider trading. She almost certainly would have gotten away with it. The problem is she lied to a federal agent while trying to hide her inside trading. That's what they charged and convicted her with.
In terms of immigrants as a percentage of population the US is nothing special. Canada has a lot more (closer to 19% instead of 14%). Many European countries have statistics similar to that of the US (Sweden has a bit more, the UK a bit less, etc).
But I digress. The problem with the existing system is that it is insane and a waste of resources. One has to get a lawyer and wait for a long time with no certainty in the outcome while dealing with an extremely opaque bureaucracy where anything can go wrong at any time.
This has nothing at all to do with volume. Indeed, a simpler more streamlined and less opaque system would help with volume.
Say a system like the Canadian or the UK one. There's a point scale. You can compute the number of points you get ahead of time, now there's a simple web form actually. If you cross the threshold you will get in. There are known wait times, you can just call the embassy. There are points for things the country needs (particular jobs), for certain qualifications (degrees, etc), for language proficiency, some regional tweaks, family, and having an employment offer.
And extremely importantly. You get permanent resident status (green card), not an H1B. What Americans don't realize is that you want to hand out green cards. H1B lower both your salaries and ours. The H1B restricts immigrant mobility, can't move to a better job and raise the average salary, and encourages people to go home with all of their newly gained knowledge and money.
I'll give you a personal example of the difference between a sane and an insane system:
As an 11 year old I filled out all of our Canadian immigration paperwork (my parents checked it but it was correct and they didn't change it). We knew we would get in based on our points. The embassy told us the timeframe in which we should expect our paperwork to go through. It went through a bit early. We moved to Canada.
Now for the US. As a near-30-year old with a US PhD working at a top research institution I have to pay a specialized law firm several thousand dollars, spend weeks getting paperwork, bugging people in several countries to write absurd letters, building a case, etc. All of this to basically the same thing. And in the end, who knows what will happen because there are no standards, no appeal, no one to discuss anything with. Oh, and I have no idea what the processing time is.
So no. It is not an issue of "why can't they just let me in". It's a system that hurts your salary by restricting my mobility, hurts me by making me pay lawyers needlessly, hurts the image of the US by creating disgruntled people, and hurts the economy by routing business and increasingly prestigious conferences elsewhere. It just makes no sense.
Yeah, that's exactly what our paper said 5 years ago!
They didn't even cite us :(
"Measuring Social Biases in Grounded Vision and Language Embeddings" https://arxiv.org/pdf/2002.08911