Tell HN: TechTestNow, an app to help you hire the best coders
8 comments
I do like the idea of compiler driven tests over Q&A, but you still fall in the same trap of only getting answers for a very contrived case. There are more dynamics to a good developer than just a few hypothetical situations.
For example someone could be great at Hibernate but not really have some a lot of work with pure JPA. Just because they are not familiar with the JPA does not mean that they have not mastered ORM concepts and would be able to do JPA development as good as the rest of us in a week or so.
Which brings me to my next point. I would throw time completely out. With the situation I just framed a senior level developer is going to do what all of us do. Fire up the browser look at the JPA api and implement it. This is what makes a good developer. Someone who can gather code together and create a solution.
On almost any given project there are going to be new API's that developers will have to familiarize themselves with. The ones that can are far more valuable developers, because they disseminate that information back to the rest of the team.
Your test does not account for that and by making it time based you are disadvantaging the very ones that posses that valuable skill. If the answer is right, it's right.
As well you should consider offering the service to recruiter and hiring line managers, not as a competitive test, but rather as a filtering tool. Competitions are regarded in academia, but in the corporate world people just want to know if you can do the job.
For example someone could be great at Hibernate but not really have some a lot of work with pure JPA. Just because they are not familiar with the JPA does not mean that they have not mastered ORM concepts and would be able to do JPA development as good as the rest of us in a week or so.
Which brings me to my next point. I would throw time completely out. With the situation I just framed a senior level developer is going to do what all of us do. Fire up the browser look at the JPA api and implement it. This is what makes a good developer. Someone who can gather code together and create a solution.
On almost any given project there are going to be new API's that developers will have to familiarize themselves with. The ones that can are far more valuable developers, because they disseminate that information back to the rest of the team.
Your test does not account for that and by making it time based you are disadvantaging the very ones that posses that valuable skill. If the answer is right, it's right.
As well you should consider offering the service to recruiter and hiring line managers, not as a competitive test, but rather as a filtering tool. Competitions are regarded in academia, but in the corporate world people just want to know if you can do the job.
What you say is quite true and resonates with my personal experiences.
I myself do not possess an encyclopedic knowledge, nor will you see my fingers flying over the keyboard.
On the other hand, what I have noticed is that my understanding of the basics of coding, i.e. how to mix and match loops, arrays, conditionals, etc has vastly improved over time.
I frequently forget advanced or language specific topics - and I don't really care, since I can easily re-read about them online and refresh myself.
I have worked as an examiner for computer science university students and also as an interviewer for new hires for companies. What I have noticed is that the best candidates are similar - they have a rock solid grasp of the fundamentals of the language, and just twist and turn that knowledge to acquire the results they need.
You yourself have mentioned this - a good developer with an understanding of ORM can easily adapt to whichever framework he is required to use.
This mastery of the basics is what we are trying to assess with TechTestNow. You might notice that none of the questions require in-depth knowledge of the APIs (although that knowledge might help you solve some of them faster, there is always a solution that uses only basic constructs).
The main reason the tests are timed is because, all said and done, a good programmer does need to have a certain coding speed !
Also, in real life, a lot of time is spent not on figuring out how to solve a problem, but rather on figuring out what the problem is in the first place - what do we really need to do ? (and can it be done at all ?). Although the problems given in TechTestNow might seem contrived, they do have the advantage of being unambiguous, so that the programmer can just concentrate on writing a solution.
I myself do not possess an encyclopedic knowledge, nor will you see my fingers flying over the keyboard.
On the other hand, what I have noticed is that my understanding of the basics of coding, i.e. how to mix and match loops, arrays, conditionals, etc has vastly improved over time.
I frequently forget advanced or language specific topics - and I don't really care, since I can easily re-read about them online and refresh myself.
I have worked as an examiner for computer science university students and also as an interviewer for new hires for companies. What I have noticed is that the best candidates are similar - they have a rock solid grasp of the fundamentals of the language, and just twist and turn that knowledge to acquire the results they need.
You yourself have mentioned this - a good developer with an understanding of ORM can easily adapt to whichever framework he is required to use.
This mastery of the basics is what we are trying to assess with TechTestNow. You might notice that none of the questions require in-depth knowledge of the APIs (although that knowledge might help you solve some of them faster, there is always a solution that uses only basic constructs).
The main reason the tests are timed is because, all said and done, a good programmer does need to have a certain coding speed !
Also, in real life, a lot of time is spent not on figuring out how to solve a problem, but rather on figuring out what the problem is in the first place - what do we really need to do ? (and can it be done at all ?). Although the problems given in TechTestNow might seem contrived, they do have the advantage of being unambiguous, so that the programmer can just concentrate on writing a solution.
Your method of grading the tests fails. I couldn't determine if you were counting number of times compiled against the person, or just looking for something VERY specific in the final result, but both methods failed to properly categorize technically right solutions, even if they didn't meet what was entered as "correct". Even worse, there's no interface to determine what solution you were even looking for, requiring the test requester to either trust your (unique) judgements on correctness of answers or review the playback of every entry, removing the major benefits of the site. The fact that every person gets random questions also doesn't help compare capabilities of different testers. And the final thing missing, the ability to see the requirements + final code submitted without going through the playback.
Note: This isn't a "wah, I did it all right and it said I was wrong!", I specifically used correct, valid answers that were unlikely to match the model you were looking for (empty for loop body for decrementing a variable to a certain value, etc.).
Note: This isn't a "wah, I did it all right and it said I was wrong!", I specifically used correct, valid answers that were unlikely to match the model you were looking for (empty for loop body for decrementing a variable to a certain value, etc.).
I went through your solutions and I must disagree. Your solutions did not generate the correct output for the questions.
For example, I'll take the following question :
'Given the string variable 's' and the integer 'n', return a new string consisting of n repetitions of s.
For example, if s = "abc" and n = 3, you should return "abcabcabc".'
The code you have written is as follows :
String duplicate(String s, int n) {
If we analyze the following code (assuming that s = "abc" and n = 3), initially, the variable 'result' is assigned the value "abc".
The loop then runs 3 times - when n = 3, when n = 2 and when n = 1. Which effectively means that "abc" is concatenated 3 more times giving the string "abcabcabcabc", which is incorrect.
I do agree that the solution is almost right. Note that you are allowed to compile your code and run it against our testdata (and even your own test cases) in order to establish it's correctness - which means that you did have the opportunity to find out that there was a bug in your code and fix it.
But in computing, how often is it acceptable to get things 'almost' right ? Would you be happy if Google returned search results which were almost right ?
I know that if a human conducted the interview, he would have helped guide you through the test - in fact, we are almost done adding the capability for recruiters to monitor test sessions in real time and converse with candidates.
I can also provide a case by case analysis of your other solutions if you wish.
For example, I'll take the following question :
'Given the string variable 's' and the integer 'n', return a new string consisting of n repetitions of s.
For example, if s = "abc" and n = 3, you should return "abcabcabc".'
The code you have written is as follows :
String duplicate(String s, int n) {
String result = s;
// First assignment already done
while (n > 0) {
result += s;
n--;
}
return result;If we analyze the following code (assuming that s = "abc" and n = 3), initially, the variable 'result' is assigned the value "abc".
The loop then runs 3 times - when n = 3, when n = 2 and when n = 1. Which effectively means that "abc" is concatenated 3 more times giving the string "abcabcabcabc", which is incorrect.
I do agree that the solution is almost right. Note that you are allowed to compile your code and run it against our testdata (and even your own test cases) in order to establish it's correctness - which means that you did have the opportunity to find out that there was a bug in your code and fix it.
But in computing, how often is it acceptable to get things 'almost' right ? Would you be happy if Google returned search results which were almost right ?
I know that if a human conducted the interview, he would have helped guide you through the test - in fact, we are almost done adding the capability for recruiters to monitor test sessions in real time and converse with candidates.
I can also provide a case by case analysis of your other solutions if you wish.
question 2 of 6 for duck:
Write code to trap these exceptions and return 1 if a FineException is thrown and return 0 if a CoarseException is thrown.
Others indeed do have some errors as I quickly typed them up (off by 1, forgetting that java array length isn't the same as C++ std::vector.size). Number 3 could definitely use a better example, such as a = 12 (so the result would be {12, 11, 10}.
I admit, I checked number 2 closely, saw a horribly invalid failure reported and only glanced over the rest. Independent of the other failures, you cannot have incorrectly graded questions, and the rest of the points still stand, the requester has nothing to compare against, leaving the judgement up to your pass/fail (which is clearly not 100%) and the random assignment of questions to different test takers results in no real comparison ability between skills.
int exceptionTrap(int a) {
try {
exceptionThrower(a);
} catch (FineException fe) {
return 1;
} catch (CoarseException ce) {
return 0;
}
return -1;
}
Others indeed do have some errors as I quickly typed them up (off by 1, forgetting that java array length isn't the same as C++ std::vector.size). Number 3 could definitely use a better example, such as a = 12 (so the result would be {12, 11, 10}.
I admit, I checked number 2 closely, saw a horribly invalid failure reported and only glanced over the rest. Independent of the other failures, you cannot have incorrectly graded questions, and the rest of the points still stand, the requester has nothing to compare against, leaving the judgement up to your pass/fail (which is clearly not 100%) and the random assignment of questions to different test takers results in no real comparison ability between skills.
You're right about question 2, my apologies. There indeed was a bug in the test data for that question (which we have fixed now). However, the rest of the questions given in the test work properly.
The reason different test takers get (somewhat) different question sets is because our algorithm alters the difficulty of the questions based on the test taker's performance. I.e. if you did the first few questions well, you'd start getting progressively tougher questions. On the other hand, if you do badly, you'll be given successively easier questions.
The reason different test takers get (somewhat) different question sets is because our algorithm alters the difficulty of the questions based on the test taker's performance. I.e. if you did the first few questions well, you'd start getting progressively tougher questions. On the other hand, if you do badly, you'll be given successively easier questions.
[deleted]
Wow the recorded sessions feature is really cool. Props to you guys, I hope you do well.
Where are the questions? The UI is not very intuitive. Good luck anyway!
I think that the confusion is because you attempted to take the test using the login credentials for the recruiter.
Recruiters cannot take tests - only candidates can. Note that you can easily create a new account at the 'take test' page (please use one of your email addresses).
Recruiters cannot take tests - only candidates can. Note that you can easily create a new account at the 'take test' page (please use one of your email addresses).
Recruiters should be able to see the questions and answers.
go through one of the test link and create an account. it'll take you to the test. I gave the recruiter login which was used to create those tests because it's easier to check it out.
What is the difference between this and http://codility.com/?
TTN has quite a few more features. The key ones would be :
1. Viewing a playback of how the candidate coded the question.
Even a bad candidate might be able to 'prettify' his final code (or even craftily cut and paste code while solving the problem).
However, it is almost impossible to do so in a playback of the entire coding process.
It is like the difference in complexity between creating a fake photo vs creating a fake video !
2. Adaptive testing
Depending on the performance of the candidate, the questions presented will increase or decrease in difficulty - allowing the system to accurately assess the candidate's skill level (that is the advantage of using several small questions, as opposed to a few large questions).
3. The ability to embed the testing widget both on your website or even on a job posting on a job site - that way candidates never even have to leave your website.
1. Viewing a playback of how the candidate coded the question.
Even a bad candidate might be able to 'prettify' his final code (or even craftily cut and paste code while solving the problem).
However, it is almost impossible to do so in a playback of the entire coding process.
It is like the difference in complexity between creating a fake photo vs creating a fake video !
2. Adaptive testing
Depending on the performance of the candidate, the questions presented will increase or decrease in difficulty - allowing the system to accurately assess the candidate's skill level (that is the advantage of using several small questions, as opposed to a few large questions).
3. The ability to embed the testing widget both on your website or even on a job posting on a job site - that way candidates never even have to leave your website.
Number 1 is actually a downside, to me. I don't really care whether a coder starts with ugly code (don't we all?) but I care what it looks like when he hits "submit". Also, do the rules specifically disallow writing the code in my IDE and then copy/pasting it in? I think that would be a silly rule. What about a good developer who writes a test first, and of course, codes against that test.
However, number 2 and 3 are nice.
However, number 2 and 3 are nice.
Both of the points you make are quite valid.
As you said, it is the final code which is more important. But don't you think that it is also important to view the process by which the programmer arrived at the final code ?
An experienced coder will quickly reach a solution. Although he might make several mistakes on the way, he can usually quickly detect and correct them.
Someone less experienced would take longer, make more mistakes and probably take several attempts to correct those mistakes before getting it right.
Even though the final code might be more or less the same, the playback would help you discern who has better coding skills.
Of course, this would mainly be of use when attempting to choose between two closely matched coders.
Regarding copying and pasting - at the moment the rules don't explicitly ban this. The problem with allowing it is that the likelihood that the coder might attempt to cheat (and get away with it) is vastly increased (and cutting and pasting makes the playback feature worthless !).
That is why we went to the trouble of enabling syntax highlighting, compilation and testing within the widget, so that it would be as close to an IDE as possible.
BTW - regarding testing - aside from the 2 default test cases, it is also possible to enter your own testdata in the widget.
As you said, it is the final code which is more important. But don't you think that it is also important to view the process by which the programmer arrived at the final code ?
An experienced coder will quickly reach a solution. Although he might make several mistakes on the way, he can usually quickly detect and correct them.
Someone less experienced would take longer, make more mistakes and probably take several attempts to correct those mistakes before getting it right.
Even though the final code might be more or less the same, the playback would help you discern who has better coding skills.
Of course, this would mainly be of use when attempting to choose between two closely matched coders.
Regarding copying and pasting - at the moment the rules don't explicitly ban this. The problem with allowing it is that the likelihood that the coder might attempt to cheat (and get away with it) is vastly increased (and cutting and pasting makes the playback feature worthless !).
That is why we went to the trouble of enabling syntax highlighting, compilation and testing within the widget, so that it would be as close to an IDE as possible.
BTW - regarding testing - aside from the 2 default test cases, it is also possible to enter your own testdata in the widget.
The coding window assumed I have an american keyboard, which made it sort of hard to program.
grate work guys
http://beta.techtestnow.com/
At the moment, we support 2 languages Java and Ruby. We are adding more questions and languages gradually. And as soon as we have more data, we are going to analyze and fine tune the ranking algorithms.
Here are two example tests,
Java : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c2479e455a032.22415626
Ruby : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c247a1e3fc618.66097557
To view the system as the recruiter, login as "sandaru1/hnhackers". (please don't delete/deactive above tests)
Any feedback would be appreciated.