Prophecy.io | Think Cursor for Data Analysts | US (onsite SF, remote)
Prophecy is used by data engineers in the largest enterprises (SeriesB, $120M raised). Our new product for data analysts is agentic and multi-modal (docs, low-code, code).
- AI Research Engineer - AI Agentic Software Engineer
Looking for top tier AI engineers who are excited to ship products at tremendous pace
email me - raj at prophecy dot io
Yes, but in addition to being ACID compliant (serializable ANSI), you need to support a million transactions per second - it’s not just about data size.
That’s assuming you’re not using low-code. There are inbuilt connectors to read data, transform data, read/write to pinecone, make api calls to LLMs. It is much faster to prototype with Prophecy.io
It is pointless - LlamaIndex and LangChain are re-inventing ETL - why use them when you have robust technology already?
1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.
2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.
Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN.
Yes but this is just ETL - LlamaIndex and LangChain are re-inventing it - why use them when you have robust technology already?
1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.
2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.
Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN. Here’s our talk from Data+AI Summit:
Build a Generative AI App on Enterprise Data in 13 Minutes
A lot of B2B startups can technically the cloud API to provide value added applications to Enterprises, but often the banks and healthcare companies will not want their data running through startups pipes to OpenAI pipes.
We provide a low code data transformation product (prophecy.io), and we’ll never close sales at any volume, if we have a to get an MSA that approves this. Might get easier if we become large :)
Expert in Spark, Spark internals or Expert in Query Compilers.
Level variable (2-20 yrs exp), prefer 10+ years
Prophecy.io | Data Engineer | Palo Alto, CA
Expert in writing Spark code and running Spark at scale with substantial production and performance improvement experience. Work with our customers and help product design.
Prophecy.io | Customer Success Director | Palo Alto, CA
We’re a game changing low-code data engineering platform, replacing ETL tools and modernizing the modern data stack.
You know data engineering well, have excellent customer facing experience (consulting works). You will work with our enterprise customers, coordinating with our strong data engineers to make the customers successful. Be their trusted advisor, no upsell responsibility.
Its not public language, but at Risk Management Systems, I worked on Contract Definition Language- an external DSL for computable insurance, re-insurance contracts. This was used to simulate insurance losses when catastrophic events happen (earthquake, hurricane)
Thank god solving customer problems has taken over the tech-ceo job, starting with tech is usually a bad idea, even when it is innovative (crypto-ponzi as an example)
Prophecy.io | Customer Success Lead | InPerson Palo Alto
We have customers from midsize to Fortune 500 relying on us for Data Engineering. We’re building out our success organization to onboard, enable and guide the customers. We do have a top notch data engineers in the team, need leadership. Level is flexible.
send message to raj @ prophecy.io, only primary candidates, thanks!
Prophecy.io is hiring a technical content writer in our marketing org. Requires data engineering, spark knowledge, good communication and writing skills and desire to move toward product marketing - the most in demand skill right now!
Based out of our Palo Alto office, full time, good compensation, benefits.
We’re very different from the ETL tools in that we’re bring software development best practices to data.
When you do visual drag and drop - prophecy is generating high quality code on git that is 100% open source (spark, airflow), you have tests and CI/CD - so you’re visually doing solid data engineering.
You can toggle between code and visual - so if you change the code (some), the visual graph updates - so small edits directly to git don’t break the visual layer.
All visual components are generated from a spec - think a Spark function with a some more info. So the data platform teams will create their own library/framework and roll it out to the wider teams. How it works us that in the visual editor, you start with standard Spark library, but can load visual components for delta, or encryption or data quality.
Our customers are typically fed up of these ETL tools and moving to us. We can also import the ETL formats (AbInitio, Informatica, …) in an automated way (we reverse engineered their formats and created source to source compilers)
Prophecy.io let’s you create visual components from any Spark function. Same with Airflow. So you can use standard components (built-in or your new ones) without being restricted.
Founder here - we’re working to solve this exact problem.
Our US based startup has engineering in India (20 engineers) and the biggest challenge is finding engineering leadership.
It is hard to hire in the bay area, especially leaders who have experience shipping high quality products (systems), so we built the team in India.
Many young companies like us struggle since a lot of the talent (definitely in the bay area) is locked up in the Faangs where poor fiscal/monetary policy in the US has inflated stock part of the compensation so much, that it makes no sense to leave.
So it is hard find principal engineers to do great work because they choose to go to large companies and get poor work for money.
The bar is high for us though, we’d look for engineers with specialized skills (compiler, database internals) and leadership on complex products. The middle is the dead zone (engineers with N years of experience wiring apps)
Not sure about the European market, but maybe you can try to go to a smaller company?