asavinov

You could check out a tool for trade signal generation based on machine learning and feature engineering:

https://github.com/asavinov/intelligent-trading-bot

It trains ML models based on historic data and custom features and then uses them to generate a kind of intelligent indicator between -1 and +1. This intelligent indicator is then used to make trade decisions. Frequency is a parameter and can vary from 1 minute for crypto trading to 1 day for normal exchanges.

asavinov

2년 전discuss

The sequence

    FROM Table AS t WHERE t.Condition SELECT t.col1, t.col2, ...

might be more natural than the traditional

    SELECT t.col1, t.col2, ... FROM Table AS t WHERE t.Condition

If we compare it with how loop are described in programming languages:

    Loop -> SELECT-FROM-WHERE
    Table -> Collection
    AS t -> Loop instance variable
    WHERE -> condition on instances

In Java and many other PLs, we write loops as follows:

    foreach x in Collection
        if x.field == value:
            continue
        // Do something with x, for example, return in a result set

So we first define the collection (table) we want to process elements from. Then we think about the condition they have satisfy by using the instance variable. And finally in the loop body we do whatever we want, for example, return elements which satisfy the condition.

In Python, loops also specify the collection first:

    for x in Collection:

Python list comprehension however uses the traditional order:

    [(x.col1, x.col2) for x in Collection if x.field2 == value]

Here we first specify what we want to return, then collection with condition.

asavinov

3년 전discuss

From my experience with crypto currencies and the intelligent trading bot [0] I would say that transformers will not provide significant benefits when applied to the traditional statistical (numeric) forecasting problems. Such models assume that old events do not affect too much current events.

Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago.

[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

asavinov

3년 전discuss

I agree that the conventional (numeric) forecasting can hardly benefit from the newest approaches like transformers and LLMs. I made such a conclusion while working on the intelligent trading bot [0] by experimenting with many ML algorithms. Yet, there exist some cases where transformers might provide significant advantages. They could be useful where the (numeric) forecasting is augmented with discrete event analysis and where sequences of events are important. Another use case is where certain patterns are important like those detected in technical analysis. Yet, for these cases much more data is needed.

[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

asavinov

3년 전discuss

Does it work synchronously or asynchronously? For example, if you integrate WebSockets then you could act as soon as you get new data. If it works synchronously then you regularly request data (say, every second) and then act.

If you process many positions and then choose top 5 candidates then do you choose one of them for buying or you can allocate resources between them (depending on some score)?

asavinov

3년 전discuss

When developing an automatic trading system the following aspects are important:

- Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format

- Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work

- Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds.

- Real trading and order management as well as coordination of all activities.

The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels.

I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important

asavinov

3년 전discuss

> What gives you advantage is trading algo, which is always hard to find.

At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:

https://github.com/asavinov/intelligent-trading-bot

Currently it works for cryptocurrencies but can be applied to other markets:

https://t.me/intelligent_trading_signals

> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.

It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.

asavinov

3년 전discuss

For any such tool, two questions are of primary importance:

- How connections between multiple tables are represented and managed

- How derived data is described (queries, workflows etc.)

Typically such tools are aimed at simplifying data connections but normally they end up with some kind of join-like approach which requires high expertise and is error prone. So users have to deal with something they wanted to get rid of when they buy the tool. Plato is not an exception: "No SQL needed". Yet, I could not find any information on how exactly it manages connections between tables and how the unified "virtual table" is defined.

The second question is about how we can derive new data from existing data. Ideally, users would like to have something very similar to Excel because spreadsheets are indeed extremely intuitive: we define new cells as functions of other cells (which in turn might be functions of other cells). In Plato I found "virtual columns" which should be rather useful. This is somewhat similar to the column-oriented approach implemented in Prosto [0]. Yet, what is really non-trivial is how to define (derived) columns by combining data from multiple tables.

In general, the tool looks very promising and I hope that additional features and additional information will make it really popular.

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

asavinov

4년 전discuss

> I have always kept in mind is that feature engineering is almost always the key difference between success and failure

I also developed an ML-powered service heavily relying on feature engineering

https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot

Its difference from Didact is that this intelligent trading bot is focused on trade signal generation with higher frequency of evaluation. It is more suitable for cryptocurrencies but also works for traditional stocks with daily frequencies so it could be adapted for stock picking. What I find interesting in your work is the general design of such kind of ML systems relying on feature engineering.

asavinov

4년 전discuss

> Joins are what makes relational modeling interesting!

It is the central part of RM which is difficult to model using other methods and which requires high expertise in non-trivial use cases. One alternative to how multiple tables can be analyzed without joins is proposed in the concept-oriented model [1] which relies on two equal modeling constructs: sets (like RM) and functions. In particular, it is implemented in the Prosto data processing toolkit [2] and its Column-SQL language [3]. The idea is that links between tables are used instead of joins. A link is formally a function from one set to another set.

[1] Joins vs. Links or Relational Join Considered Harmful https://www.researchgate.net/publication/301764816_Joins_vs_...

[2] https://github.com/asavinov/prosto data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

[3] Column-SQL https://prosto.readthedocs.io/en/latest/text/column-sql.html

asavinov

4년 전discuss

One idea is to use columns instead of cells. Each column has a definition in terms of other columns which might also be defined in terms of other columns. If you change value(s) in some source column then these changes will propagate through the graph of these column definitions. Some fragments of this general idea were implemented in different systems, for example, Power BI or Airtable. The main difficulty in any formalization is how to deal with columns in multiple tables.

This approach was formalized in the concept-oriented model of data which relies on two basic elements: mathematical functions and mathematical sets. In contrast, most traditional data models rely on only sets. Functions are implemented as columns. This model gets rid of joins and groupby by making data processing simpler and more intuitive.

This approach was implemented in the Prosto data processing toolkit: https://github.com/asavinov/prosto

asavinov

4년 전discuss

Looks interesting although I miss a short introduction or how-to guide. I found that one can "Create and train machine learning models to predict market values." In this context, a related project is Intelligent Trading Bot: https://github.com/asavinov/intelligent-trading-bot which is intended for generating trading signals based on ML and feature engineering

asavinov

4년 전discuss

Having Python expressions within a declarative language is a really good idea because we can combine low level logic of computations of values with high level logic of set processing.

A similar approach is implemented in the Prosto data processing toolkit:

https://github.com/asavinov/prosto

Although Prosto is viewed as an alternative to Map-Reduce by relying on functions, it also supports Python User-Defined Functions in its Column-SQL:

  prosto.column_sql(
    "CALCULATE  Sales(quantity, price) -> amount",
    lambda x: x["quantity"] * x["price"]
  )

In this Column-SQL statement we define a new calculated column the values of which are computed in Python (as a sum of two columns). An advantage is that we can process data in multiple tables without joins or groupbys which is much easier than in the existing set-oriented approaches. Another advantage is that we can combine many statements by defining a workflow in an Excel-like manner.

asavinov

5년 전discuss

Similar products:

- https://rows.com/

- https://www.rowshare.com/

- https://www.airtable.com/

- fieldbook

- Power BI

asavinov

5년 전discuss

Most of the self-service or no-code BI, ETL, data wrangling tools are am aware of (like airtable, fieldbook, rowshare, Power BI etc.) were thought of as a replacement for Excel: working with tables should be as easily as working with spreadsheets. This problem can be solved when defining columns within one table:

  ColumnA = ColumnB + ColumnC, ColumnD = ColumnA * ColumnE

we get a graph of column computations similar to the graph of cell dependencies in spreadsheets.

Yet, the main problem is in working multiple tables: how can we define a column in one table in terms of columns in other tables? For example:

  Table1::ColumnA = FUNCTION(Table2::ColumnB, Table3::ColumnC)

Different systems provide different answers to this question but all of them are highly specific and rather limited.

Why it is difficult to define new columns in terms of other columns in other tables? Short answer is that working with columns is not the relational approach. The relational model is working with sets (rows of tables) and not with columns.

One generic approach to working with columns in multiple tables is provided in the concept-oriented model of data which treats mathematical functions as first-class elements of the model. Previously it was implemented in a data wrangling tool called Data Commander. But them I decided to implement this model in the Prosto data processing toolkit which is an alternative to map-reduce and SQL:

https://github.com/asavinov/prosto

It defines data transformations as operations with columns in multiple tables. Since we use mathematical functions, no joins and no groupby operations are needed and this significantly simplifies and makes more natural the task of data transformations.

Moreover, now it provides Column-SQL which makes it even easier to define new columns in terms of other columns:

https://github.com/asavinov/prosto/blob/master/notebooks/col...

asavinov

5년 전discuss

Hamilton is more similar to the Prosto data processing toolkit which also relies on column operations defined via Python functions:

https://github.com/asavinov/prosto

However, Prosto allows for data processing via column operations in many tables (implemented as pandas data frames) by providing a column-oriented equivalents for joins and groupby (hence it has no joins and no groupbys which are known to be quite difficult and require high expertise).

Prosto also provides Column-SQL which might be simpler and more natural in many use cases.

The whole approach is based on the concept-oriented model of data which makes functions first-class elements of the model as opposed to having only sets in the relational model.

asavinov

5년 전discuss

> I think SQL is irritatingly non-composable, many operations require gymnastics to express

One approach to radically simplify operations with data is to use mathematical functions (in addition to mathematical sets) which is implemented in Prosto data processing toolkit [0] and (new) Column-SQL [1].

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit - an alternative to map-reduce and join-groupby

[1] https://prosto.readthedocs.io/en/latest/text/column-sql.html Column-SQL (work in progress)

asavinov

5년 전discuss

One alternative to SQL (type of thinking) is Column-SQL [1] which is based on a new data model. This model is relies on two equal constructs: sets (tables) and functions (columns). It is opposed to the relational algebra which is based on only sets and set operations. One benefit of Column-SQL is that it does not use joins and group-by for connectivity and aggregation, respectively, which are known to be quite difficult to understand and error prone in use. Instead, many typical data processing patterns are implemented by defining new columns: link columns instead of join, and aggregate columns instead of group-by.

More details about "Why functions and column-orientation" (as opposed to sets) can be found in [2]. Shortly, problems with set-orientation and SQL are because producing sets is not what we frequently need - we need new columns and not new table. And hence applying set operations is a kind of workaround due the absence of column operations.

This approach is implemented in the Prosto data processing toolkit [0] and Column-SQL[1] is a syntactic way to define its operations.

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit - an alternative to map-reduce and join-groupby

[1] https://prosto.readthedocs.io/en/latest/text/column-sql.html Column-SQL (work in progress)

[2] https://prosto.readthedocs.io/en/latest/text/why.html Why functions and column-orientation?

asavinov

5년 전discuss

Here is another project based on the same idea of processing data using functions:

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

Yet, here the focus is on feature engineering and rethinking how it can be combined with traditional ML. Essentially, the point is that there no big differences and it is more natural and simpler to think of them as special cases of the same concept: features can be learned and ML models are frequently are used for producing intermediate results.

asavinov

5년 전discuss

The main motivation is that the conventional approaches to data processing are based on manipulating mathematical sets for all kinds of use cases: we produce a new set if we want to calculate a new attribute, we produce a new set if want to match data from different tables, we get a new set if we aggregate data. Yet, we actually do not need to produce new sets (table, collections etc.) in many cases - it is enough to add a new column to an existing set. Here are more details about the motivation:

https://prosto.readthedocs.io/en/latest/text/why.html

Column is an implementation of a function (similarly to how table is an implementations of a set). Theoretically, this approach leads to a data model based on two core elements: mathematical functions (new) and mathematical sets (old).

This approach was implemented in Prosto which is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby.

Submissions

Show HN: Intelligent Trading Bot using machine learning and feature engineering

Data Processing Using Functions in Prosto: An Alternative to Map-Reduce and SQL

Data Processing Using Functions in Prosto: An Alternative to Map-Reduce and SQL

Show HN: Functions matter – an alternative to SQL and map-reduce data processing

Show HN: Lambdo – Feature engineering and machine learning together

When Is It Important for an Algorithm to Explain Itself?

Oracle pays artificial intelligence experts $6M

Understanding Bistro Streams: Counting Clicks by Region

Show HN: Bistro Streams – A light-weight column-oriented stream analytics server

Show HN: Bistro – A light-weight column-oriented data processing engine

Unsolved Problems in AI

From Group-By to Accumulation: Data Aggregation Revisited

Android O beta is available today

Show HN: Data Commandr – As Easy as a Spreadsheet, as Powerful as SQL

Clash with Tesla CEO Musk Forces Grohmann to Quit

Relational Join Considered Harmful

How to build the next Trello and sell it for $425M or more

comments