Intermittent IB historical tick backfill

Questions about MultiCharts and user contributed studies.
Leggo

Intermittent IB historical tick backfill

Postby Leggo » 07 Nov 2006

When I request historical tick data from InreactiveBrokers (I am using latest version of tsIBDataFeed.dll 99.99), often (especially when I want to back fill more than 2 days) the download will stop in the middle and the log on quotemanager will stay frozen saying:
"Data request for ESZ6 1 Tick Trade [......] Id=..."
After a while (from 10-20 seconds to 2-3 minutes) the download will resume and will stop again at some point, only to resume again a bit later . The chart eventualy plots but the delay is very big when adding this extra waiting time when data stops being received.

I know that the IB server sends ticks slowly but why the flow is not continuous and starts and stops? Is it the IB server that stops responding or an issue with MC? I wonder, can the download time be shortened in future version possibly?

Leggo

User avatar
Alex Kramer
Posts: 834
Joined: 23 Feb 2006

Postby Alex Kramer » 08 Nov 2006

Such delays signify that a connection was broken at some level ; could you please send us the full log from QuoteManager so we could tell what exactl happened?

Leggo
Posts: 1
Joined: 08 Nov 2006

Postby Leggo » 08 Nov 2006

Hi Alex,

I have attached the log file from yesterday. I hope it you will find it useful to identify the issue. I have indented the lines where the delay (server disconnect) occurs.

In my case this happens very often when I request a long historical tick backfill. Interestingly, the server eventually reconnects, sends some more ticks, disconnects again so on. Eventually the chart is plotted but with a serious delay. I have not noticed such delays when requesting minute backfills.

Leggo
Attachments
Interactive Brokers[tsServer].txt
(42.36 KiB) Downloaded 679 times

User avatar
Alex Kramer
Posts: 834
Joined: 23 Feb 2006

Postby Alex Kramer » 09 Nov 2006

We've analyzed the logs you sent and here's the developers conclusions -

After receiving the connection status from TWS

Historical Market Data Service query message:HMDS server disconnect occurred. Attempting reconnection...

a repeated query is made, after which TWS sends the

HMDS server connection was successful

state and then returns data. During the delays lasting 2 minutes and longer there were no messages incoming from TWS so there's nothing that could be done at our level - the data vendor was just waiting for the query to be executed.[/b]

Guest

Postby Guest » 16 Nov 2006

Well there is something you can do and that is handle data requests more asyncronously. Plot what you have already recieved and go and do something else while the connection is re-established. Currently MC waits for all the data before plotting.

User avatar
Alex Kramer
Posts: 834
Joined: 23 Feb 2006

Postby Alex Kramer » 16 Nov 2006

We'll consider this, thanks.

templ
Posts: 86
Joined: 19 Nov 2006
Has thanked: 2 times
Been thanked: 2 times

Postby templ » 20 Nov 2006

Hello,
I have MultiCharts demo version.
I received some data from the german Bund Future, but now it looks like MultiCharts QuoteManager does nothing to receive the symbols I enabled to online mode.

But if I do historical data downloads with Excle over a VBA script I can receive all the data I want! But why does MultiCharts QuoteManager only shows this log messages?

Code: Select all

[11/20/2006 23:34:21] TWS message: id= -1; errorCode= 2106; errorMsg= HMDS data farm connection is OK:ushmds2a
[11/20/2006 23:38:21] TWS message: id= -1; errorCode= 2107; errorMsg= HMDS data farm connection is inactive but should be available upon demand.:ushmds2a
In this 4 minutes from 23:34:21 to 23:38:21 MultiCharts received nothing, but I was able to download historical data with Excle over a VBA script.

Also if i do a refresh to a symbols nothing happend, not even in the log of QuoteManager.

One mor example:

in this over 10 minutes (showed in the second log code) nothing happend. But if I do a download with Excle for a us symbol and a europian symbol, the log of MultiCharts refreshed this two lines.

Code: Select all

[11/20/2006 23:49:30] TWS message: id= -1; errorCode= 2106; errorMsg= HMDS data farm connection is OK:ushmds2a
[11/20/2006 23:49:55] TWS message: id= -1; errorCode= 2106; errorMsg= HMDS data farm connection is OK:euhmds2
But if i dont receive data, and if I refresh in MultiCharts nothing happend.

Please help. Thanks.

PS: what is the diffrence in Data server mode between :
On-demand and
online?

If i open a chart in On-demand, than i can see in QuoteManager log, that datas are loading. But what in online mode? Can this mode not load missing/historical data? And what will be with the data, if for this symbol in QuoteManager is offline

User avatar
Alex Kramer
Posts: 834
Joined: 23 Feb 2006

Postby Alex Kramer » 21 Nov 2006

In the Online mode the product connects to the live datafeed as soon as possible, without checking for history or missing data.
For loading history data and repairing gaps you'll need the On-Demand mode.

Please set the mode to On-Demand and send us the entire log from QuoteManager (right-click and use Send Log)

Offline means the data for the symbol is loaded and stored only when it is charted in some window; Online means data for the symbol is collected as long as MultiCharts is running.

templ
Posts: 86
Joined: 19 Nov 2006
Has thanked: 2 times
Been thanked: 2 times

Postby templ » 21 Nov 2006

Sorry,
I dont undersand for example, how to store 1 year of minute data. I dont find an option in QuoteManager to set the data range like it could be done in charts with the Format Symbol Settings.

User avatar
Alex Kramer
Posts: 834
Joined: 23 Feb 2006

Postby Alex Kramer » 21 Nov 2006

Plot a chart for that 1 year of minute data once. Anything shown on the charts is stored in the database.

templ
Posts: 86
Joined: 19 Nov 2006
Has thanked: 2 times
Been thanked: 2 times

Postby templ » 21 Nov 2006

Plot a chart for that 1 year of minute data once. Anything shown on the charts is stored in the database.
Ok, now it looks like this is working with me.

I repeat the way I understood:
  1. add a symbol to Quotemanager (realtime status don't care?)
  2. open a chart in On-Demand mode (For loading history data and repairing gaps)
  3. setting the data range in chartsettings
  4. receiving the data in the chart
  5. control the data in Quotemanager with the Edit Data contextmenue
For a beginner it is little complex and irritating, that this procedure of collecting data is separated in two parallel operations.

What do you think to do it in Quotemanager only (if wanted), with an additive column, like I described here?
In QuoteManager an additive column for Historical Status. This status should show the stored record (absolute available) as you can see in Realtime Status the received records of this session.
Sreenshot of QuoteManager with an additive column for Historical Status:
Image

http://forum.tssupport.com/viewtopic.php?t=938&start=75

How to load and store several symbols like 20 or more, for example if you begin with Multicharts or if you subscribe to for marketdata by an additional exchange etc.?

User avatar
Alex Kramer
Posts: 834
Joined: 23 Feb 2006

Postby Alex Kramer » 21 Nov 2006

Considering the technical limitations, our method of adding symbols is, as agreed by a number of testers, very convenient and more logical than of a number of other products.
Unlike, say QuoteTracker or AB we have (especially for IB) a well-workin lookup search, with automatic detection of exchanges and currencies, the user does not need to know the exact specifications.
Is there any example you would like us to follow - can you name such a product?

Leggos

Postby Leggos » 28 Nov 2006

I am revisiting this thread because of some further observations.

Again the problem remains that historical tick data from IB are downloaded very slowly by Quotemanager and in addition there are frequently long periods often in excess of 2 minutes when no data is received. The big delay appears only when downloading tick historical data, minute data are downloaded without problems.

I have just used Fibonacci Trader software (the version for IB) on a free trial basis. The software can download historical tick data from IB at a very high speed. A couple of months worth of ticks for a single symbol download in a matter of a couple of minutes,a process that would take Multicharts possibly around an hour and a half (or even more?) with the delays (disconnects) currently experienced.

Now another user (widgetman) states that NT is also very fast when downloading IB tick data: forum.tssupport.com/viewtopic.php?t=2643


So it seems that the issue lies with the MC implementation rather than with IB historical servers. I believe this is something that needs to be fully addressed and ultimately improved, as it greatly affects all IB customers that are currently using or that will possibly consider to use MC in the future.

I'll just add that this is not meant as criticism but as input to help you eventually make MC which is a very promising software into an indispensable trading tool.

Leggo

templ
Posts: 86
Joined: 19 Nov 2006
Has thanked: 2 times
Been thanked: 2 times

Postby templ » 30 Nov 2006

http://forum.tssupport.com/viewtopic.php?t=1633&start=0
In this MultiCharts beta release, it is much faster than in my normal/old version 1.90.468.865 ! :idea:
I tried it unfortunately as recently as today.


Return to “MultiCharts”