Historic data from native IB python API
This is the second in a sequence of posts on the way to use the native python API for interactive brokers. This post is an update of the publish I wrotehere, which used the third celebration API swigibpy.
Okay so that you have controlled to run the time telling code in mylast put up.
Now we will do some thing a chunk more interesting, get a few market expenses. Arguably it's far nevertheless not that thrilling, and these things will by no means be as interesting as a respectable ebook or a very good bottle of crimson wine, but we've got all got to get the cash to shop for the ones books and bottles and this is higher than many other options for reaching that.
Note: This publish has been updated to apply a more robust method for handling concurrency.
Let us peruse the IB ABI commands...
Whenever you are trying something new out with the API you need to identify the EClient (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The FriggingManual is worth doing.And indeed there isn't always simply one but four places in the weird and mysterious world of EClient wherein it looks as if prices can be determined (ignoring whacky methods for such things as alternatives):
- reqMktData
- reqHistoricalData
- reqRealTimeBars
- reqMarketDepth
Market Data:Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.
Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.
Real Time Bars:Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.
Market Depth:Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...
Becausereal time bars,market data andmarket depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).
Historical Data AKA Those Who Cannot Remember the Past Are Doomed...
First get the code from this gist somewhere on a server far far away...Also the relevant guide pages are right here.
Getting some rate data
Just run the file (assuming you still have an IB connection available as per the previous post). You should get something like this:Getting complete agreement details from the server...
Getting historical statistics from the server... Should take 10 seconds to complete
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]
"Obviously" those are tuples containing day by day costs (date, open, high, low, near, extent) for the September 2018 Eurodollar futures agreement. If you get an blunders you would possibly want to alternate the expiry date for the agreement. I'll display you a way to do that later inside the submit.
(Note before mixing these with any form of different costs or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:fifty nine:59 to make certain there may be no moderate look forward bias.This is discussed more right here.)
The IB settlement object or How IB Knows What We Want
Some code:ibcontract = IBcontract() ibcontract.SecType = "FUT"
ibcontract.LastTradeDateOrContractMonth="201809"
ibcontract.Symbol="GE"
ibcontract.Alternate="GLOBEX"(Note if to procure an mistakes above here is your hazard to trade the expiry to the modern-day one)
Here is a real 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from thehttps://www.interactivebrokers.com/ product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information touniquely identify a contract:
- Getting the exchange name right - it needs to be exactly the same as on the product page.
- Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver ("USD").
- Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000)
- Be careful, some contracts actually have an expiry date in a different
- You have to specify the exact expiry YYYYMMDD of a VIX contract rather than just YYYYMM because there are now weekly expiries.
(* This is a funny story)
The next aspect we need to do is get the agreement info. This might also sound a piece weird, we already have the contract information on the grounds that we've got just described it? In practice the definition of a contract may be very light and the information we get lower back are plenty richer.
There is one particular case in which you have to do that. To see why suppose it's far some months time and you are trading hundreds of futures contracts. A fill comes in for a selected agreement tool code and month/12 months. However as we will see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the complete expiry 'yyyymmdd'. You can't simply take the first 6 characters of that both due to the fact a few futures simply expire the month earlier than they must. So to be on the safe aspect I commonly get the expiry from IB before trading to ensure I am applying fills towards the ideal thingy.
Actually I don't do this straight away before trading but in a separate procedure that populates a database of contract facts...
So we now want to "solve" this settlement into a fully featured contract which well specifies the whole expiry date:
resolved_ibcontract=app.Resolve_ib_contract(ibcontract)
Getting complete agreement details from the server...
By the way if you haven't specified the initial contract object precisely enough, so there are multiple possible contracts, then you'll get a warning. You may also get a warning if the IB server doesn't deliver a "I am done with sending you contracts now sir" message by activating the wrapper method contractDetailsEnd()
All that is Fascinating Rob But How will we Actually get the Prices?!
Historic_data = app.Get_IB_historical_data(resolved_ibcontract)
This calls the following code in the TestClient item:
def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day", tickerid=DEFAULT_HISTORIC_DATA_ID): """ Returns historic prices for a agreement, up to today ibcontract is a Contract :returns list of charges in 4 tuples: Open excessive low close quantity """
self.Wrapper.Init_error() print("Getting historic information from the server... ")
## Make a place to store the data we're going to return historic_data_queue = finishableQueue(self.Init_historicprices(tickerid))
nowadays = datetime.Datetime.Now() # Request a few historical facts. Native approach in EClient
self.reqHistoricalData( tickerid, # tickerId, ibcontract, # contract, datetime.datetime.today().strftime("%Y%m%d %H:%M:%S %Z"), # endDateTime, durationStr, # durationStr, barSizeSetting, # barSizeSetting, "TRADES", # whatToShow, 1, # useRTH, 1, # formatDate False, # KeepUpToDate <<==== added for api 9.73.2 [] ## chartoptions not used )
historic_data_queue = finishableQueue(self.Init_historicprices(tickerid))
The general structure of this function should be obvious to anyone who read the last post. We:## Wait till we get a finished information, an error, or become bored ready MAX_WAIT_SECONDS = 10 historic_data = historic_data_queue.Get(timeout = MAX_WAIT_SECONDS) whilst self.Wrapper.Is_error(): print(self.Get_error()) if historic_data_queue.Timed_out(): print("Exceeded maximum look ahead to wrapper to verify finished - appears to be ordinary behaviour") self.CancelHistoricalData(tickerid) return historic_data
- call some init functions in the wrapper instance (self.wrapper) so there is somewhere for error messages and our price data to go
- Ask the IB API server for something with the inherited EClient method self.reqHistoricalData()
- Wait patiently until the data is received, or until we get bored
- Complain about any errors, or return the results as appropriate
The self.reqHistoricalData() asks the IB server for historical price data. We can set length of time (a year in this case, although because we are using a demo account there isn't that much data available) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.) even if your hard drive could store them.
About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a'pacing violation' (basically, don't take the mickey).
This is only one aspect of the tale, as....
Oh Yeah I bear in mind now, we need to have an EWrapper characteristic at the server side...
From the TestWrapper class, as usual we have a storage method where we put stuff in a Queue:## Historic information code
def init_historicprices(self, tickerid): historic_data_queue = self._my_historic_data_dict[tickerid] = queue.Queue() return historic_data_queue
And a few techniques in EWrapper that have been overriden:
def historicalData(self, tickerid , bar): ## Overriden approach ## Note I'm selecting to disregard barCount, WAP and hasGaps however you could use them if you want bardata=(bar.Date, bar.Open, bar.High, bar.Low, bar.Near, bar.Volume) historic_data_dict=self._my_historic_data_dict ## Add on to the present day records if tickerid now not in historic_data_dict.Keys(): self.Init_historicprices(tickerid) historic_data_dict[tickerid].Positioned(bardata)
def historicalDataEnd(self, tickerid, begin:str, end:str): ## overriden method if tickerid now not in self._my_historic_data_dict.Keys(): self.Init_historicprices(tickerid) self._my_historic_data_dict[tickerid].Positioned(FINISHED)There is no magic in creating these things; to write this stuff you look in the API reference to see what the wrapper function that requires overriding (in this case historicalData) returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data. In theory historicalDataEnd would also be called, however as with getting contract details this might never happen (I've had mixed success with it).
<snip>
Back inside the TestClient item:
historic_data_queue = finishableQueue(self.Init_historicprices(tickerid))
## Wait till we get a completed statistics, an errors, or become bored waiting MAX_WAIT_SECONDS = 10 historic_data = historic_data_queue.Get(timeout = MAX_WAIT_SECONDS)
It's now worth understanding what a finishableQueue is;
## marker for whilst queue is finishedFINISHED = object() STARTED = object() TIME_OUT = object() magnificence finishableQueue(object): def __init__(self, queue_to_finish): self._queue = queue_to_finish self.Fame = STARTED def get(self, timeout): """ :param timeout: how long to wait before giving up looking forward to a FINISHED flag :go back: list of Queue factors """ contents_of_queue=[] completed=False at the same time as not completed: attempt: current_element = self._queue.Get(timeout=timeout) if current_element is FINISHED: completed = True self.Status = FINISHED else: contents_of_queue.Append(current_element) ## keep going and try and get greater statistics except queue.Empty: ## If we hit a day out it is maximum probably we're not getting a finished element ## surrender and go back what we have finished = True self.Popularity = TIME_OUT go back contents_of_queue def timed_out(self): return self.Status is TIME_OUT
This is just a easy magnificence for maintaining song of Queues of more than one gadgets inside the wrapper thread wherein the consequences have some 'end' factor, which may not come.
And once again, back in the TestClient object:
historic_data = historic_data_queue.Get(timeout = MAX_WAIT_SECONDS)
at the same time as self.Wrapper.Is_error(): print(self.Get_error()) if historic_data_queue.Timed_out(): print("Exceeded most look forward to wrapper to affirm completed - appears to be regular behaviour") self.CancelHistoricalData(tickerid) go back historic_data
So we get this output:
Getting historical statistics from the server... Should take 10 seconds to complete
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]
(By the way precisely the identical finishableQueue idea is used when we clear up the contract details)
That then is the whole lot you need to recognize about historic records. I will cover marketplace records, i.E. The incredible global of streaming facts, in the subsequent submit.
This is the second in a series of posts. The previous post was:qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html
The next post is qoppac.Blogspot.Co.Uk/2017/03/streaming-market-facts-from-local.Html
