Testing Windows Live Writer

Started using Windows 7 on a netbook (my first one) this week. The first impression of a netbook is that ‘I wish this would fit in my pocket – It has everything I want in a netbook’. However, Windows 7 *seems* a bit heavy on a device with lower end parameters.

While discovering various new things under the hood, I stumbled upon this – the program I am using to compose this blog entry – Windows Live Writer. This is an old program – but new to me :)

This wonderful program allows me to WYSIWIG-compose blogs while offline and post them when I please. I can preview the blog entry while offline (once I have downloaded the whole blog of course).

All in all, I am quite impressed with this little program. Hopefully, the next blog entry will be about extending this program to add funky things. Now, it already provides adding photo albums (illustrated below with pics of my darling daughter), etc. It will be a challenge to top that. A challenge fit for a geek…

View Full Album

Online Stock Quotes - Using Python S60

I recently came across an article about a service provided by the Yahoo! finance website, which return details about stock quotes in comma separated values (CSVs). Many folks have used this service in their programs to display (almost) real time stock data. Of course, I had to make my phone do the same :) As we will see, Python makes it look easy - no, Python makes it look GOOD!

Background

The Yahoo! finance service http://quote.yahoo.com/d/quotes.csv is a very simple yet powerful service. The user can control the fields in the stock quote data through a format string. The return data contains stock data fields in CSV format. The fields and their order is based on the format string. This is another example of the usage of this service.

Cache Management

The application on the phone should minimize the network traffic. However, care should be taken that the data is updated as frequently as possible to keep the real-time nature of the data. To achieve this, there is a very rudimentary cache management technique implemented in the module. The cache management checks how old the stock data is and if it is older than a certain expiry. The network traffic is generated only if the data is expired.

Chart

This module also allows downloading the chart for a given symbol. The chart is downloaded from yet another Yahoo! finance service http://ichart.finance.yahoo.com/t The response of this service is a intra-day chart for the latest day. If a chart exists, the response is of type image/png. This can be used as a test.

Implementation

The following code snippet is from the quotes.py module. This piece works well on 3rd edition phones:

# These times should be set to minimize the network traffic
QUOTEEXPIRY = 60 # 1 Minute
CHARTEXPIRY = 300 # 5 Minutes
CACHEDIR = 'c:\\cache\\'

# If the file does not exist or if the creation time of the
# existing file is older than expiry, then return True
def is_expired(filename, expiry):
    expired = False
    if not os.path.exists(filename):
        expired = True
    else:
        lastmod = os.stat(filename)[ST_MTIME]
        currtime = int(time.time())
        if currtime > lastmod + expiry:
            expired = True
    return expired

# This function downloads the chart for a given symbol
# If the symbol does not have a chart, the function returns
# False, else returns True
def get_chart(symbol):
    retval = True
    filename = CACHEDIR+symbol.upper()+'chart.png'
    if is_expired(filename, CHARTEXPIRY):
        name, status = urllib.urlretrieve('http://ichart.finance.yahoo.com/t?s='+symbol.upper(), filename)
        if status.type != 'image/png':
            retval = False
            os.remove(filename)
    return retval

# Get the CSV from yahoo server and return parsed values
# Format is known and hence the cryptic retval parsing
def get_quote(symbol):
    filename = CACHEDIR+symbol.upper()+'quote.txt'
    if is_expired(filename, QUOTEEXPIRY):
        urllib.urlretrieve('http://quote.yahoo.com/d/quotes.csv?s='+symbol.upper()+'&d=t&f=sl1d1t1c1ohgvj1pp2xwenr', filename)
    f = file(filename)
    data = f.read()
    f.close()
    names = ['sym', 'last', 'date', 'curtime', 'change', 'open', 'high',
             'low', 'vol', 'mcap', 'close', 'pctchg', 'exchange', 'annrange',
             'earnings', 'name', 'peratio']
    values = map(lambda s: s.strip('"'), data.split('\n')[0].split(','))
    return dict(zip(names, values))

The meaning of the format string can be found from the correspondance between the string and the names array.

The complete implementation is located here

Summary

This log is yet another example of power and elegance of Python. The way zip() function simplifies creation of a very usable dictionary out of an array.

Google Local's AJAX Data Objects Demystified

JSON is slowly, but steadily, becoming an Internet standard. Refer to the RFC 4627. Not too long ago, Google Local server changed its AJAX data object format from XML to JSON-like notation. In this log, we will see how we tweak the Google Local data object to suit our JSON parser (a.k.a. eval())

Background

JSON standard requires the name in the name value pair to be a string enclosed in quotation marks. The Google Local server sends AJAX data object in a JSON-like format in which the name is not enclosed in quotation marks. For example:

{
  title: "from: ... to: 15939 ... - Google Maps",
  vartitle: "",
  url: "/maps?saddr=...&daddr=...&ie=UTF8",
  urlViewport: false,
  form:
  {
    selected: "d",
    q:
    {
      q: "from: ... to: ..."
    },
    .
    .
    .
}

This disparity between actual data object and the standard can easily be eliminated by enclosing all the names in quotes. The best way to achieve this is to use regular expression. A regular expression can easily find a name by searching for a alphanumeric word before colon. Of course, care must be taken not to confuse words before colons within a string value. The following code snippet shows how to achieve this:

# Javascript constants not known to python
false = False
true = True
null = None

# Helper RegEx based function to enclose names in quotes
# (which is required in JSON) - Google server does not put
# quote around names
# E.g. {title: "From: Home To: Office"} is converted into
#      {"title": "From: Home To: Office"}
import re
pat = re.compile(r"""
 (?P<name>\w+)       # Name of JSON pair
 \s* :               # Whitespace, and a colon
 |                   # Or
 (".*?")             # Enclosed in double quotes
""", re.VERBOSE)
def subfunc(match):
    # If pattern was found enclosed in double quotes,
    # do not substitute
    if match.group(2):
        return match.group(2)
    else:
        return '"'+match.group(1)+'"'+':'

This "pre-processed" data object is ready to be consumed by our JSON parser to give us all the information we need from the Google Local server.

Summary

This little log just goes on to show that with Python (even on an S60 handset) we can get the information we need in the format we want with minimal effort. I wasted much time thinking about parsing techniques before I stumbled across this simple solution. And there is unmistakable elegance in simplicity, don't you think?

JSON Parser Using Python

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. More information about JSON can be found at http://www.json.org/. Yahoo's Douglas Crockford developed this format and within a short period, many web based services have adopted JSON as the primary data interchange format. In this log, we will look at one such service.

Background

There are a bunch of JSON parsers out there, such as:

• simplejson
• json-py

An example JSON document appears as follows:

{
  "Image": {
  "Width": 800,
  "Height": 600,
  "Title": "View from 15th Floor",
  "Thumbnail": {
    "Url": "http://www.example.com/image/481989943",
    "Height": 125,
    "Width": "100"
    },
  "IDs": [116, 943, 234, 38793]
  }
}

For a Python developer, a JSON document is nothing but a dictionary with each element being either an element, array or another dictionary. We can use this fact to our advantage, by simply using the eval() function provided by Python to not only parse this document, but to create a data structure in one statement!

The following code snippet shows how to use the eval() function to parse JSON object. This snippet is based on http://developer.yahoo.com/python/python-json.html:

# If needed set the proxy address
import os
os.environ['http_proxy'] = 'http://[proxy_host]:[proxy_port]/'
import urllib
APP_ID = 'YahooDemo' # Change this to your API key
SEARCH_BASE = 'http://api.search.yahoo.com/WebSearchService/V1/webSearch'

class YahooSearchError(Exception):
    pass

def search(query, results=20, start=1, **kwargs):
    kwargs.update({
        'appid': APP_ID,
        'query': query,
        'results': results,
        'start': start,
        'output': 'json'
    })
    url = SEARCH_BASE + '?' + urllib.urlencode(kwargs)
    f = urllib.urlopen(url)
    buff = f.read().replace('\\/', '/')
    f.close()
    result = eval(buff)
    if 'Error' in result:
        # An error occurred; raise an exception
        raise YahooSearchError, result['Error']
    return result['ResultSet']

info = search('json python')
results = info['Result']
for result in results:
    print result['Title'], result['Url']

Caveats

The eval() function is not safe from programmer errors. Putting a try...catch around the function call fixes that problem. But there is another sinister problem with eval(), not unlike SQL. The eval() function will not distinguish between a JSON object and a malicious statement. Consider the following:

import os
.
.
eval("os.remove('something_very_important')")

Care should be taken to use eval when the string to be evaluated is from a trusted source. I consider Yahoo! or Google servers trusted.

Summary

This article briefly touches upon using the eval() function to understand JSON documents. This will be further elaborated when we will look at Google Local client-server interactions later. I hope you find this trick useful...

Virtual Earth Maps on the Go

For a map client, it is necessary to get the maps for a given location. Most GPS navigation systems store the maps of the entire nation at all zoom levels in their memory. On handsets, this might not be the best strategy. We know that Virtual Earth server has all the maps required for a map client, just an HTTP request away. All we need to know is how to get them.

Background

In the last log, we checked out some of the geocoding services provided by virtual earth and used them in our Python on S60 programs. In this one, we will see how to download the maps of the location of your choice. Our primary goal here is to find a mapping function from latitude, longitude, zoom to a URL to download the map. Once we have a mapping function, Python will take care of all the basic functionality to give us a fully functional map client without having to store large amount of data.

How Virtual Earth Stores Maps

The Virtual Earth map system, like many map systems (e.g. Google Maps) uses Mercator projection to fit Earth's curved suface onto a flat sheet (screen in present case). The Mercator projection as described in the Wikipedia article is a mathematical scheme that converts latitude into an y value and longitude into x. These x and y values are then used very easily to store and retrieve maps from the server. When I started putting together a map client for Virtual Earth, I thought that once this mapping function is implemented, half my job will be done. But that wasn't to be as Virtual Earth server stores quad-tree encoded x, y values (a la Google Maps satellite images server).

Mapping Function

The mapping function to map the latitude, longitude, zoom to map URL is as follows:

url = f(latitude, longitude, zoom)

import math
earthRadius = 6378137
earthCircum = earthRadius * 2.0 * math.pi
earthHalfCirc = earthCircum / 2.0

def LongitudeToXAtZoom(lon, zl):
    arc = earthCircum / ((1 << zl) * 256)
    metersX = earthRadius * DegToRad(lon)
    return int(round((metersX + earthHalfCirc) / arc))

def LatitudeToYAtZoom(lat, zl):
    arc = earthCircum / ((1 << zl) * 256)
    sinLat = math.sin(DegToRad(lat))
    metersY = earthRadius / 2 * math.log((1.0 + sinLat) / (1.0 - sinLat))
    return int(round((earthHalfCirc - metersY) / arc))

def DegToRad(d):
    return d * math.pi / 180.0

def RadToDeg(r):
    return r * 180.0 / math.pi

def TileToQuadKey(tx, ty, zl):
    quad = ''
    for i in range(zl, 0, -1):
        mask = 1 << (i - 1)
        cell = 0
        if (tx & mask) != 0:
            cell = cell + 1
        if (ty & mask) != 0:
            cell = cell + 2
        quad = quad + str(cell)
    return quad

def GetTileSpecs(lat, lon, zoom):
    tx = LongitudeToXAtZoom(lon, zoom) / 256
    cx = LongitudeToXAtZoom(lon, zoom) % 256
    ty = LatitudeToYAtZoom(lat, zoom) / 256
    cy = LatitudeToYAtZoom(lat, zoom) % 256
    server = ((tx & 1) + ((ty & 1) << 1)) % 4
    q = TileToQuadKey(tx, ty, zoom)
    filename = 'r%s.png' % q
    url = 'http://r%d.ortho.tiles.virtualearth.net/tiles/%s?g=22' % (server, filename)
    return (filename, url, cx, cy)

lon = -117.068092
lat = 32.9913528
zoom = 17
print GetTileSpecs(lat, lon, zoom)

This Python code snippet is another exmaple of how easily we can implement mapping client in Python. Adding Python for S60 wrapper around this to have a map client on the phone. I have done some ground work to get you started. Avaiable at:

• velocation.py


• vemaps.py

Summary

Hopefully, this log will get you excited, not just about Python, but about mapping techniques as well. It is very interesting to learn how map service providers implement their services. I hope you can build upon the available code to add address search and direction finding using the routines in velocation.py. Happy mapping...

Virtual Earth Geocoding Services - Python Style

In this log, I will take you through using Python to utilize the Virtual Earth geocoding services on your PyS60 handset.

Background

Geocoding is the analysis technique of geo-demographic data such as ZIP codes, counties, regions, etc. These techniques include, among others, translating a postal address into a geographical coordinates. Web portals such as Google Maps and Virtual Earth allow the users to do this through their portals. What if you had power to do this from a handset, which might not be able to open these JavaScript heavy web portals?

Virtual Earth Services

Virtual Earth is a Microsoft product that provides web bases map services amongst other to the users. You can find a US postal address on the map, find directions between 2 points, locate yourself on the map (e.g. based on your IP address) and lot more. These services are consumed by the code-behind running on the website and are invisible to the user. But a simple network packet watcher shows the following:

Find Address:
http://local.live.com/search.ashx?b=[address]

Locate Me:
http://local.live.com/WiFiIPService/locate.ashx

Find Directions:
http://local.live.com/directions.ashx?start=[address1]&end=[address2]

The responses are in form of C# code. E.g. when searching for an address, you can expect:
/*1.3.0515*/
SetViewport(32.9500497939651,-117.086802489646,32.9307362060349,-117.117483510354);
VE_Scratchpad.AddLocation('12278 Scripps Summit Dr, San Diego, CA 92131-3697, United States', 32.940393, -117.102143, '');

Python Solution

Of course, you can use all the frameworks available to achieve what we are set out to do. Our solution is quite elegant as you will see in some time. We make use of the fact that C# syntax resembles that of Python when it comes to arrays. If we take out comments, function names, and class names, we have tuple of arrays of tuples (see the example above).

The solution looks as follows:

# If needed set the proxy address
import os
os.environ['http_proxy'] = 'http://[proxy_host]:[proxy_port]/'

# Definations for eval()
false = 0
true = 1

import urllib

# To remove the comments and non-py elements in the response buffer
import re

comment_pat = re.compile('(/\*.*?\*/)(".*?")', re.S)
def subfunc(match):
   if match.group(2):
      return match.group(2)
   else:
      return ''

nonpy_pat = re.compile('(new [\w\s,.]+)(".*?")', re.S)

def getAddress(address):
   encaddress = urllib.urlencode({'b': address})
   url = 'http://local.live.com/search.ashx?%s' % encaddress
   sock = urllib.urlopen(url)
   resp = comment_pat.sub(subfunc, sock.read())
   sock.close()
   lines = resp.split(';')
   for instruction in lines:
      if instruction.find('VE_Scratchpad.AddLocation') > -1:
         break
   locationstr = instruction.replace('VE_Scratchpad.AddLocation', '')
   (infostr, lat, lon, pad) = eval(locationstr)
   return (infostr, lat, lon)

def locateMe():
   url = 'http://local.live.com/WiFiIPService/locate.ashx'
   sock = urllib.urlopen(url)
   resp = comment_pat.sub(subfunc, sock.read())
   sock.close()
   locationstr = resp.replace('SetAutoLocateViewport', '')[:-1]
   (lat, lon, zoomradius, unknown, message) = eval(locationstr)
   return (lat, lon, zoomradius)

if __name__ == "__main__":
   result = getAddress('12278 scripps summit dr san diego ca')
   print result
   result = locateMe()
   print result

Summary

This example shows how simply Python eval() allows parsing complex text. I have left few issues to be addressed, such as ambigous address, etc. Moreover, you can explore the directions as well. Hope you enjoy!

Text To Speech Using Python for S60

Recently, I came across a very interesting post to symbianexample.com by Artem http://symbianexample.com/texttospeech. This article publishes a 'hidden' S60 feature - An MMF plugin that is able to play text!! This plugin apparently exists only for S60 2.8+ phones. I have not been able to confirm that.

Imagine your program synthesizing any text... Who would not like their programs to speak intelligently to the user? I for one would for sure!!

Now, I am a die-hard PyS60 fan... I jumped on this opportunity to enable PyS60 scripts with TTS. The result was a Python extension, _ttsplayer.pyd.

_ttsplayer.pyd is a C++ Python extension DLL. This DLL wraps the CTtsPlayer class provided by Artem in his original post. The extension provides only one method:

playtext(text) # text is unicode

The usage of this module is very simple:

import ttsplayer
ttsplayer.playtext(u'Hello Mr. Anderson')

As per Artem's article, this MMF plugin synthesizes the speech using Speaker INDependent (SIND) framework. As you will find out when you start using this module, the quality of the generated speech is poor. Although, in more recent S60 devices (such as N75 - cool phone! oh sorry, cool computer!) we see evidence of a High Quality TTS framework. I am not sure whether SIND framework will eventually merge with this High Quality TTS framework. I can only hope that it does!

The package is published here: http://www2.cs.uh.edu/~nikhilv/PyS60/ttsplayer.zip

The package contains the following components:

• Source package - CTtsPlayer class and Python module wrapper, MMP file, PKG file, bld.inf file. These file allow you to create PYD extension module using your favorite S60 SDK.
• Python module - This module wraps the C++ extension DLL loading etc.
• Example python script - To show usage in a PyS60 script.
• Unsigned SIS file S60 3.0 - You can sign this SIS using your favorite certificate and install it if you do not want to go through the trouble of compilation.

I hope you like this little extension module. There are countless uses for this module. One can imagine a PyS60 script that gets directions from one location to another (see Rantings of a Snake Charmer IV) and dictates the direction steps to the user using TTS!

Thanks to Artem for finding this hidden feature in S60!