Enter the Tatrix

Being an Outlook addict, I’m always pleased when I see an “Add to Outlook” or “Download as ICS” button on a website. Everyone from Virgin Blue to Facebook do it.

Unfortunately there was no such file for Remix so instead of just loading it into my private calendar, I thought I’d make the ICS feed.

Here they are:

http://tatham.oddie.com.au/files/RemixSydney.ics

http://tatham.oddie.com.au/files/RemixMelbourne.ics

See you tomorrow / Thursday!

I know where I am. My phone knows where it is. Why doesn’t the webpage know where I am?

Think about these scenarios which will become more and more prominent as “the mobile web” starts to prevail:

1. I visit a cinema website on my mobile. Rather than having to choose my cinema first, the website already knows which suburb I’m in so it defaults to the nearest cinema first.
2. I need a taxi. I don’t know where I am, but my phone does. I want to be able to book a taxi and have the website discover my location automatically.

The key idea that I’m exploring here is the ability for a webpage to access this information through the standard browser interface.

I have a plan for making this a reality.

Windows Mobile has already taken a step towards baking location awareness into the OS with their GPS Intermediate Driver. The idea is that the operating system handles communication with the GPS unit, including all the various protocols. Applications then have a unified API for accessing GPS data. This proxy effect also facilitates non-exclusive access to the GPS.

But this doesn’t go far enough. Even with this unified API, very few applications are actually location aware. More importantly, I don’t want to have to download and install a piece of software on my device just to be able to see movie times. It’s just not going to happen.

We’ve also been making the assumption that location data comes from a GPS. Enter GeoPriv.

With the continuing rollout of VOIP, there are obvious challenges about the loss of location awareness. The current analog network makes call tracing relatively easy. It’s a fixed line of copper and the phone company knows where it terminates. This is a legal requirement for emergency call routing, as well as being immensely useful for scenarios such as a national number auto-routing to your nearest local store. Both of these scenarios become immensely difficult when you can’t even rely on there being a physical phone anymore - a piece of software with a network connection is now a fully fledged communication device that needs to support these scenarios somehow.

There’s an IETF working group tasked to solve this exact problem. The privacy impacts of sharing location data are so important that it’s in the name. They are the “Geographic Location/Privacy working group”, or “GeoPriv“. The best part is, they are living in a reality and delivering useful technology - and fast.

There are a number of key concepts they have identified:

• We can’t go jamming a GPS chip in every single device we manufacture. We need to be able to capitalize on the ecosystem surrounding our existing devices to surface the information we already have.
• Privacy is a critical element of any solution expecting wide spread adoption and trust.
• There are two possible types of location you could need:
  • civic location (level, street number, street, suburb, etc)
  • geographic location (latitude, longitude, elevation)

Lets step away from mobile devices briefly and consider the laptop I’m writing this post on. My laptop doesn’t know where it is. Neither does my WiFi router, or my DSL modem. My ISP does though.

At some stage in the future, my modem will start receiving an extra DHCP option. In the same  way that my ISP supplies me with network settings like DNS when I connect, they will also start pushing out the address of their Location Information Server. My DSL modem will then push this setting out across my network. Finally, my laptop will be able to query this service to find out my current civic and/or geographic location. The privacy controls around this are beyond the scope of this post.

By asking the service provider for the information, these same techniques also works for mobile devices, 3G data connections, and all those other wonderful wireless technologies. Cell-based triangulation is already in use by phone companies around the world, including our national carrier here in Australia, however the interfaces are in no way standardized. The Location Information Server (LIS) and the HTTP Enabled Location Delivery protocol (HELD) solve this problem.

Now that our device is capitalising on the network ecosystem, getting it into the browser is the easy part. All that’s left is a thin veneer of JavaScript.

Location awareness is only becoming an increasing demand. I want to start the process of rolling in the JS layer now, so that as the supporting technologies come to fruition, we have the access layer to make them useful.

Inline with the XMLHttpRequest object that we’ve all come to know and love, I’ve started writing a spec for a GeographicLocationProvider object.

With XMLHttpRequest, we can write code like this:

var client = new XMLHttpRequest();
client.onreadystatechange = function()
{
    if(this.readyState == 4 && this.status == 200)
        alert(this.responseXML);
}
client.open("GET", "http://myurl.com/path");
client.send();

I want to be able to write code like this:

var provider = new GeographicLocationProvider();
provider.onreadystatechange = function()
{
    if(this.readyState == 2)
        alert(this.geographic.latitude);
}
provider.start();

Again, usage is conceptually similar to the XMLHttpRequest object:

1. Initialize an instance of the object
2. Subscribe to the state change event
3. Set it free

The potential states are:

• STOPPED. This is the state the the object is initialized in, and the state that it returns to if stop() is called.
• RESOLVING. The object has been started, but not location information is available yet. In this state the browser could be:
  • prompting the user for permission,
  • searching for location sources (like GPS hardware, or an LIS endpoint), or
  • waiting for the location source to initialize (like connecting to satellites, or talking to the LIS)
• TRACKING. A location source has been found and location data is ready to be queried from the provider.
• UNAVAILABLE. No location data is available, and none is likely to become available. The user may have denied a privacy prompt, their security settings may have automatically denied the request, or there may be no location sources available. It is possible for the provider to return to the RESOLVING state if a location source become available later.

In more complex scenarios, the provider can be primed with a specific request to aid in evaluation of privacy policies and selection of location sources. For example, browsers may choose to hand over state-level civic location data without a privacy prompt. This data could also be obtained from an LIS, without needing to boot up a GPS unit. If the webpage requested highly accurate geographic location data, the browser would generally trigger a privacy prompt and boot up the most accurate location source available.

While we’ve now simplified the developer experience, the complexity of the browser implementation has mushroom clouded. How do we reign this in so that it’s attractive and feasible enough for browser implementers? How do we demonstrate value today?

You might have noticed that in my discussion of the JS layer I drifted away from the GeoPriv set of technologies. While any implementation should be harmonious with the concepts developed by the GeoPriv working group, we aren’t dependent upon their technology to start delivering browser-integrated location awareness today.

There are numerous location sources which can be used:

• Statically configured location - with the network fingerprinting technology already in Vista, it would be relatively easy to prompt users for their civic location the first time location data is needed on a particular network.
• GPS Intermediate Driver - already rolled into the Windows Mobile platform.
• Location Information Servers - can be added to the mix later as LIS deployments become prevalent. This is the only one that is GeoPriv dependant.

The civic and geographic schemas have already been delivered by the working group as RFC 4119. There has been an incredible amount of discussion involved in developing a unified schema that can represent civic addresses for anywhere in the world, and this schema should be adopted for consistency. (Do you know the difference between states, regions, provinces, prefectures, counties, parishes, guns, districs, cities, townships, shis, divisions, boroughs, wards, chous and neighbourhoods? They do.)

Who is responsible for delivering this unified location layer?

I keep talking about the browser being responsible for managing all these location sources. Other than the JS layer, all of this infrastructure is client independent, so why don’t we just make the browser a dumb proxy to a unified location service. This service should be a component of the operating system, accessible by software clients (like Skype) and webpages via the browser proxy.

Windows Mobile has already started in the right direction with their GPS Intermediate Driver, however this is only one element of a wider solution.

What do I want?

1. I want to see a “Location” icon in the settings page of my mobile device, the Control Panel of my Vista laptop and the System Preferences panel of my Mac.
2. I want the browsers to expose the GeopgraphicLocationProvider object for JS clients. (The start of this specification is online now already.)
3. I want the browsers to proxy location requests to the operating system store, along with hints like which zone the website is in.
4. I want the operating system store to be extensible, implementing its own provider model which allows 3rd party developers to supply their own location data from software or hardware services as new discovery mechanisms are developed. We shouldn’t have to wait for widespread adoption before sources are surfaced into the store, and this should be open to software developers as well as driver developers.
5. I want the operating system store to be accessible by any application, including 3rd party browsers.

How am I going to make this happen?

Dunno.

Right now, I’m documenting the start of what will hopefully be a fruitful conversation. Participate in the conversation.

At CodeCamp I caught up with ShaneMo quickly to talk about my upcoming Remix session.

I just got an email from Microsoft a few minutes ago about their launch of the MS Press store here in Australia - www.mspress.com.au.

Best of all, for the month of May we all get 50% off! Just use the code MSPLIVE0508 when you checkout.

If you provide some feedback about the site you go in the draw for an XBOX 360.

It was great to see this afternoon’s announcement on the Remix website that Mark Pesce will be taking the top presentation slot.

Mark did the locknote at last year’s Web Directions conference in a Sydney with his “Mob Rules” talk. He discussed some amazing social aspects of the internet and, in a more general sense, the unprecedented communications in poor communities. Over the last decade we’ve gone from more than 50% of the worlds population never having made a phone call, to over half the population now owning a mobile phone. The social impacts of this are quite surprising in ways that people never could have predicted.

I highly recommend watching the Mob Rules talk when you get a chance. You can watch it all online at http://www.webdirections.org/resources/mark-pesce.

He’s an amazing presenter, whos presentation technique reminds me a lot of Al Gore (another speaker I have immense respect for).

I look forward to seeing what he’ll deliver at Remix.

For a comparison of various ASP.NET state management techniques (including cookies) and their inherit advantages and disadvantages, see my previous post - Managing State in ASP.NET - which bag to use and when. This post covers the purely technical side of dealing with cookies in ASP.NET

I’m glad the ASP.NET team gave us access as raw as they did, but it also means that you need to have an understanding of how cookies work before you use them. As much as it might seem, you can’t just jump in and use them straight away.

First of all, you need to understand how they are stored and communicated. Cookies are stored on the user’s local machine, actually on their hard-drive. Every time the browser performs a request, it sends the relevant cookies up with the request. The server can then send cookies back with the response, and these are saved or updated on the client accordingly.

Now you need to remember that cookies are uploaded with every request, so don’t go storing anything too big in there particularly as most users have uplinks which are much slower than their downlinks. With this in mind, you should generally try and just store an id in the cookie and persist the actually data in a database on the server or something like that.

Finally, you only need to send cookies that have changed back with the response, somewhat like a delta. This introduces some intricacies, like “how do you delete a cookie” which I’ll discuss in a sec.

Reading Cookies

This is the easy part. Because cookies are uploaded with every request, you’ll find them in the Request.Cookies bag. You can access cookies by a string key (Request.Cookies["mycookie"]) or enumerate them. Cookies can contain either a single string value, or a dictionary of string values all within the one cookie. You can access theses by Request.Cookies["mycookie"].Value and Request.Cookies["mycookie"].Values["mysubitem"] respectively.

Creating Cookies

The Request.Cookies bag is writeable, but don’t let that deceive you. Adding cookies here isn’t going to help you, because this is just the upload side. To create a new cookie, we need to add it to the Response.Cookies bag. This is the bag that is written back to the client with the page, and thus it plays the role of our “diff”.

The HttpCookie constructor exposes very simple name and value properties:

new HttpCookie(”mycookie”,”myvalue”)

To get some real control over your cookies though, take a look at the properties on the HttpCookie before you call Response.Cookies.Add.

• Domain lets you restrict cookie access to a particular domain.
• Expires sets the absolute expiry date of the cookie as a point in time. You can’t actually create a cookie which lives indefinitely, so if that’s what you’re trying to achieve just set it to DateTime.UtcNow.AddYears(50).
• HttpOnly lets you restrict the cookie to server side access only, and thus prevent locally running JavaScript from seeing it.
• Path lets you restrict cookie access to a particular request path.
• Secure lets you restrict cookie access to HTTPS requests only.

If a cookie is not accessible to a request, it will just not be uploaded with the request and the server will never know it even existed.

Updating Cookies

Any cookie that exists in Response.Cookies will get sent back to the client. The browser will then save each of those cookies, and override any cookies that have the same key and belong to the same domain, path, etc.

This means that to update the value of a cookie, you actually have to create a whole new cookie in the Response.Cookies collection that will then override the original value.

Updating the value of cookies is rare though, because as suggested early you should only be aiming to save an id in the cookie, and ids don’t generally have to change.

Deleting Cookies

As with updating, this is rare, but sometime you gotta do it.

We know that we only have to send back cookies that we want to update, so how do we delete a cookie? Not specifying it in the response just tells the browser it hasn’t changed.

Well, this is fun.

Create a new cookie in the Response.Cookies bag with the same key as the one you are trying to delete, then set the expiry date of this cookie to be in the past. When the client browser attempts to save the cookie it will override what was there with an expired cookie, thus effectively deleting the original cookie.

Remember that your user’s time maybe offset from your servers, so play it safe and set the expiry date to at least DateTime.Now.AddHours(-24);

There’s been some discussion on the Readify mailing lists lately about all the different types of ASP.NET state mechanisms. There didn’t seem to be a good comparison resource online, so I thought it’d be my turn to write one.

Session State

The most commonly used and understood state bag is the good old Session object. Objects you stored here are persisted on the server, and available between requests made by the same user.

• Security isn’t too much of an issue, because the objects are never sent down to the client. You do need to think about session hijacking though.
• Depending on how ASP.NET is configured, the objects could get pushed back to a SQL database or an ASP.NET state server which means they’ll need to be serializable.
• If you’re using the default in-proc storage mode you need to think carefully about the amount of RAM potentially getting used up here.
• You might lose the session on every request if the user has cookies disabled, and you haven’t enabled cookie-less session support, however that’s incredibly rare in this day and age.

Usage is as simple as:

Session["key"] = “yo!”;

Application State

Application state is also very commonly used and understood because it too is a hangover from the ASP days. It is very similar to session state, however it is a single state bag shared by all users across all requests for the life of the application.

• Security isn’t really an issue at all here because once again, the objects are never sent over the wire to the client. With application state, you also don’t have the risk of session hijacking.
• Everything in the bag is shared by everyone, so don’t put anything user specific here.
• Anything you put here will hang around in memory like a bad smell until the application is recycled, or you explicitly remove it so be conscious of what you’re jamming in to memory.

I can’t remember ever seeing a legitimate use of application state in ASP.NET. Generally using Cache is a better solution - as described below, it too is shared across all requests, but it does a very good job of managing its content lifecycle.

I’d love to know why the ASP.NET team included application state, other than to pacify ASP developers during their migration to the platform.

Usage is a simple as:

Application["key"] = “yo!”;

HttpContext / Request State

Next up we have HttpContext.Current.Items. I haven’t come across a good name for this anywhere, so I generally call it “Request State”. I think that name clearly indicates its longevity - that is, only for the length of the request.

It is designed for passing data between HTTP modules and HTTP handlers. In most applications you wouldn’t use this state bag, but its useful to know that it exists. Also, because it doesn’t get persisted anywhere you don’t need to care about serialization at all.

Usage is as simple as:

HttpContext.Current.Items.Add(”key”, “yo!”);

View State

Ah … the old view state option that sends chills down the spine of any semantic web developer who longs for the days when the web worked like the web instead of winforms hacked into HTML. (Don’t worry - ASP.NET MVC lets us return to those glory days!) But enough with my whining …

View state is used to store information in a page between requests. For example, I might pull some data into my page the first time it renders, but when a user triggers a postback I want to be able to reuse this same data.

While it makes life easier for us young drag-n-drop developers, it is a force to be reckoned with carefully.

• View state gets stored into the page, and if you save the wrong content into it you’ll rapidly be in for some big pages. I’ve seen ASP.NET pages with 10KB of HTML and 1.2MB of view state. Have a think about how long that page took to load!
• It’s generally used for controls to be able to remember things between requests, so that they can rebuild themselves after a postback. It’s not very often that I see developers using view state directly, but there are some legitimate reasons for doing so.
• Each control has its own isolated view state bag. Remember that pages and master pages each inherit from Control, so they have their own isolated bags too. View state is meant to support the internal plumbing of a control, and thus if you find that the bags being isolated is an issue for you then it’s a pretty good indicator that you’ve taken the wrong approach with your architecture.
• It can be controlled on a very granular level - right down to enabling or disabling it per control. There’s an EnableViewState property on every server control, every page (in the page directive at the top) every master page (also in the directive at the top), and an application wide setting in web.config. These are all on my default, but the more places you can disable it in your app, the better.

A full explanation of ViewState is beyond the scope of this article, but I highly recommend that every ASP.NET developer read TRULY Understanding ViewState by Dave Reed.

If you want a simpler discussion, be sure to take a look at my previous post - Writing Good ASP.NET Server Controls.

Usage is as simple as:

ViewState["key"] = “yo!”;

Control State

Control state is somewhat similar to view state, except that you can’t turn it off.

The idea here is that some controls need to persist values across requests no matter what (for example, if it’s hard to get the same data a second time ’round).

I’m a bit hesitant about the idea of control state. It was only added in ASP.NET 2.0 and in many ways I wish they hadn’t. Sure, some controls will break completely if you do a postback without view state having being enabled. What if I never expect my page to postback though? Maybe I want to be able to turn it off still. Unfortunately I think this comes from the arrogance that is ASP.NET not trusting the browser to even wipe its own ass … even the most personal of operations must go via a server side event, so you’ll always do a postback - right? Wrong.

If you’re a control developer, please be very very conscious about your usage of control state.

Usage is a bit more complex … you need to override the LoadControlState and SaveControlState methods for your control. MSDN is a good place to find content for this - take a look at their Control State vs. View State Example.

Cache

Cache is cool. As a general rule, it’s what you should be using instead of Application.

Just like application, it’s shared between all requests and all users for the entire life of your application.

What’s cool about Cache is that it actually manages the lifecycle of its contents rather than just letting them linger around in memory for ever ‘n ever. It facilitates this in a number of ways:

• absolute expiry (”forget this entry 20 minutes from now”)
• sliding expiry (”forget this entry if it’s not used for more than 5 minutes”)
• dependencies (”forget this entry when file X changes”)

Even cooler yet, you can:

• Combine all of these great features to have rules like “forget this entry if it’s not used for more than 5 minutes, or if it gets to being more than 20 minutes after we loaded the data, or if the file we loaded it from changes”.
• Handle an event that tells you when something has been invalidated and thus is about to be removed from the cache. This event it is per cache item, so you subscribe to it when you create the item.
• Set priorities per item so that it can groom the lower priority items from memory first, as memory is needed.
• With .NET 2.0, you can point a dependency at SQL so when a particular table is updated the cache automatically gets invalidated. If you’re targeting SQL 2005 it maintains this very intelligently through the SQL Service Broker. For SQL 2000 it does some timestamp polling, which is still pretty efficient but not quite as reactive.

Even with all this functionality, it’s still pathetically simple to use.

Check out the overloads available on Cache.Items.Add();

Profile

I don’t really think of profile as state. It’s like calling your database “state” - it might technically be state, but who actually calls it that?! :p

The idea here is that you can store personalisation data against a user’s profile object in ASP.NET. The built in framework does a nice job of remembering profiles for anonymous users as well as authenticated users, as well as funky things like migrating an anonymous user’s state when they signup, etc.

By default you’d run the SQL script they give you create a few tables, then just point it at a SQL database and let the framework handle the magic.

I don’t like doing this because it stores all of the profile content in serialized binary objects making them totally opaque in SQL and non-queryable. I like the idea of being able to query out data like which theme users prefer most. There’s a legitimate business value in being able to do so, as trivial as it may sound. (If you think it sounds trivial, go read Super Crunchers - Why Thinking-by-Numbers Is The New Way To Be Smart by Ian Ayres.)

This problem is relatively easily resolved by making your own provider. You still get all the syntactic and IDE sugar that comes with ASP.NET Profiles, but you get to take control of the storage.

Cookies

Cookies are how the web handles state, and can often be quite useful to interact with directly from ASP.NET. ASP.NET uses cookies itself to store values like the session ID (used for session state) and authentication tokens. That doesn’t stop us from using the Request.Cookies and Response.Cookies collections ourselves though.

• Security is definitely an issue because cookies are stored on the client, and thus can be very easily read and tampered with (they are nothing more than text files).
• Beware the cookies can often be access from JavaScript too, which means that if you’re hosting 3rd party script then it could steal cookie contents directly on the client = major XSS risk. To avoid this, you can flag your cookies as “HTTP only”.
• They are uploaded to the server with every request, so don’t go sticking anything of substantial size in there. Even on my broadband connection, my uplink is 1/24th the speed of my downlink. Typically you will just store an id or a token in the cookie, and the actual content back on the server.
• Cookies can live for months or even years on a user’s machine (assuming they don’t explicitly clear them) meaning they’re a great way of persisting things like shopping carts between user visits.

I’m glad the ASP.NET team gave us access as raw as they did, but it also means that you need to have an understanding of how cookies work before you use them. As much as it might seem, you can’t just jump in and use them straight away.

For a rather in-depth look at exactly how cookies work, and how to use them in ASP.NET, look at my post: Using cookies in ASP.NET.

Query Strings

The query string is about as simple as you can get for state management. It lets you pass state from one page, to another, even between websites.

I’m sure you’re all familiar with query strings on the end of URLs like ShowProduct.aspx?productId=829.

Usage is as simple as:

string productId = Request.QueryString["productId"];

I hope that’s been a useful comparison for you. If you think of any other ways of storing state in ASP.NET that you think I’ve missed, feel free to comment and I’ll add them to the comparison.

Update 15Apr08: Added cookies and query strings. Hidden form fields still to come.

Continuing my recent focus on ASP.NET MVC, I’ll now be presenting it at the upcoming Remix conference.

Microsoft Remix 2008 is being held in Sydney on the 20th May and Melbourne on the 22nd May.

My talk is:

Developing great applications using ASP.NET MVC and ASP.NET AJAX: Learn how to use ASP.NET MVC to take advantage of the model-view-controller (MVC) pattern in your favourite .NET Framework language for writing business logic in a way that is de-coupled from the views of the data. Then add ASP.NET AJAX for a highly interactive front end.

Get all the details from the Remix site and get your ticket soon. ($199 is very cheap for a full copy of Expression Suite!)

It was the day after April Fools, and I awoke in the morning eager to iron my shirt and get to work.

I was however distracted by the email waiting on my screen.

From: support@mvpaward.com [mailto:support@mvpaward.com]
Sent: Wednesday, 2 April 2008 1:26 AM
To: tatham@oddie.com.au
Subject: [MVP] Congratulations! You have received the Microsoft MVP Award

Dear Tatham Oddie,

Congratulations! We are pleased to present you with the 2008 Microsoft® MVP Award! The MVP Award is our way to say thank you for promoting the spirit of community and improving people’s lives and the industry’s success every day. We appreciate your extraordinary efforts in Windows Live Platform technical communities during the past year. Microsoft will soon send your MVP Award gift package. It is our way to say “thank you for making a difference.” You will receive an e-mail message in the next 10 business days that contains your MVP Award gift package shipping information and your tracking number.

On behalf of everyone at Microsoft, thank you for your contributions to technical communities.

Sincerely,

Roseanne Stamell, your MVP Lead

Yay.

Thanks to all the guys in Microsoft DPE who’ve continued to support me over the years. Frank, Coatesy, Fin, Kordahi and their peers.

As you may have gathered from my last two posts, I’ve now joined the world of Readify. As of last Monday, I am a Senior Consultant with the organisation.

I’ll be working with them 80% of my time (41 weeks out of 52), and using the remaining time to work on my startups like www.tixi.com.au (also only a fortnight old).

I’m looking forward to my time at Readify as an opportunity to get some more corporate experience under my belt, and get access to larger projects that I wouldn’t have been able to before as an independent. It’ll also give me some more opportunities to present and engage with the community, as well as being part of a great group of consultants who I can call on for help at 3am.