Logo courtesy of the W3C
Remember the furor that erupted on the web and the wmlprogramming Yahoo group over transcoders? Specifically, the ones that Vodafone UK and other carriers implemented with the goal of making non-mobile web pages more usable on mobile handsets? The issue was that these services had the (hopefully unintended) consequence of degrading or completely breaking numerous mobile web sites and services. Ring tone and game downloads no longer worked, some mobile web sites didn’t load or displayed malformed content. Sites that relied on being able to detect mobile browsers and deliver optimized content no longer worked as designed because the transcoders removed or changed the HTTP headers that identified the browser, user’s language settings or preferred content types.
Developers and content providers upset with the disruptions to their sites and business models rallied around Luca Passani, co-creator of the WURFL mobile device repository to formulate a document, “Rules for Responsible Reformatting: A Developer Manifesto” which suggested guidelines that transcoders should follow to avoid disrupting existing mobile sites and services. The “Manifesto” was generally well received in the mobile web development community and was also endorsed by three of the major transcoder vendors InfoGin, Volantis and Openwave. Openwave provides the OpenWeb transcoder used by Sprint.
The furor had died down, but now it’s been rekindled by the W3C’s release of a draft of its “Content Transformation Guidelines” recommendations, which like the Manifesto, describes best practices for mobile transcoders. The Guidelines are open for public comment until Sept 16. Anyone can comment by sending an e-mail to [email protected] (note that your e-mail address will be published on the web – I recommend that you use a disposable e-mail address!). All comments can be viewed at lists.w3.org/Archives/Public/public-bpwg-comments
Their are some clear differences between the two documents. The W3C Guidelines doesn’t go as far as the Manifesto in limiting what is acceptable behavior by transcoders, particularly with regard to changing HTTP headers. The Guidelines also place a larger part of the responsibility for interoperability on mobile web developers and publishers rather than transcoder vendors. All of which understandable given that the members of the W3C, including the Mobile Web Best Practices Working Group, are primarily representatives of commercial organizations including mobile carriers, transcoder vendors and large web publishers while the Manifesto comes out of the mobile web developer community.
There are lively discussions about the W3C Guidelines going on both through the comment process and on the wmlprogramming Yahoo group. Luca and others have proposed changes to the Guidelines that would bring it closer to the Manifesto. It remains to be seen if these suggestions will make it into the final document.
In reading both the Guidelines and the Manifesto, I noticed that neither said anything about allowing end-users to opt out of transcoding. I think it is essential that all transcoders offer ordinary users a way to disable the transcoder, either temporarily for the current site or page, or globally for all sites.
In spite of everyone’s best intentions, transcoders will sometimes break sites that would otherwise be usable on a given handset. If you have ever used Skweezer, Mowser or the transcoders that Google, Yahoo, AOL and Microsoft Live use to reformat web content returned from search queries, I’m sure you’ve noticed that some sites come up blank, broken into one line pages or missing essential parts and functionality when transcoded. Fortunately those services are opt-in, no one is forced to use them, and except for Yahoo and AOL, they offer links to the original version. With the carrier transcoders there is often no way to opt out, leaving users no alternative if the transcoded version of one of their favorite sites is broken.
Secure sites pose a special problem for transcoders as they can not be transcoded without decrypting the the secure content, potentially exposing users to “man in the middle” attacks. Users need to be able to opt out of transcoding when doing online banking or shopping when account information or credit car numbers are being passed.
I’ve left a comment with the W3C arguing for making user opt-out a requirement for all transcoding proxies, I won’t bore you by reprinting it here as it essentially restates the above. You can read it in the W3G BPWG archives.
The issue of transcoding, is a very important one for the future of the web on mobile devices. Take a few minutes to skim the Guidelines and the discussions at the W3C and wmlprogramming and if sufficiently motivated, email comments on the W3C Guidelines to [email protected] by the Sept. 16th deadline.
Related:
Vodafone’s Heavy-Handed Transcoder
How Web to Mobile Transcoding Should Work
OpenWeb and InfoGin Adopt the Developer Manifesto!
karl dubost,
> What do you mean by substantial developer?
I mean that a substantial percentage, say a third, of voting W3C members ought to be software developers, including the independent individual developers who I believe are responsible for many if not most of the current mobile web sites.
W3C membership to too expensive for most open source and independent developers. I believe, that in an ideal world, membership in standards bodies should be based on technical qualifications and willingness to contribute rather than money. Obviously the elimination of membership dues would create a funding problem for the W3C. I don’t have a solution to that unfortunately :)
For groups within the W3C working on standards areas that directly impact end users (like transcoding) I’d like to see a similar percentage of knowledgeable user representatives.
When I look at the current W3C membership I see mainly employees of large US and European firms, many of which are vendors of products that are affected by W3C standards. Vendors should certainly have a say in the standards process but I feel that there needs to be a more balanced representation that includes all stakeholders.
The MW BPWG, is determining standards for transcoding proxies. This standards will affect 4 players, transcoding product vendors, mobile web developers, mobile content distributors and end-users. Yet the of the nine names listed as contributors to the guidelines I see the names of representatives of six transcoder vendors or providers, a carrier that has been a leader in deploying some particularly disruptive transcoders, another large carrier. a hardware vendor, and a W3C staff member. No end user representatives and no one to speak for the needs of independent mobile developers and content providers.
> Specifications are going through a cycle of publication/comments until last call
> phase is reached. This is an open process as everyone can comment and get an
> answer. Usually developers have a lot of weight in the discussion when they can
> show that certain things don’t work in implementations.
> The Candidate Recommendation is here for that, demonstrating implementability > of features.
I applaud the level of openness in the process but don’t think it goes far enough. While anyone can comment, only members can vote to approve. Given the concentration of transcoding vendors on the MW BPWG I fear that the resulting standard will be biased towards the needs of vendors rather than the creators and consumers of mobile web content.
>What would you suggest that will lead to better implementability and
>interoperability?
I feel that the burden of not breaking existing mobile sites and services should fall on the handful of vendors not the hundreds of thousands of existing mobile sites and services. To achieve this,
1) With very few exceptions, transcoding proxies should not change or modify request headers, as doing so adversely effects current mobile services. The only case where a transcoder should be allowed to change headers is if a user explicitly requests that the transcoder impersonate a PC browser.
2) Transcoders should not touch HTTPS traffic as doing so degrades security.
3) Transcoding proxies must allow end users to disable them, both globally and on a site by site or page by page basis.
@Karl
Hi Karl,
> Specifications are going through a cycle of publication/comments
> until last call phase is reached. This is an open process as everyone
> can comment and get an answer. Usually developers have a lot
> of weight in the discussion when they can show that certain things
> don’t work in implementations.
Having being involved with W3C in the past (with not too good results, I admit), I think that what you write is technically true, yet not good enough to deliver the best solutions in the interest of the industry and the progress of technology (not too mention the biblical time spans which are needed by W3C to produce anything, but I digress…).
Here comes a word that you probably heard many times “consensus”. This is what W3C is all about. Looking for consensus. Yet, this is the killer. This is what makes everything impossible in W3C. It has almost become an obsession for me. I think I got this condition where one mentions “consensus” and I start to shoot from the hips.
W3C works like this: a few companies (who have paid the W3C ticket) start discussing about a recommendation. Discussion goes over for months and months, and no agreements is in sight because everyone is at the same level and everyone has the same veto power over the ideas of others, no matter how good or bad. When I was in the BPWG, not even the objective of the group was clear. But despair not. After a while, a miracle starts to happen. The editor starts to speak a language that resembles English, but is no longer English. It’s “consensusese”. It’s a special lingo that only participants in the Working Group understand. This language has a magical feature: everyone of the WG partecipants will read what they want to read in documents authored with the magical consensusese language. That’s it. Magically, one fine day you have it: consensus. Everyone in the working group agrees. It is now time to introduce developers and whoever else inspires to comment on the document.
The first problem of course is that the great majority of people don’t speak consensusese. Since the language does not make sense to them (and they probably have more interesting things to do in their lives), they abandon the task.
There are some survivors though, who, thanks to an incredible mix of motivation, coffee and, I imagine, lack of a normal life, will decipher it and make some disruptive comments which bring out the inconsistencies and (in the case of CT) the insanity of the precious consensual manuscript.
Unfortunately, this is too late. The WG has used way too many man years to reach consensus and those comments are typically discarded with a motivation or another. But that’s not it. The comments are used by the WG to build the anti-bodies for those who will dare to criticize the recommendation in the future (after all, 5 years of work and an open review process have taken place!).
In short, I think W3C should review its process and find way to create specs that are clearer than what we have seen with BPWG, Mobile OK and CTG. Abandoning this crazy notion of consensus at all costs would be a great start.
Luca
@Dennis
> However I do think it is acceptable to change the UA in
> one specific case: when a user explicitly requests user-agent
> spoofing because they need to access a feature of a site that
> is blocked from mobile devices
Good point, Dennis. I’ll need to take a short step back. Over the past year, to properly defend developers, the barricades have been built over the following concept: “It’s the money of the content owner that is at stake => content owners have a right to decide how their content is rendered without interference. Full stop”. IMO this is really important, because it is a solid reason to tell carriers to keep their hands off content they have not paid for.
Because of this, I do not particularly want to take the role of user-champion when talking about transcoders. I admit that, if taken to the extreme, the concept above would mean that also things like OperaMini, Skweezer and widgets of different kinds are not OK. Taken even further, this might also imply that users should not install those pop-up blockers (but that’s probably going a bit too far ;)
Anyway, I am happy that someone (who has not economic interest in transcoders nor is connected to one of the offending companies in any way) takes the defense of the user. I understand your point that opt-in transcoders can be OK in some circumstances. In fact, I went as far as suggesting that CTG is split in two: opt-in transcoders and “you-cannot-get-away-from-it-unless-you-are-a-hacker” transcoders. This would have simplified things a lot, IMO. Opt-in transcoders could get away with a large amount of hacking, while the others wouldn’t. Easy, isn’t it? well, not so easy according to W3C, which would not go for this suggestion. This is not surprising after all. CTG was created by Novarra, Vodafone, Google and ATT (among few others) and those company are not paying W3C to do something that does not help in their ultimate objective: get the W3C stamp on a hack (transcoding) which is part of a strategy to make the mobile internet an extra toy in carriers’ backyards.
Think about it: transcoding is a huge hack from the beginning to the end. Why would they need a W3C spec if it wasn’t just a matter of buying some “moral” justification for the indefensible?
In short, transcoders can do all the hacks they want, but they MUST not pollute everyone’s HTTP.
Luca
Pingback: El cuento de nunca acabar: Transcoders « Blog de Francisco Velázquez
Hi Dennis,
You said:
>I really wish standards bodies like the W3C were more egalitarian. There needs to be substantial developer and user representation. Web standards are too important to be left to vendors who sometimes seem to be using the standards process to gain or maintain a competitive advantage.
What do you mean by substantial developer?
Specifications are going through a cycle of publication/comments until last call phase is reached. This is an open process as everyone can comment and get an answer. Usually developers have a lot of weight in the discussion when they can show that certain things don’t work in implementations.
The Candidate Recommendation is here for that, demonstrating implementability of features.
What would you suggest that will lead to better implementability and interoperability?
Thanks :)
Thanks for the comment Luca,
I agree that not modifying the Manifesto now that vendors have signed is the right thing to do.
Regarding not changing headers, especially the User-Agent. I consider it extremely important that transcoders not modify headers. This is necessary to protect existing mobile services, especially as you say, the “little guys”.
However I do think it is acceptable to change the UA in one specific case: when a user explicitly requests user-agent spoofing because they need to access a feature of a site that is blocked from mobile devices.
For example, Friendster.com has started to redirect all mobile user-agents to a very limited mobile site that is missing essential features such as editing profiles, uploading photos, and viewing forums. This happens even if the mobile browser is S60WebKit, Netfront or Opera – all of where quite usable with the full Friendster.com site before the forced redirection started.
With regards to HTTPS, you are absolutely right, for the W3C to allow transcoders to decrypt secure transmissions is just crazy. That completely destroys the integrity of secure transactions. The banking industry, in particular, won’t accept that. I fear the banks’ reaction will be to block all mobile devices from any sort of online banking.
I really wish standards bodies like the W3C were more egalitarian. There needs to be substantial developer and user representation. Web standards are too important to be left to vendors who sometimes seem to be using the standards process to gain or maintain a competitive advantage.
Hi Dennis, thank you for bringing attention to the W3C activity. I couldn’t agree more with what you wrote: “The issue of transcoding, is a very important one for the future of the web on mobile devices.”
While I am here, I would like to address some of the points you correctly raise in your document.
> In reading both the Guidelines and the Manifesto, I noticed
> that neither said anything about allowing end-users to
> opt out of transcoding.
this is correct. If I was to go for version 1.1, this would certainly be submitted for discussion to the community of developers on WMLprogramming and would probably find its way into the new version. There are a few other rules in line for the new version (some of which were already implied by the Manifesto, but they seem to be disregarded in practice):
– When a mobile site is detected, the transcoders should get completely and totally out of the way.
– Operator navigation bars (be it headers or footers) must not be injected (it’s not your content, so hands off please)
– HTTPS is sacred. If a content owner has gone all the way and implemented HTTPS, it means they want secure transmission and they do not expect anyone to mess with it.
Now you may wonder why I have not created a new version already. Well, the reason is that three vendors already signed, and changing the document does not authorize me to transfer the signature to the new Manifesto (particularly after, as you say, things had calmed down a bit and operators had become more sensitive to developer requests).
In this context, the W3C CT Guidelines (CTG) are a bit disappointing. It is true that CTG and the Manifesto ended up overlapping to a large extent, but in my opinion, too many basic things are not properly handled by the CTG. As you know, the NUMERO-UNO rule in the Manifesto is “Do not change the User-Agent string no matter what”. Simple, unamiguous and effective. If you look at CTG, it does have the rule (4.1.5) “Other than to comply with transparent HTTP operation, proxies should not modify request headers…”. Alas, it also has a nasty caveat:
“..unless the user has specifically requested a restructured desktop experience;”
Now, if you look at Novarra’s and VodaUK’s positioning when they launched their transcoder last year, they were presenting the service as Internet on the mobile phone. This means that anyone who wants to hack HTTP requests the way they want would just need to claim that they are implementing “full web on a mobile phone” to be abusive and walk away with a clean record.
Another aspect that irritates me about the CTG is the fact that they simply adopted the proprietary conventions invented by novarra (x-device-headers). Apart from the fact that it is weird to justify that they went all the way to specify how to hack headers one second after they stated that headers should not be hacked, I think it does not make W3C any honor to simply endorse without any change what one of the vendors at the table has done unilaterally and among a lot of criticism.
Finally, CTG’s position wrt HTTPS is a crying shame. I think an institution like W3C should either stay silent on transcoders and HTTPS (assuming that,as a transcoder, you are not supposed to mess with HTTPS), or, if they do take a stance, they should clearly state that messing with HTTPS is a big no-no. Unfortunately, none of this is happening. If you look at 4.3.6.2, CTG accepts the idea that HTTPS content may be re-written as long as users are advised of “security implications”. Apart from the fact that what advising the user of security implications means is not explained anywhere, my opinion is that nobody, not even users, should have the right to opt for lesser security than the content provider has decided to establish by choosing HTTPS. The W3C knows this, in fact, CTG reads:
“For clarity it is emphasized that it is not possible for a transforming proxy to transform content accessed via an HTTPS link without breaking end to end security.”
To me, this is cowardice. W3C should be braver than this and defend everyone’s HTTP, rather than being totally at the mercy of those who pay to check to sit at their table. All considered, I think this points to the core of the problem. Transcoders can be very evil because they pollute HTTP, which is the air that the mobile ecosystem breaths.
While the big guys will always find a way out of transcoder issues, it’s the small guys that get screwed and, not coincidentally, it was the small guys who get really mad about transcoders.
If you look at who created CT, you will not see anyone representing developers. Can good guidelines come out of that group? Hardly.
W3C is now requesting comments and this is good. What remains to be seen is whether those comments (and there are many of them) will be heard and adopted, OR if they will simply be used by W3C to build the anti-bodies and reject them with ready-made excuses. Time will tell.
Thank you
Luca Passani
Editor of the Manifesto for Responsible Reformatting
Dennis, I saw your e-mails only after my post here. I think you stand correct in your interpretation of MAY and it seems to me like Jo confirmed that user choice is very important.
Hopefully the document will be updated to reflect this desire.
Thanks for your comment, Andrea.
I’ve added links to the W3C document as you suggested.
Regarding section 4.1.5.3, I don’t think it really requires a transcoder to offer users opt-out from transcoding. It sys:
“Proxies may offer users an option to choose to view a restructured experience even when a Web site offers a choice of user experience.”
I take this to mean that the proxy (transcoder) MAY allow the use to view a restructured (transcoded) version of the site even if the site offers a mobile version. In other words the user can opt-in to transcoding of mobile sites.
That’s good as far as it goes but it doesn’t go far enough. It says nothing about the opt-out case of allowing the user to choose to view an un-transcoded version when if site does not offer a mobile one. In terms of not breaking sites and content delivery mechanisms opt-out is far more important than opt-in.
I’m also uncomfortable with the use of the word “may”. I’m not very experienced at interpreting standards documents but to me “may” seems to allow the implementer to exclude this feature and still be able to claim to be compliant.
Jo Rabin posted a reply to my comment on the BPWG list where he points out other references to user opt-in/opt-out in the document. Please see my reply to Jo’s comment regarding those points.
Hi Dennis,
I think it is appropriate to also provide a link to the latest draft of the W3C document. So here it is, http://www.w3.org/TR/2008/WD-ct-guidelines-20080801/.
In response to your complaint about giving the user a choice, I think chapter 4.1.5.3 does exactly that ( see http://www.w3.org/TR/2008/WD-ct-guidelines-20080801/#sec-user-selection ).
If I had to give a high level overview, I would say that the manifesto is about telling to transcoding proxies how to detect a mobile-specific site (and get out of the way) as opposed to the W3C guidelines that describe how users, proxies and mobile sites can live together hopefully happily. A lot of the text in the W3C doc is in fact about interaction between different applications. Reading from the Abstract, “This document provides guidance to content transformation proxies and content providers as to how inter-work when delivering Web content.”.