SkyDrive and Windows 8.1

This blog post is part of a series of guest posts we’re publishing this week from different people in groups across Microsoft who helped us build Windows 8.1. – Brandon

My name is Adam Czeisler and I am the Development Manager for one half of the SkyDrive team in Windows Services. Our team, called SkyDrive Devices & Roaming is responsible for building the client software that is the expression of SkyDrive on all devices, including Windows tablets and PCs. The other half of the team, called SkyDrive Cloud Services, is responsible for building the web service that powers these experiences. They also build the SkyDrive website at http://www.skydrive.com.

In the past two years leading up to our work in Windows 8.1, our team built and released several SkyDrive apps on all major platforms. We released apps for Windows Phone, Xbox, iOS, Android, and a Store app for Windows 8. We also released sync clients that perform two-way synchronization of files to the SkyDrive service on Windows 8/7/Vista as well as on Mac OSX. This was a great learning experience for our team. Over the course of several updates to these apps we have learned how people want to access their files from their devices, what features matter most on each device, and also of course the unique set of development challenges and opportunities these different platforms offer. All of this work prepared us for this past year when we set out, together with experts in the Windows team to bring the best of all SkyDrive experiences to Windows.

Before Windows 8.1, our engineering designs fell into two major categories, apps and sync clients. The apps communicate with the SkyDrive service using a stateless JSON API in a design pattern similar to a website. In fact the SkyDrive apps browse the hierarchy of files and folders in SkyDrive via the same JSON API used by the SkyDrive website. This design is helpful because we strive to make the feature set be consistent across all endpoints. The image below shows the SkyDrive app on Xbox 360, built on the JSON API (watch the video).

In contrast the sync clients used a proprietary sync protocol directly with a lower layer of the SkyDrive service stack. This sync protocol is generally limited to supporting “CRUD” operations on files (Create, Read, Update and Delete).

There are pros and cons to each engineering approach. The design used by the apps is good because it allows them to have a small footprint on the device. The apps are able to browse very large data sets relatively quickly because the JSON API lets them ask for any view of the tree independently. There is no need to download the full body of the files in order to browse the collection of files. Caching items like thumbnails, tree structure, etc. can be done in a dynamic way and the cache can be managed independently of the core logic. Because of this design choice, these apps are well suited to small storage devices such as phones or tablets.

On the other hand, this approach has two major drawbacks, 1) offline operations can’t be done on the files and 2) since most operations are performed directly against the service, the performance is limited to the speed of the network rather than the speed of the local storage on the device.

The sync clients have the opposite characteristics. On the positive side, they naturally have full CRUD operations available both offline and online. And when the apps are interacting with the files, they are doing this against the local storage in the device so the speed is often one or two orders of magnitude higher than when accessing this content from the network.

But the sync approach has a major drawback – in order to browse or interact with any of the files on SkyDrive they all have to be fully downloaded on the device. This solution just isn’t acceptable on a device with limited storage because the slice of data that would fit on the devices is typically too small to be useful. Also the raw time to download and sometimes even the cost of downloading this data can be potentially very large.

In Windows 8 we tackled these problems by releasing both a SkyDrive modern app and a SkyDrive sync client. We had all the pros and all the cons of both engineering designs. In addition the app and the sync client couldn’t communicate with each other which led to some temporary inconsistencies that could be confusing to people who used both. When we started planning for Windows 8.1 we knew we wanted to do something better. Could we have the best of both worlds?

To do it, we knew we would need a solution that let people browse the full SkyDrive namespace in Explorer and in the SkyDrive modern app. We also wanted them to be able to perform CRUD operations against this dataset, even when offline. But we didn’t want to have to download the entire SkyDrive to the local device to accomplish this. Could this be done? We weren’t sure, so we joined forces with the experts in the Windows team to see if we could build a hybrid solution that gave us everything we wanted.

Our team has deep expertise in file sync technology. Several members have released multiple file sync products over the years; some have even worked on similar technology in Microsoft Research. But we didn’t have deep experience with the Windows Shell architecture, codebase or APIs. Clearly we would need this in order to deliver the best integration of SkyDrive into Windows possible. Luckily the team in Windows with the most expertise in this area was formed to help us design and implement the integration of SkyDrive into Windows. This team, called Cloud Experience, was our main partner in Windows during the year. After discussing the problem above they came up with a design for a new platform in Windows called “smart files”. Smart files are files that contain only metadata and no “body”. In addition they have a special behavior where applications or components on the system can register to be the provider for the contents of the file and when a caller reads the body of the file via certain APIs, these providers are invoked to satisfy this request. By hooking the SkyDrive sync engine into this pipeline as a provider for files in the SkyDrive namespace, we could seamlessly provide access to data stored in the service without forcing all the data to exist on the device before accessing it. That sounded very promising, so this was the design we chose.

Now we had a solution that reduced the space needed on the device to access the full SkyDrive namespace while still allowing full CRUD operations to be performed both online and offline via Explorer in the Desktop. But we also needed the SkyDrive modern app to be able to view and modify these files. So we rebuilt the SkyDrive app on top of the Windows Shell APIs, which are built on top of the core file system with additional behavior such as mentioned above for smart files. This meant that the SkyDrive app could now have offline access as well as have a spectrum of performance from network to local speeds depending on whether or not the files were downloaded to the local machine. Awesome. Now we just had to build it!

Working directly in Windows this past year to integrate SkyDrive deeply into the operating system was a fantastic journey for our team. We learned firsthand about the tremendous breadth and depth of scenarios that the Windows Shell supports by integrating both as a provider in the sync engine and a consumer in the modern app. Through the process we gained a deep respect for the people who built that system over the past decades in such a layered and extensible way that allowed it to be augmented with a service-backed file system in such a short time with relatively few architectural changes.

For example, the image below shows the existing file copy dialog in Explorer effortlessly lighting up a scenario where the sync engine is automatically invoked to download several images as they are being copied to the Desktop from a folder in SkyDrive. Stats like percent complete, throughput and remaining bytes are shown to the user. You can even pause or abort the operation through this user interface and because of the deep integration with the Windows Shell, the sync engine will respond to these commands directly.

Another highlight was working closely with the group in Windows dedicated to performance analysis and improvements. With their help and guidance we made several key changes to minimize the impact of the sync engine on the system. For example, for more efficient monitoring of changes in the file system we integrated into a built-in Windows component called the Change Tracker. This allowed us to eliminate nearly all direct scans of the local files in SkyDrive, which in the standalone version of the sync client were necessary to ensure we didn’t miss any changes while the process wasn’t running. With help from the Cloud Experience team we cut the memory footprint in half by leveraging the search indexer to store file transfer status. We also did work to protect the CPU idle state by creating a special dormant mode in the sync engine so that when there was no outstanding work to be done we could “go to sleep” and have nearly zero impact on the rest of the system.

Switching gears to the modern SkyDrive app, I want to highlight another great partnership we had this year with the team that builds the apps and controls related to photo and image processing in Windows, called the Apps for Creative Expression team. In Windows 8 they built the modern Photos app that aggregated SkyDrive photos as well as photos from your local device. The app had a very nice interface for viewing photos but both of our teams were frustrated that customers could hit “dead ends” in both the SkyDrive app and the Photos app when interacting with images on SkyDrive. For example you couldn’t move, rename or edit photos in SkyDrive from the Photos app and in the SkyDrive app the viewer was not nearly as rich as the one in the Photos app. We really wanted to solve these problems together in Windows 8.1. The way we did this was to build both the Photos app and the SkyDrive app using nearly all of the same source code and binaries, with some small runtime differences in default views and theming. In addition to working directly in the core apps, the Apps for Creative Expression team designed and built the beautiful photo viewer/editor control that is used in both apps.

For a great example of the types of scenarios that are enabled by the integration of the SkyDrive sync engine into Windows and the sharing of code between the Photos and SkyDrive apps, try browsing full-screen through some photos in SkyDrive using either the modern Photos or SkyDrive app, swipe to bring up the app bar and then choose ‘Edit’. The full file will be automatically downloaded from the SkyDrive service and the photo editor control will present you with a variety of useful and powerful touch-enabled tools to enhance your photos. If you save the changes, they will be automatically uploaded to SkyDrive via the sync engine. No matter whether you interact with your SkyDrive photos via the Photos app or the SkyDrive app you will always have the full set of appropriate features available to you.

One of our philosophies as a team has been that clients are an extension of the service. We think of them as simply another endpoint that expresses the service to the customer, much like a browser. This means that all clients should have roughly the same feature set and capabilities. They will be of course tailored to the device but the customer experience should be familiar in every expression of the service.

Besides making sure the experiences are consistent and familiar on every endpoint, we also need to ensure that the operations on all of our endpoints are reliable and robust. From an engineering point of view, in order for any service to achieve the kind of reliability that our customers expect there are three capabilities required. Engineers need to be able to 1) measure reliability, 2) diagnose issues and 3) deploy fixes. This process must be repeatable in a tight iterative loop. As the cycle continues, quality rises.

When we designed our original standalone sync clients, we realized we had gaps in all three areas above that we needed to fill. So over the years we have iterated on solutions to increase our capabilities in measurement, diagnostics and deployment. We created a telemetry system that measures sync convergence across all of our clients on a daily basis. We built a pipeline to gather anonymous data that lets the team diagnose issues that are flagged in the telemetry system. And finally we built an updating mechanism for the Windows sync client through which we could deploy fixes to all endpoints with no user-interaction. When we integrated the SkyDrive sync engine into Windows 8.1 we were able to keep all of these capabilities by either bringing them into the operating system or using other pre-existing mechanisms. Because of this, we are able to continue our work to monitor and improve the experience of our customers using the product each day.

I am extremely excited for the GA release of Windows 8.1. Knowing that millions of customers will be using SkyDrive in more useful ways than they have ever been able to do before, with many of them using a cloud file service for the first time ever, is humbling, exciting and inspiring all at the same time. As a team we will face new challenges as we scale up to meet the demands of the millions of new customers we hope to gain. We will learn and we will improve continuously. Our goal is to delight every person who uses SkyDrive, every minute and every day they use it, so that they are customers for life.

And by the way, there are tons of other great features in Windows 8.1 besides the SkyDrive integration. One of my favorites is the new small-size tiles on the Start screen. I can get all of my most important apps on one screen now, which is always alive with color and activity. Here is my own Start screen running Windows 8.1. I hope you enjoy it as much as I do!

(This is Adam’s Start screen in Windows 8.1!)

Adam Czeisler
Development Manager
SkyDrive

View the original article here