Videoskillet: Bringing Cooking TV Into the Kitchen

IS 246 Final Project - VideoSkillet – Milestone 5

VideoSkillet: Bringing Cooking TV into the Kitchen

VideoSkillet Problem Description

Market Potential

Theoretical Interests

Related Work

Use Scenarios

Demonstrated scenario – basic functionality of the prototype

Future scenario – advanced functionality

The First VideoSkillet Interactive Prototype

Prototype scope

Prototype overview

A note about looping

The interaction device

Prototype implementation

Prototype System Requirements

Prototype walk-through - Overview – screen elements......

Recipe overview/skim

Navigation through main menus and sub-menus

Random access to recipe steps

Random access to recipe steps

Substitution of technique

Future Work and Opportunities

Figure 1: Original Concept: Analyzing Content Locally

Figure 2: Revised Concept: Using Production Metadata

Dan Perkel, Krista Gettle, Maria Lawrence, Ryan Shaw, Vam Makam

1 of 17

IS 246 Final Project - VideoSkillet – Milestone 5

VideoSkillet Problem Description

Traditional culinary information sources, such as cookbooks, magazines, and televised cooking shows are limited in their ability to describe various aspects of the cooking process. For example, it is difficult to convey descriptive information for a certain technique or food texture or consistency in text recipes and still photos. Alternatively cooking shows from the FoodNetwork provide rich visual information regarding cooking techniques and procedures. But most cooking shows make heavy use of time ellipsis, significantly abbreviating the duration of particular recipe steps. Since the story time for preparing a recipe is usually much longer than the screen time dedicated to it in a cooking show, it is difficult for a cook to follow along with the chef and use the program as a guide while cooking. Our goal is to develop a multimedia system that leverages and aggregates a variety of culinary information. The VideoSkillet system will associate web-based recipe content with Food Network video content, effectively re-editing the video content to make it more useful for people to learn from and cook with at home. The application will be built on top of a personal video recorder and cable service, allowing cooks at home to access the cooking show content whenever they choose to use it. VideoSkillet will allow random access to culinary content based on a variety of criteria including ingredients, steps, techniques, equipment, and chef.

Market Potential

We predict that the VideoSkillet application has significant market potential. Early adopters of the system will probably be food enthusiasts who currently watch the Food Network. In addition to this core audience we believe that VideoSkillet will also be attractive to the much larger community of cooks who currently rely on text-based recipes. By aligning the video content more closely with the actual processes of cooking and making the video content more retrievable and usable, we think that all cooks will ultimately find value in the system, from novices to professional. See the presentation produced for milestone 4 for demographic information about the Food Network audience and cable television subscribers. Future market analysis will include demographics and potential market penetration for the larger community of cooks and food enthusiasts who already purchase cookbooks, culinary magazines, and cooking equipment, as well as, those who frequent cooking websites such as epicurious.com, and culinarycafe.com.

Theoretical Interests

The primary theoretical challenges presented by this system relate to video content analysis, retrieval and distribution, as well as interaction design. We initially envisioned building a system that models culinary knowledge and vocabulary well enough so that it can parse and annotate video content automatically. What we now realize is that the content itself would need to be produced in a much more homogenous and standardized way to be machine-read and described. This would detract from the instructive and entertainment value of the video content, and would probably not be acceptable to the Food Network producers. So instead we need to research and propose viable production processes and shooting plans that meet both the high entertainment standards required for broadcast culinary programming and the reuse requirements of VideoSkillet. The challenges in interaction design pertain to the platform, i.e. television and set top boxes, and the unique application context of the kitchen and cooking. Some aspects of the system (searching, browsing, meal planning) could be done from any room in the house, but the cooking mode is intrinsically situated in the social and dynamic environment of kitchens and in the processes of food handling.

Related Work

VideoSkillet’s level of interactivity requires that areas of current multimedia research converge to allow users to interact with cooking videos in the kitchen. Specifically, we explored existing work on applications that relate video and web content, recipe metadata frameworks, and interactive video tools that are adaptable for the kitchen.

The Food Network offers two means to customize content: FoodTV.com and Food Network OnDemand Service. The website provides access to recipes, techniques, and limited video content. The ‘Recipe Step-by-Step’ section of the website uses video in a manner similar to VideoSkillet functionality. Users can navigate between steps and review the recipe. Another link offers still shots of the various steps. All of these functions operate independently limiting navigation options. It is not possible to toggle between video steps and text steps, for example. Food Network’s television feed customization option, OnDemand, allows users to access a block of shows televised within the last 4 weeks without restriction. Food Network OnDemand allows users to choose the time that they will view a show. They can also pause rewind and fast forward a show while watching it. But no further navigation is possible.

Integrating these various content sources will require a metadata framework to associate content. Some work has been done in this area, mostly for practitioners wanting to share recipes online. RecipeML, an XML specification for recipes, has been developed by FormatData to standardize the terms used in recipes and the relationships between aspects of a recipe such as ingredients, techniques, and equipment. Incorporating metadata of this sort into production content will enable a high degree of flexibility between recipe components

Another element critical to VideoSkillet’s adoption is the ability to use interactive video in the kitchen environment. Beyond Connected has released a device called ‘The iCEBOX’ that allows users to browse the Internet, watch television or a DVD, and listen to FM-Tuner. It is also allows a variety of input mechanisms including touchscreen, remote control, and washable keyboard. While, the iCEBOX does not have significant processing capability, it begins to bring together the functionality of the web and video in the kitchen.

Use Scenarios

Demonstrated scenario – basic functionality of the prototype

Joe is a novice cook who occassionally watches the Food Network and enjoys trying new recipes and learning new techniques. He is going to a holiday potluck party tomorrow and wants to bring a pecan pie. He remembers a Martha Stewart show from a few weeks ago in which she and her mom make several pie crusts and fillings. He uses the VideoSkillet system to find the pecan pie recipe they made in the show. It involves a separate pie dough recipe and the pecan pie with a decorative crust. Joe doesn’t have time to make the pie dough from scratch, so he buys several pie shells and sets out to cook along with Martha to make the pecan pie filling and the decorative pie crust.

When Joe launches VideoSkillet Cooking mode, the system will offer a video skiom of the recipe by default. The skim is a brief video that shows each of the major recipe steps either in process or completed, ending with the original shot of the finished product. This is similar to the way Joe usually reads the recipe several times before getting started, to make sure he has a sense for how the whole process will work. He likes the video skim even better because it reminds him of the steps visually, rather than leaving him with only the text to read over and over again.

Next Joe wants to check the ingredients list and the equipment list to make sure he has everything the recipe requires. He uses the right and left arrows on his remote control to move back and forth across the main menu items in the horizontal navigation bar at the bottom of the VideoSkillet Cooking mode screen. When he chooses a section, e.g. Ingredients, and clicks the OK button, an overlay of a text list of ingredients comes up on the left side of the screen. The skim will continue to play in a loop until Joe selects a different video segment, e.g. video associated with a step or a technique in the recipe. Whenever a vertical menu is showing the video is presented in a smaller portion of the screen, while maintaining the proper aspect ratio. He can use the up and down arrows on his remote control to access more information about particular ingredients, steps, techniques or equipment. (Please see the Prototype Walk-Through for screenshots of this interface.)

Now Joe is ready to get started on the recipe. He chooses to start with step 2 in which Martha is rolling out the dough. He selects the step and the system plays that step with the audio muted until Joe requests the next step.

After rolling out the dough, Joe uses the Steps menu to jump to the fourth step regarding creating edges. In this segment Martha is making a crimped edge. The interface indicates to Joe that there are substitute techniques associated with this step, so he accesses the techniques menu and selects the create edges technique to access the available options. He selects the leaf technique and watches it for a moment, before he realizes that it is too time-intensive for him today. He returns to the Steps interface and picks up where he left off with the step about creating crimped edges. He finishes the recipe, bakes his pie and everyone enjoys a particularly tasty and beautiful pie at the potluck the next day.

Future scenario – advanced functionality

An amateur cook who wants to share her “famous pot roast” recipe with her family and friends:

Marian friends and family have always loved her delicious pot roast. They often call her and ask her to remind them of the ingredients or the steps.

She signed up for the VideoSkillet service recently because she loves to cook and to watch cooking shows, and because she wants to create her own video recipes to share.

Today she made a pot roast and her husband took a picture of her and the finished product.

She logs onto the VideoSkillet site to create Marian’s Famous Pot Roast recipe. The system provides her with a form where she can enter text descriptions of the ingredients and instructions and any introductory or closing comments she wants to add. She can upload digital video or stills to accompany any or all of the text segments. When she uploads visual media the system asks her to provide a set of additional descriptions related to the content. When she submits the new recipe the system returns suggestions for media segments in the VideoSkillet repository that she might want to link to steps in her recipe. She chooses which segments she wants to add and finalizes her recipe. This recipe becomes part of her personal cookbook and part of the VideoSkillet repository. She can always access and edit it. When her new recipe is successfully completed, the system prompts her to send off emails to her friends and family notifying them that her video-enhanced famous pot roast recipe is now available on VideoSkillet.

The First VideoSkillet Interactive Prototype

Prototype scope

Since VideoSkillet is a large and complex system comprised of many modules and application layers, we decided to select only one mode for the interactive prototype.

We chose to demonstrate the Cooking mode for a number of reasons:

It represents the key competitive advantage and value proposition of VideoSkillet. Cooking mode allows users to cook along with cooking shows, controlling the pace of work, and aligning video with the linear progression of the recipe. This augmented interaction with televised cooking show content is a key benefit of the system.

VideoSkillet’s viability as an useful tool hinges on this interface. The design of this interface is central to demonstrating our vision for the technology.

It presents a number of compelling interaction design questions such as

what types of computing technologies will be welcomed in the kitchen?

what are the range of potential input devices to the system?

what existing technologies and interfaces will help guide the design?

Prototype overview

The prototype demonstrates the following functionality:

A video skim provides an overview of an entire recipe. This is analogous to reading a recipe all the way through in a cookbook before starting to cook.

Navigation between the main parts of the interface: recipe steps, ingredients, techniques, and equipment.

Random access of recipe steps.

Finding and watching alternative techniques for those used in the recipe.

Looping steps so that the user can control when he/she is ready to move ahead.

A note about looping

We surmised that the audio track repeat might become annoying to users while they are looping through a step. Our prototype confirmed this hypothesis. We were not able to implement loops with muted audio, i.e. audio track on the first iteration only, in this version of the prototype. We recommend that you turn the volume off manually while using the prototype. Also, we received feedback suggesting the use of slow-motion while looping. We found this idea very compelling, but there were concerns that slow motion might interfere with the user's understanding of the rhythms of particular techniques, which is one of the primary benefits of video over text. In further work we would like to test the usability of slow motion video in this application context.

The interaction device

Initial brainstorming led us to conceive of multiple input devices that could be used to control VideoSkillet. These ranged from voice control, to a tablet and stylus, to gestural interfaces, to a machine-washable remote control. For initial versions of the application we sought to integrate the system as seamlessly as possible into the existing technological infrastructure with which our user community is familiar. For this reason we chose to prototype a system tht uses a typical remote control like those used for digital cable or PVRs as an input device. These remotes typically have left/right/up/down buttons for navigation and other buttons for interactions such as: an “OK,” “Exit,” “Menu,” “Guide,” and “Info” button.

Prototype implementation

We implemented the prototype using Macromedia Flash MX and Adobe Premiere. We also went through a few rounds of paper/pen iterations.

Prototype System Requirements

The Flash 6 player (or later versions). The prototype can be viewed as a standalone Flash movie or through a web browser that has the Flash 6 (or later) plug-in.

Prototype walk-through - Overview – screen elements

Recipe overview/skim

Pressing the OK button from the opening screen launches directly into the full screen view of the recipe overview. The skim consists of key frames with animated transitions.

Pressing the left and right buttons navigates through the main menu choices: Steps, Ingredients, Techniques, and Equipment. Pressing the “OK” button brings up the sub-menu. The current movie will continue to play. This is similar to functionality found in satellite TV and digital cable interactive program guides (except Comcast’s which only lets you hear what you were watching rather than see a small version of it also).

Navigation through main menus and sub-menus

The left and right buttons navigate the main menu. The up and down button navigate various sub-menus. A text description of the sub-item appears to the left. If a description is too long, the “info” button (not implemented) would launch a sub-window with more text.

Each sub-menu should have a “Browse all” option (currently, only the Steps sub-menu does). This would bring up a DVD-like full screen menu that allows visual browsing of all the items in that menu (not implemented).