September 2004 doc.: IEEE802.11-04/1166r0
IEEE P802.11
Wireless LANs
Minutes for the Task Group T September 2004 Session
Date:
September 14 - 16, 2004
Authors:
Areg Arimian, Roger Skidmore
e-Mail: ,
Monday, September 13, 2004
4:00 PM – 6:00 PM
- Chair calls the meeting to order at 4:00 PM – Charles Wright
- Chair comments
- Mike Goettemoeller appointed secretary for Tom Alexander who is absent
- Chair asks for objections for Portland minutes, none noted, Portland minutes accepted.
- Chair asks for objections to agenda, none noted, agenda accepted.
- Chair has made a call for presentations
- 11-04/1009 – “Framework, Usages, Metrics Proposal for TGT”, Pratik Mehta
- 11-04/1017 – “Comments on Wireless Performance & Prediction Metrics”, Mark Kobayashi
- 11-04/xxxx – “Systems supporting devices under test”, Mike G.
- Chair asks for approval of modifications to agenda, note noted, revised agenda accepted.
- Discussion of timeline going forward, progress since Portland – IEEE NesCom approval to form TGT, technical presentations, re-present ideas.
- Chair requested to comment on teleconferences and status of group as a whole – agreed that metrics need to be derived first before going forward.
- Chair makes move to recess until 4pm tomorrow; Roger makes motion, Lars 2nds. Approved by acclaimation.
Tuesday, September 14, 2004
4:00 PM – 6:00 PM
- Chair calls the meeting to order at 6:00 PM – Charles Wright
- Roger Skidmore appointed secretary
- Technical Presentation – Framework, Usages, Metrics Proposal for TGT – 11-04/1009r1 – Pratik Mehta
- Comment – Requesting an example of a submetric as described on slide 11.
- Comment – Presenter response is that packet loss would be an example submetric of the throughput metric.
- Comment – On slide 12, directionality could be an important factor in non-data oriented applications as well.
- Comment – Chair calls for discussion.
- Comment – Engineers tend to emphasize the confidence level of wireless measurements.
- Comment – Presenter responds that confidence levels and similar things of a statistical nature are part of the methodology used to measure a given metric/submetric.
- Question – Taking an application-centric approach will allow the group to better yield results in a reasonable amount of time?
- Comment – Presenter refers to slide 10, responding that it is better to map where the group wants to be as part of the process.
- Comment – Testing a device for the home may be very different from testing a device for an enterprise even for the same application.
- Comment – Presenter responds that defining the usage and environment (per slide 10) helps partition the problem.
- Question – How do you know that you are actually testing an “average” device rather than a “best-of-the-batch” or “worst-of-the-batch” type device?
- Comment – Presenter responds that some sort of calibration, normalizing, or random sampling procedure may be required.
- Comment – Being sensitive to the amount in variability in the tested response of a device is important.
- Question – Any thought to common sources of interference and how those affect certain receive architectures?
- Comment – Presenter responds that no specific thoughts have been centered on that topic as yet.
- Chair – As in interferers unduly affecting a test?
- Comment – Uses in a home may involve myriad devices including non-802.11 devices.
- Chair – Topic begins to border on a co-existence issue.
- Comment – Certain radios have more robust interference rejection.
- Question – Does this mean that multiple environments need to be introduced for individual metrics?
- Comment – Presenter responds that it is possible that metrics and submetrics will be tested differently across different use cases.
- Comment – Prediction folks should be able to identify what types of interference scenarios should be analyzed.
- Question – Is this considering a single device under test or a linked set of devices?
- Comment – Presenter responds that typically single devices have been considered, but the group needs to take this as a point to discuss.
- Comment – Should be looking at packet errors as a metric tested against multipath and other physical layer things. Can deduce throughput from packet error rate.
- Comment – Presenter responds that the user expectation/view is typically in terms of throughput, but that packet loss could/should be a metric listed on slide 13. Slide 13 was not intended to list every possible metric.
- Question – Final step in process on slide 10 is prediction. When will there be a focus on prediction?
- Comment – Presenter responds that predictions will have a better chance of being accurate with measurement data. Group should certainly tackle predictions.
- Chair – The group is no longer predicting performance.
- Question – Was prediction was cut from scope?
- Chair – Yes. It is within scope “to enable prediction”. The development of prediction models and prediction algorithms do not fall within the scope of the group.
- Question – Beginning to hear talk of device classification. Is it within the scope of the group to discuss device classes?
- Chair – Device classifications or qualifying devices is outside of scope. Ranking or qualifying devices is not the goal of the group. The output of the group could be used to enable device ratings, but the goal of the group is not create ratings.
- Comment – Believe that group is on track to enabling measurements, but do not have a feeling on what the group can do to enable predictions. Request for presentation on predictions and what is needed to enable them.
- Technical Presentation – Comments on Wireless Performance & Prediction Metrics – 11-04/1017r0 – Mark Kobayashi
- Comment – On slide 10, producing something that simply says system A is “better” that system B is worthless. Need something quantifiable. For example, quantifying device range.
- Chair – Comment that range may vary depending on the type of test.
- Comment – Presenter comments that could develop categories of particular tests (e.g., bad channel models) that may help simulate actual conditions.
- Chair – In other words, want to map “user experience” into some form of repeatable test. What do people think about the channel models developed for .11n?
- Comment – You’ll get differing responses on that.
- Chair – Need to solicit input on channel models from the larger 802.11 group.
- Comment – 802.19 utilizes channel models for coexistence.
- Comment – Interference and multipath can be tested separately from channel models. Certain “barebones” metrics need to be tested and dealt with separately.
- Question – Does the material presented in 1017r0 conflict with that in 1009r1?
- Comment – Presenter indicates that 1017r0 and 1009r1 are very complimentary in terms of the basic framework. Presenter prefers to look at usage model 1, 2, and 3 and produce a set of common metrics across all usage models; the goal being to identify similarities in the tests performed for each usage models.
- Comment – Is a timeline being proposed?
- Comment – Presenter indicates no specific timeline is being suggested, but believes work needs to get underway.
- Comment – Need to limit time spent discussing metrics in order to help move the work flow along.
- Comment – Considering common metrics across platforms/usage models will be beneficial.
- Comment – The presentations 1017r0 to 1009r1 appear to have distinct approaches. 1009r1 appears to address “real” usage models and seems to cover a large audience. The specific test environment for a given submetric test (per 1009r1) can characterize the submetric/metric very well. If the group were to only do Cabled Environment (for example), and then go one metric at a time, it will be somewhat limiting in terms of the usability of the group’s output.
- Comment – Presenter indicates that sets of identical/near identical tests need not be repeated for different usage models.
- Question – Would it be beneficial to mimic a measured channel using a waveform generator with devices in an anechoic chamber in order to achieve repeatability in a controlled environment?
- Comment – Presenter indicates that the suggestion is possibly a good way of performing a test.
- Comment – There will be a flow of data building up from semiconductor companies, to manufacturer, to system integrator, to IT manager. Each one higher up the chain is relying on decisions/results of those below them. TGT should try to get everyone along the chain talking the same language.
- Comment – 1017r0 and 1009r1 are actually very different from one perspective. What is needed first is a test from the point of view of the user. Then afterwards, when that particular use case is finished, begin working backward to deal with other usage cases. Commenter is concerned that time will be wasted trying to analyze too many potentially very different usage models looking for commonalities rather than tackling one particular set of usage models, finishing them, and then proceeding.
- Meeting in recess at 6:00 until 7:30.
Tuesday, September 14, 2004
7:30 PM – 9:30 PM
- Chair calls meeting to order at 7:40 PM
- Chair – Open forum for discussion on previous presentations – brainstorming
- Comment – Need to avoid discussions of metrics without having an end goal or at least a particular use case in mind.
- Comment – What about a test configuration?
- Comment – Can’t discuss a test configuration without knowing what metric you’re searching for. Suggest considering metrics in different levels geared towards a particular audience (e.g., system integrators, semiconductor, etc.).
- Comment – Should not want to specify a particular vendor’s test equipment (i.e., “golden node”). For example, could have two devices under test where you simply set them up side-by-side, but that may not be valid.
- Comment – Difficult to completely mimic all test conditions without some form of standardization of test equipment. Is a faulty test due to the device under test or because the test equipment or conditions were ever so slightly off?
- Comment – Possible test flaws could be listed as test preconditions (e.g., could a listed as limitations of testing systems) or test “gotchas”. Then if test was not 100% repeatable, would be due to one or more of the possible test preconditions.
- Comment – For throughput and forwarding rate, we know what theoretical maximums are. For latency and jitter, have no minimums other than zero. So preconditions could also include bounds.
- Comment – Varying degrees of test could also be a measure of the repeatability of the test. The more repeatable the test, the closer the test preconditions would need to be to the required, precise test conditions and/or setup.
- Question – Is non-repeatable/instantaneous measurement testing infringing on .11k?
- Comment – .11k could serve as a vehicle for fulfilling the mechanism or procedure used in the test.
- Comment – There is a big difference between a company with heavy investment in test equipment/labs and significant experience in equipment analysis and an IT manager who is evaluating a product offering. Those two categories of “consumers” for TGT’s output are very different in their needs.
- Comment – There is no way to simulate in a lab every possible scenario an IT manager may experience, it is possible to specify sets of tests an IT manager could take the time to setup and perform. The sets of tests may have varying complexity and/or range of error based on how closely the IT manager can duplicate the test conditions/preconditions.
- Comment – Do we know how current IT managers qualify equipment and network configuration?
- Comment – It varies widely. The wireless knowledge of IT managers factors heavily into this.
- Question – Is this group going to focus on how to deploy a network?
- Chair – No, this is outside of the scope of this group. This group is focused on measuring device performance under certain test conditions and then utilizing that information to make performance predictions later.
- Comment – At the end of the day, we need to measure metrics.
- Comment – The types of tests that you need to run will vary depending on who is doing the testing. Semiconductor companies need different tests than IT managers. Need categories of tests for categories of users.
- Example PHY layer metrics
- Packet error rate (PER)
- Factors inherent in the device (recommended practice would specify bounds on the following for particular PER tests)
- Error vector magnitude (EVM)
- Transmit power
- Receive sensitivity
- Comment – What is the most representative controlled test for 802.11 wireless performance?
- It would exercise the entire device in one test.
- It would focus only on performance aspects affected by the MAC, PHY, or antenna.
- It would be a “macro” measurement, not a “micro” measurement (e.g., focus on “communication system” level metrics – forwarding rate, loss, delay, jitter, antenna gain, FER vs. input signal level).
- Comment – Why use layer 2 traffic for testing?
- TGT must stay in the domain of 802.11
- Using true or simulated application traffic obscures the effects of wireless
- TCP is a common offender.
- Specifying application level traffic makes it hard to decide where to stop (e.g., which favorite video or voice standard do we exclude?)
- Comment – The “big four” layer 2 measurements are forwarding rate, MSDU loss, packet delay, and jitter all made under various conditions (e.g., with and without multipath, with and without adjacent channel interferers, and varying signal levels)
- Question – Is multipath still an issue with OFDM?
- Comment – Yes. OFDM may be resistant to multipath, but papers presented here have shown that multipath needs to still be considered.
- Comment – All of the metrics we are discussing measuring for wireless have already been defined as well as procedures for measuring them on wired networks. TGT should focus on layer 2 metrics and leave upper layers alone (e.g., application layer).
- Meeting in recess at 9:35 PM until 4:00 PM Thursday afternoon.
Thursday, September 16, 2004
4:00 PM – 6:30 PM
1. Chair calls the meeting to order at 4:00 PM
2. Announcement regarding file server being temporarily unavailable for uploading documents
3. New items added to today’s agenda:
- Technical Presentation -- 11-04/1131r1, “A Metrics and Methodology Starting Point for TGT”, Charles Wright, Mike Goettemoeller, Shravan Surineni, Areg Alimian
- Motions or straw polls arising from 11-04/1009r1 and 11-04/1131r1
- Technical Presentation -- 11-04/1106r0, “Systems Supporting Device Under Test”, Mike Goettemoeller
- Technical Presentation -- 11-04/1132r0, “Enabling Prediction of Performance”, Roger Skidmore
4. Amendments to the agenda accepted by unanimous consent