Android System Verification Part 5

In the previous article I introduced the use of Specman for generating sequences of commands to send to the Android emulator. I demonstrated a better way to generate hardware stimulus compared to using the original telnet connection to the Android console. The technique was also better than trying to think of all possible combinations of commands and trying to create a script or C program to cover everything. Today, I will demonstrate how to use coverage to understand the generated sequences and identify areas were additional sequences might be needed to cover holes.

The first thing we can easily do is to create a coverage item for the kind of sequence we generate and send to the Android emulator:

extend sw_sequence {
   cover ended is {
        item kind using ignore = (kind == RANDOM or
                                  kind == SIMPLE or
                                  kind == MAIN);
        item kind using also text = "sequence type";
   };
};

Transition coverage is an easy way to see which type of command followed another command type. For example, did we ever change the network delay after a call was placed on hold? This can be done with a single line to the previous cover statement:

transition kind;

Another coverage scenario might to be cover the length of the SMS messages received. Historically, the maximum length of an SMS was set at 160 characters, this includes any letters, numbers, and punctuation. Subsequently most phones started supporting longer messages by breaking them into multiple 160 character SMS messages. Coverage can be used to make sure Android correctly handles all different message lengths.

When I was doing some testing of the SMS message length I came across this post, a user complaint about the Android Messenger saying that only a maximum of 3*160 characters are supported and larger messages were converted to MMS (which may have different pricing depending on the user's plan).

extend  SMS sw_sequence {
   cover ended is also {
      item sms_cmd using text = "SMS command type";
      item sms_length using ranges = {
          range([0], "Zero Length", UNDEF, 1);
          range([1..160], "Normal Length", UNDEF, 1);
          range([161..480], "Long Length", UNDEF, 1);
          range([481..MAX_INT], "Very Long Length", UNDEF, 1);
      };
      item sms_type using text = "Normal or Long SMS (160 characters or less is normal)";
   };
};

Coverage provides a way to merge and view the results of multiple tests. This gives a complete view of all of the system activity over the full set of tests. The screen shot below shows the coverage GUI. The right side shows the transition coverage of the sequence kind.

 

The next step is to implement checking so that we can be sure that the generated sequences actually succeed. Right now, we can send a command to make a phone call, put a call on hold, or set the GPS location, but how do we know these commands were actually carried out correctly? This is where checking comes in. Checking is needed to close the loop so we can send commands and check they were done correctly, but this will be the topic for next time.

Jason Andrews