A thesis submitted to the Graduate Faculty of the University of Colorado at Colorado Springs in partial fulfillment of the requirements for the degree of Master of Science
Department of Computer Science

© Copyright By David Robert Stites 2012 
All Rights Reserved

Mobile devices, such as smartphones or PDAs, have become increasingly popular with consumers and often provide essential functionality in their everyday life. Usually these mobile devices contain a great deal of sensitive information such as addresses, contacts, ingoing/outgoing call logs, SMS messages and, on the latest models, a calendar, emails and potentially the user’s current location. A smartphone or mobile device today can be as powerful as a desktop or laptop in some respects and, while the latest models feature a complete OS, for many users these devices are “just phones” so there is an underestimation of the risk connected to mobile device privacy. There is a currently existing privacy problem associated with user and hardware tracking in mobile devices. Users can be tracked without their knowledge and consent and have rich profiles built about them using their hardware interface address regarding their location and preferences. This information can be potentially cross correlated to other existing datasets to build advertising profiles for these users. The mitigation to this problem using a framework to support randomly generated, disposable hardware addresses.

Full text of the document can be found here: PDF

But first, a bit of background. What is IPC? Courtesy of WikiPedia, “IPC is a set of methods for the exchange of data among multiple threads in one or more processes. Processes may be running on one or more computers connected by a network. IPC methods are divided into methods for message passing, synchronization, shared memory, and remote procedure calls (RPC). The method of IPC used may vary based on the bandwidth and latency of communication between the threads, and the type of data being communicated.”

Originally, you could have done IPC in OS X using mach messages, which is how drivers traditionally communicated.

While information sharing and modularity are definitely some of the benefits of IPC, one of the biggest wins in my mind is the fact that we can perform privilege separation with IPC. Consider the following: you have wrote some code that will take some user input and crunch on it and then return a result. Note that this doesn’t have to be intensive computation, it could be as easy as interpolating an NSString. User input is a taint source, meaning that the input data is untrusted and could potentially (and perhaps unintentionally) be malicious. If your program were running in a privileged mode or had some increased set of ACLs, then if the input were able to exploit a vulnerability, then it would be able to inherit the same privilege level as the application.

Another benefit of this separation is that suppose the input causes the program to crash. If the processing were done in the main application, it would crash the entire application. If the processing is done in the daemon, then the daemon can crash and the application would still be alive and well.

I have attached a project that demonstrates NSURLConnection’s ability to say “Hello.” hello.tar.gz

Double Armor
StegaGram protects your communication in two ways. First, it locks the message so that it can only be read by the person to whom you’re sending the message. Then, it hides the locked message inside a picture, so that it doesn’t even look like a locked message is being sent. As an analogy, consider keeping your valuables in a strong safe, located in your front yard. It’s a great safe, but why invite attention to the fact that you have it? Using StegaGram is like keeping that strong safe hidden in a secret panel behind a picture in your house.

No Password, No Problem
Short passwords are not very secure because they can be quickly guessed by computer programs. Long passwords are better, but can be hard to remember. We chose to avoid these problems altogether by using long strings of random numbers, known as keys. If you want more details you can read more below. Otherwise, suffice it to say that it’s stronger than a password but you don’t need to remember anything. You just need to pass a key to your friend using a QR Code — those barcode-looking squares you see all over the place.

Under the Hood
You know those cars that look cool from the outside but lack actual power and performance when driving? Yeah, that’s not us. StegaGram is clean and easy to use, but also employs the latest methods of cryptography and steganography. In fact, our initial version was denied for public distribution because it was too strong. We had to tone it down a bit. As for our hiding methods, they don’t just avoid detection by the human eye. We use a technique that passes well under the radar of digital analysis programs which search for anomalies in histograms.

For the Nerds
StegaGram uses a combination of asymmetric cryptography and an optimized version of the Graph-Theoretic approach to steganography. The asymmetry of the cryptographic keys allows for a distributed authentication model, similar to that used in the PGP community. Our initial version uses 2048-bit RSA encryption. As for the key exchange, the QR-Code method prevents the classic ‘Man-in-the-Middle Attack’ used against the Diffe-Helman pattern, because there is no communication over a network during the exchange. In addition, our steganographic algorithm preserves first-order statistics, unlike most other freely-available steganographic software. For more details, take a look through our research paper.

Fine Print
This application was created for academic and recreational purposes, and comes with no guarantees or warrantees for its information protection. It’s pretty burly, as mentioned above, but is used at your own risk. Thank you to Alex Renger for the idea of StegaGram and for the steganographic algorithm. Thank you to Dr. Yue from the University of Colorado at Colorado Springs for his teaching and support.

Buffer Overflows

A Survey on Mobile Device Security

SQL Injections

Initially I say “No, but it adds to the key space” (key space is the total number of possibilities that an attacker must try).  What I actually mean to say is “Yes, it does increase the “hardness” of the password (and increases the key space), but it is not strictly necessary to use.”

One of the tips I like to use is to create a password that is composed of several words that you like or are particularly memorable.  For example, you could probably look around your office or home and do this.  Looking around me, I could choose projectorchocolateglobemarker (you could choose something else that is easier to remember and I probably would too, but this is just an example).

Consider this: if a password was composed of lower case, upper case, numbers and special characters, that would be 72 characters to chose from.  If your mean system administrator said that your password had to be composed of 8 characters combining those 4 types of characters, then the total number of combinations would be ~7.22 x 10^14 (assuming a password length of 8 characters).  That’s a lot of possible combinations a cracker/hacker must try!

Now consider the password where I ran the words together (projectorchocolateglobemarker).  This would be resistant to a dictionary attack.  A dictionary attack uses a targeted technique of successively trying all the words in an exhaustive list called a dictionary (from a pre-arranged list of values).  Commonly, these pre-arranged lists are actually words from a dictionary because people are lazy and they will make their password is easy to remember (not through any fault of their own but humans are notoriously bad at remembering things that doesn’t make sense or they have no connection to).   While those individual words (projector, chocolate, globe and marker) would all be in the dictionary, the concatenation of them would not.  While we have fewer total characters to choose from (26 lower case characters), the total number possible combinations for the password would be orders of magnitude larger.  In fact, it would be ~1.052 x 10^41.  That is a much larger key space!

Now, if you haven’t glossed over by now, consider a computer that could do 1 key attempt per microsecond (which is certainly not out of the realm of possibilities), that is about 1 million key attempts (to crack) per second.  The first password would take 722M seconds (22 years), the second would take ~10^27 years.  Clearly, we can see which is more secure (and I didn’t need a lot of characters to do it).  These longer, harder passwords, are also more immune to what we call “rainbow table” attacks.

A lot of the password stuff is what is called “security theater.”  Security theater is a term that describes security countermeasures intended to provide the feeling of improved security while doing little or nothing to actually improve security.  Having a system administrator create policies that don’t make sense (such as the crazy combinations of letters, numbers, special chars) when the password is far less secure is an example of security theater.  Bruce Schneier uses it a lot to describe TSA security.

Another way to create a memorable password is to think of a memorable phrase such as “my sister likes to eat juicy orange every day”.  Then, take the first letter of each word and combine them to make your password.  In this example, your password would be msltejoed.

Lastly, I wanted to mention one last tip.  If a user could setup a policy that would lock out after a certain number of failed attempts, such as 3, this is the most secure way to do a pass-worded system because it wouldn’t allow an attacker to do a “brute force” attack where they try all the different types of keys.

This particular licensing module has support for multiple license types: Trial, Single Version and Lifetime.  Also, the license has support for information such as name, email, license number, license type, expiration date and version number.  There is support for blacklisted, invalid, phony, and expired keys.  The one thing I want to mention before we get into the code is that you’ll need a public and private keypair (you can use OpenSSL to do this) in .der format (an X509 certificate using OpenSSL again).  So without further ado…

Licensing code zip

License.java: This is your license object.  It will be written to disk and checked with the program starts.  It contains all the information the managers will need to check the license.

package License;

import java.io.*;
import java.util.*;

public class License implements Serializable {
    private static final long serialVersionUID = 1L;

    private String name;
    private String email;
    private String licenseNumber;
    private LicenseType licenseType;
    private Date expiration;
    private String version;

    public License() {
        name = "";
        email = "";
        licenseNumber = "";
        expiration = new Date();
        version = "";
        licenseType = LicenseType.TRIAL;
    }

    public License(String name, String email, String licenseNumber, Date expiration, LicenseType licenseType, String version) {
        this.name = name;
        this.email = email;
        this.licenseNumber = licenseNumber;
        this.expiration = expiration;
        this.licenseType = licenseType;
        this.version = version;
    }

    // getters and setters here
}

KeyStatus.java: This is an enumeration that returns the status of a key validation operation.

package License;

public enum KeyStatus {
    KEY_GOOD,
    KEY_INVALID,
    KEY_BLACKLISTED,
    KEY_PHONY,
    KEY_EXPIRED
}

LicenseType.java: This is an enumeration that represents the type of license being generated.

package License;

public enum LicenseType {
    TRIAL, SINGLE_VERSION, LIFETIME
}

LicenseFileFilter.java: This is purely for usability on the UI end.  We can provide the JFileChooser this filter and it will find only our license files.

package License;

import Utility.FileExtension;
import java.io.File;
import javax.swing.filechooser.FileFilter;

public class LicenseFileFilter extends FileFilter {
    public boolean accept(File f) {
        return f.isDirectory() || f.getName().toLowerCase().endsWith(FileExtension.ZIP);
    }

    public String getDescription() {
        return "License files";
    }
}

LicenseManager.java: This is the main workhorse for the licensing code.  It generates the keys and checks them.

package License;

import java.io.*;
import java.net.*;
import java.security.*;
import java.util.*;
import javax.swing.JOptionPane;

public class LicenseManager {
    private static LicenseManager instance;
    public static boolean IS_TRIAL = true;
    public static boolean IS_LICENSED =  false;
    public static License LICENSE = null;
    private static final int ENTROPY = 456456456;
    private static final String HEXES = "0123456789ABCDEF";

    public static final String LICENSE_FILENAME = "license";
    public static final String HASH_FILENAME = "license.sha1";
    public static final String SIGNATURE_FILENAME = "license.sig";

    private static final int KEY_LEN = 62;
    private static final byte[] def = new byte[]{24, 4, 124, 10, 91};
    private static final byte[][] params = new byte[][]{{24, 4, 127}, {10, 0, 56}, {1, 2, 91}, {7, 1, 100}};
    private static final Set blacklist = new TreeSet();

    private Timer t;
    private static final int DELAY = 900000;

    static {
        blacklist.add("11111111");
    }

    protected LicenseManager() {
        t = new Timer();
        t.scheduleAtFixedRate(new CheckLicenseTask(), DELAY, DELAY);
    }

    public static LicenseManager getLicenseManager() {
        if (instance == null) {
            instance = new LicenseManager();
        }

        return instance;
    }

    /**
     *
     * @param lic
     */
    private void writeLicenseFile(License lic, String path) {
        try {
            ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File(path + LICENSE_FILENAME)));
            oos.writeObject(lic);
            oos.close();
        }
        catch(Exception ex) { }
    }

    public static KeyStatus readLicenseFile(String licensePath, String signaturePath, String hashPath) {
        try {
            // read in file and validate the LICENSE based on the signature
            // this will remove changes of faking a LICENSE file
            // the LICENSE file has to be signed with our key
            File licenseFile = new File(licensePath);
            File signatureFile = new File(signaturePath);
            File hashFile = new File(hashPath);

            KeyStatus status = EncryptionManager.getEncryptionManager().verify(licenseFile, signatureFile, hashFile);

            if(!status.equals(KeyStatus.KEY_GOOD)) {
                return KeyStatus.KEY_INVALID;
            }

            ObjectInputStream ois = new ObjectInputStream(new FileInputStream(licenseFile));
            LICENSE = (License)ois.readObject();

            String lic = LICENSE.getLicenseNumber();
            if(LICENSE.getLicenseType().equals(LicenseType.TRIAL)) {
                IS_TRIAL = true;

                Calendar c = Calendar.getInstance();
                if(c.getTime().after(LICENSE.getExpiration())) {
                    return KeyStatus.KEY_EXPIRED;
                }

                Date expiration = LicenseManager.getLicenseManager().LICENSE.getExpiration();

                long val = expiration.getTime() - c.getTime().getTime();
                val /= (1000 * 60 * 60 * 24);
                JOptionPane.showMessageDialog(null, "This is a trial version of the software.  You have " + val + " days remaining in your trial.", "Trial Version", JOptionPane.INFORMATION_MESSAGE);
            }
            else if(LICENSE.getLicenseType().equals(LicenseType.SINGLE_VERSION)) {
                IS_TRIAL = false;
                status = checkKey(lic);
                IS_LICENSED = true;
            }
            else if(LICENSE.getLicenseType().equals(LicenseType.LIFETIME)) {
                IS_TRIAL = false;
                status = checkKey(lic);
                IS_LICENSED = true;
            }

            if(!status.equals(KeyStatus.KEY_GOOD)) {
                return status;
            }

            return KeyStatus.KEY_GOOD;
        }
        catch(Exception ex) {
            System.out.println(ex.toString());
            return KeyStatus.KEY_INVALID;
        }
    }

    /**
     *
     * @param name
     * @param email
     * @param authCode
     * @param licenseType
     * @param expiration
     * @param version
     * @param path
     */
    public void createLicense(String name, String email, String authCode, LicenseType licenseType, Date expiration, String version, String path) {
        byte[] entropy = null;

        try {
            MessageDigest digest = MessageDigest.getInstance("SHA-512");
            digest.reset();
            entropy = digest.digest(getByteArrayFromHexString(authCode));
        }
        catch(NoSuchAlgorithmException ex) { /* this will never happen */ }

        License lic = new License(name, email, LicenseManager.makeKey(ENTROPY, entropy), expiration, licenseType, version);
        writeLicenseFile(lic, path);
    }

    /**
     *
     * @return
     */
    private static byte[] getHardwareEntropy() {
        byte[] mac;
        try {
            NetworkInterface ni = NetworkInterface.getByInetAddress(InetAddress.getLocalHost());
            if (ni != null) {
                mac = ni.getHardwareAddress();
                if (mac == null) {
                    mac = def;
                }
            } else {
                mac = def;
            }
        }
        catch (Exception ex) {
            mac = def;
        }

        byte[] entropyEncoded = null;
        try {
            MessageDigest digest = MessageDigest.getInstance("SHA-512");
            digest.reset();
            entropyEncoded = digest.digest(mac);
        }
        catch(NoSuchAlgorithmException ex) { /* this will never happen */ }

        return entropyEncoded;
    }

    /**
     *
     * @param seed
     * @param a
     * @param b
     * @param c
     * @return
     */
    private static byte getKeyByte(final int seed, final byte a, final byte b, final byte c) {
        final int a1 = a % 25;
        final int b1 = b % 3;
        if (a1 % 2 == 0) {
            return (byte) (((seed >> a1) & 0x000000FF) ^ ((seed >> b1) | c));
        } else {
            return (byte) (((seed >> a1) & 0x000000FF) ^ ((seed >> b1) & c));
        }
    }

    /**
     *
     * @param s
     * @return
     */
    private static String getChecksum(final String s) {
        int left = 0x0056;
        int right = 0x00AF;
        for (byte b : s.getBytes()) {
            right += b;
            if (right > 0x00FF) {
                right -= 0x00FF;
            }
            left += right;
            if (left > 0x00FF) {
                left -= 0x00FF;
            }
        }
        int sum = (left << 8) + right;
        return intToHex(sum, 4);
    }

    /**
     *
     * @param seed
     * @param entropy
     * @return
     */
    public static String makeKey(final int seed, byte[] entropy) {
        // fill keyBytes with values derived from seed.
        // the parameters used here must be exactly the same
        // as the ones used in the checkKey function.
        final byte[] keyBytes = new byte[25];
        keyBytes[0] = getKeyByte(seed, params[0][0], params[0][1], params[0][2]);
        keyBytes[1] = getKeyByte(seed, params[1][0], params[1][1], params[1][2]);
        keyBytes[2] = getKeyByte(seed, params[2][0], params[2][1], params[2][2]);
        keyBytes[3] = getKeyByte(seed, params[3][0], params[3][1], params[3][2]);
        for(int i = 4, j = 0; (j + 2) < entropy.length; i++, j += 3) {
            keyBytes[i] = getKeyByte(seed, entropy[j], entropy[j + 1], entropy[j + 2]);
        }      

        // the key string begins with a hexadecimal string of the seed
        final StringBuilder result = new StringBuilder(intToHex(seed, 8));

        // then is followed by hexadecimal strings of each byte in the key
        for (byte b : keyBytes) {
            result.append(intToHex(b, 2));
        }

        // add checksum to key string
        String key = result.toString();
        key += getChecksum(key);

        return key;
    }

    /**
     *
     * @param key
     * @return
     */
    private static boolean validateKeyChecksum(final String key) {
        if (key.length() != KEY_LEN) {
            return false;
        }

        // last four characters are the checksum
        final String checksum = key.substring(KEY_LEN - 4);
        return checksum.equals(getChecksum(key.substring(0, KEY_LEN - 4)));
    }

    /**
     *
     * @param key
     * @return
     */
    public static KeyStatus checkKey(final String key) {
        if (!validateKeyChecksum(key)) {
            return KeyStatus.KEY_INVALID; // bad checksum or wrong number of
            // characters
        }

        // test against blacklist
        for (String bl : blacklist) {
            if (key.startsWith(bl)) {
                return KeyStatus.KEY_BLACKLISTED;
            }
        }

        // at this point, the key is either valid or forged,
        // because a forged key can have a valid checksum.
        // we now test the "bytes" of the key to determine if it is
        // actually valid.

        // when building your release application, use conditional defines
        // or comment out most of the byte checks! this is the heart
        // of the partial key verification system. by not compiling in
        // each check, there is no way for someone to build a keygen that
        // will produce valid keys. if an invalid keygen is released, you can
        // simply change which byte checks are compiled in, and any serial
        // number built with the fake keygen no longer works.

        // note that the parameters used for getKeyByte calls MUST
        // MATCH the values that makeKey uses to make the key in the
        // first place!

        // extract the seed from the supplied key string
        final int seed;
        try {
            seed = Integer.valueOf(key.substring(0, 8), 16);
        } catch (NumberFormatException e) {
            return KeyStatus.KEY_PHONY;
        }

        // test key 0
        final String kb0 = key.substring(8, 10);
        final byte b0 = getKeyByte(seed, params[0][0], params[0][1], params[0][2]);
        if (!kb0.equals(intToHex(b0, 2))) {
            return KeyStatus.KEY_PHONY;
        }

        // test key1
        final String kb1 = key.substring(10, 12);
        final byte b1 = getKeyByte(seed, params[1][0], params[1][1], params[1][2]);
        if (!kb1.equals(intToHex(b1, 2))) {
            return KeyStatus.KEY_PHONY;
        }

        // test key2
        final String kb2 = key.substring(12, 14);
        final byte b2 = getKeyByte(seed, params[2][0], params[2][1], params[2][2]);
        if (!kb2.equals(intToHex(b2, 2))) {
            return KeyStatus.KEY_PHONY;
        }

        // test key3
        final String kb3 = key.substring(14, 16);
        final byte b3 = getKeyByte(seed, params[3][0], params[3][1], params[3][2]);
        if (!kb3.equals(intToHex(b3, 2))) {
            return KeyStatus.KEY_PHONY;
        }

        // test the hardware entropy
        byte[] encodedEntropy = getHardwareEntropy();
        for(int i = 16, j = 0; (j + 2) < encodedEntropy.length; i += 2, j += 3) {
String kb = key.substring(i, i + 2);             byte b = getKeyByte(seed, encodedEntropy[j], encodedEntropy[j + 1], encodedEntropy[j + 2]);             if(!kb.equals(intToHex(b, 2))) {                 return KeyStatus.KEY_INVALID;             }         }         // if we get this far, then it means the key is either good, or was made         // with a keygen derived from "this" release.         return KeyStatus.KEY_GOOD;     }     /**      *       * @param n      * @param chars      * @return       */     protected static String intToHex(final Number n, final int chars) {         return String.format("%0" + chars + "x", n);     }          /**      *       * @param raw      * @return       */     public static String getHexStringFromBytes(byte[] raw) {         if ( raw == null ) {             return null;         }         final StringBuilder hex = new StringBuilder( 2 * raw.length );         for ( final byte b : raw ) {             hex.append(HEXES.charAt((b & 0xF0) >> 4)).append(HEXES.charAt((b & 0x0F)));
        }

        return hex.toString();
    }

    /**
     *
     * @param s
     * @return
     */
    public static byte[] getByteArrayFromHexString(String s) {
        int len = s.length();
        byte[] data = new byte[len / 2];
        for (int i = 0; i < len; i += 2) {
            data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
                                 + Character.digit(s.charAt(i+1), 16));
        }
        return data;
    }

    private class CheckLicenseTask extends TimerTask {
        public CheckLicenseTask() { }

        @Override
        public void run() {
            System.out.println("checking license");
        }
    }
}

EncryptionManager.java: This module will manage encryption, decryption, signing and validation operations using your keys.  It is a singleton object.

package License;

import Utility.Logger;
import java.io.*;
import java.security.*;
import java.security.spec.*;
import java.util.Arrays;
import javax.crypto.*;

public class EncryptionManager {

    private static EncryptionManager instance;
    // this file should be in your jar
    private static final String PUBLIC_KEY_FILE = "/License/public_key.der";
    // this file will be on your hard drive
    private static final String PRIVATE_KEY_FILE = "/path/to/your/private_key.der";
    private static PublicKey publicKey;
    private static PrivateKey privateKey;

    protected EncryptionManager() throws GeneralSecurityException {
    }

    public static EncryptionManager getEncryptionManager() {
        if (instance == null) {
            try {
                instance = new EncryptionManager();

                try {
                    privateKey = loadPrivateKey(PRIVATE_KEY_FILE);
                }
                catch (Exception ex) {
                    //Logger.getLogger().ALog("private key failed to load - couldn't instantiate encryption manager.\n" + ex.toString());
                }
                try {
                    publicKey = loadPublicKey(PUBLIC_KEY_FILE);
                }
                catch (Exception ex) {
                    //Logger.getLogger().ALog("public key failed to load - couldn't instantiate encryption manager.\n" + ex.toString());
                }
            }
            catch(GeneralSecurityException ex) {
                //Logger.getLogger().ALog("couldn't instantiate encryption manager.\n" + ex.toString());
            }
        }

        return instance;
    }

    /**
     *
     * @param filename
     * @return
     * @throws Exception
     */
    private static PublicKey loadPublicKey(String filename) throws Exception {
        DataInputStream dis = new DataInputStream(File.class.getResourceAsStream(filename));
        byte[] keyBytes = new byte[dis.available()];
        dis.readFully(keyBytes);
        dis.close();

        X509EncodedKeySpec spec = new X509EncodedKeySpec(keyBytes);
        KeyFactory kf = KeyFactory.getInstance("RSA");
        return kf.generatePublic(spec);
    }

    /**
     *
     * @param filename
     * @return
     * @throws Exception
     */
    private static PrivateKey loadPrivateKey(String filename) throws Exception {
        File f = new File(filename);
        FileInputStream fis = new FileInputStream(f);
        DataInputStream dis = new DataInputStream(fis);
        byte[] keyBytes = new byte[(int) f.length()];
        dis.readFully(keyBytes);
        dis.close();

        PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(keyBytes);
        KeyFactory kf = KeyFactory.getInstance("RSA");
        return kf.generatePrivate(spec);
    }

    /**
     *
     * @param dataToHashPath
     * @return
     */
    public static byte[] digest(File dataToHashPath) {
        try {
            InputStream fin = new FileInputStream(dataToHashPath);
            MessageDigest md5Digest = MessageDigest.getInstance("MD5");

            byte[] buffer = new byte[1024];
            int read;

            do {
                read = fin.read(buffer);
                if (read > 0) {
                    md5Digest.update(buffer, 0, read);
                }
            } while (read != -1);
            fin.close();

            byte[] digest = md5Digest.digest();
            if (digest == null) {
                return null;
            }

            return digest;
        } catch (Exception e) {
            return null;
        }
    }

    /**
     *
     * @param dataToVerify
     * @param signatureFile
     * @param hashFile
     * @return
     * @throws NoSuchAlgorithmException
     * @throws NoSuchProviderException
     * @throws InvalidKeyException
     * @throws SignatureException
     * @throws NoSuchPaddingException
     * @throws FileNotFoundException
     * @throws IOException
     */
    public static KeyStatus verify(File dataToVerify, File signatureFile, File hashFile) throws NoSuchAlgorithmException, NoSuchProviderException, InvalidKeyException,
            SignatureException, NoSuchPaddingException, FileNotFoundException, IOException {
        // first validate the hash of the file
        FileInputStream hashfis = new FileInputStream(hashFile);
        byte[] hashToVerify = new byte[hashfis.available()];
        hashfis.read(hashToVerify);
        hashfis.close();

        byte[] licenseBytes = digest(dataToVerify);
        if(!Arrays.equals(licenseBytes, hashToVerify)) {
            Logger.getLogger().ALog("key failed to pass hash check");
            return KeyStatus.KEY_INVALID;
        }

        // now validate that we were the ones who shipped it
        Signature rsaSignature = Signature.getInstance("SHA1withRSA");
        rsaSignature.initVerify(publicKey);

        FileInputStream sigfis = new FileInputStream(signatureFile);
        byte[] sigToVerify = new byte[sigfis.available()];
        sigfis.read(sigToVerify);
        sigfis.close();

        FileInputStream datafis = new FileInputStream(hashFile);
        BufferedInputStream bufin = new BufferedInputStream(datafis);

        byte[] buffer = new byte[1024];
        int len;
        while (bufin.available() != 0) {
            len = bufin.read(buffer);
            rsaSignature.update(buffer, 0, len);
        };

        bufin.close();

        if (rsaSignature.verify(sigToVerify)) {
            return KeyStatus.KEY_GOOD;
        } else {
            Logger.getLogger().ALog("key failed to pass signature check");
            return KeyStatus.KEY_INVALID;
        }
    }

    /**
     *
     * @param dataToSign
     * @param signatureFilePath
     * @param hashFilePath
     * @throws NoSuchAlgorithmException
     * @throws NoSuchProviderException
     * @throws InvalidKeyException
     * @throws SignatureException
     * @throws FileNotFoundException
     * @throws IOException
     */
    public static void sign(byte[] dataToSign, String signatureFilePath, String hashFilePath) throws NoSuchAlgorithmException, NoSuchProviderException, InvalidKeyException,
            SignatureException, FileNotFoundException, IOException {
        // initialize the signing algorithm with our private key
        Signature rsaSignature = Signature.getInstance("SHA1withRSA");
        rsaSignature.initSign(privateKey);
        rsaSignature.update(dataToSign, 0, dataToSign.length);

        // sign it
        byte[] sig = rsaSignature.sign();

        // save the signature to disk to verify later
        FileOutputStream fos = new FileOutputStream(signatureFilePath);
        fos.write(sig);
        fos.close();

        fos = new FileOutputStream(hashFilePath);
        fos.write(dataToSign);
        fos.close();
    }
}

To perform this audit, the author launched one application at a time and used WireShark to capture and analyze packets. The experiment was performed on an open network that the author created. The access point was a Cisco Small Business router (WAP4410N) and was configured using a hidden SSID and MAC address authentication to prevent outside users from associating with the access point and introducing outside, extra packets. While the author realizes that hidden SSIDs and MAC address authentication are easily defeated mechanisms, it was used to prevent casual users from using the access point. The mobile devices used were an Apple iPod Touch 4G, an Apple iPad 1G and an iPhone 4, configured with iOS 5.0.1.

For reasons of classification, the authors created several different levels of potential security breaches. The levels are defined as:

• None: This level is defined as having no potential security breaches and no exposure of confidential information.

• Low: This level is defined as having a few potential security breaches or exposure of confidential information that could not directly affect the user, such as device IDs that could be used in tracking users (in iOS, these are called UUIDs).

• Medium: This level is defined as having several potential security breaches or exposure of confidential information that is potentially serious or if information is exposed such that an attacker would be able to identify the user on an individual basis, such as addresses, latitudes or longitudes, etc.

• High: This level is defined as having multiple potential security breaches or exposure of extremely confidential information, such as account numbers, PINs, and username/password combinations.

For more information on the specific application, including the version number of the application with the vulnerability, see Appendix A for a full listing.

The majority of the applications that were surveyed encrypted the exchanges of confidential or sensitive information, such as usernames, passwords and account numbers via SSL/TLS.

However, many applications performed some sort of tracking or storing of analytic information, such as passing the UUID in a call to a web service.  In some of the instances, this identifying information was not encrypted.  While not potentially dangerous in the sense that an attacker could use this information to “identify” a particular person, none of the applications let users know that their information such as UUID, phone OS and model, was being used or recorded, nor did they let the user “opt-out.”

The largest single potential security breach was with the Southwest Airlines application.  Due to the fact that the username and password were submitted to a web server via a POST operation in plaintext, an attacker could simply sniff for this data.  If an example was captured, one could use those credentials to log into a particular account and book travel, use award miles and possibly change information in the victims profile.  This not only obviously worrisome from the standpoint of a potential attacker fraudulently using a victims account and credit card information, but also due to the possibility of a terrorists threats in air travel.

For example, consider the possibility of a person who is currently (and rightfully) on the Department of Homeland Security’s “No-Fly” list.  If this person were able to capture a victim’s credentials and create a fake ID, he could pass through TSA security without being stopped.

Of the 253 applications surveyed, 91.7% had no risk found, 3.1% had a low risk, 2.3% had a medium risk and 2.3% had a high risk.  While it would be desirable to have no applications in the “Medium” or “High” category, the number of applications the authors found presented a security risk was both surprising and far too numerous.  There are over 500,000 applications on the iOS App Store, so extrapolating the results, there could be at least 15,500 applications in the “Low” risk category and 11,500 applications in the “Medium” and “High” risk category.

Overall, the number of applications with some sort of security risk is low.  This is not very surprising to the authors as many of these applications are in the “Top” applications list and any potential security flaws would have already been found.

Due to the fact that iOS does not have a robust privilege system, there is no way that a user could know their information was being used in a dangerous or insecure way.  While there is support for showing users there is network traffic by  using a spinning “network activity indicator”, it is certainly not mandatory for them to do so.  In fact, a legitimate or malware application could access the network interfaces, sending and receiving information and never alert the user on iOS.

Developers typically do not follow the principle of least privilege.  If an application needs a set of privileges for functionality, they will request them up front, not just when they are needed.  This is particularly dangerous because this could be an entry point for an attacker to compromise the application.

[19] performed research where they surveyed 940 Android applications and found that more than 50% required 1 extra unnecessary permission and 6% required more than 4 unnecessary permissions.  The reasons that developers may request more permissions than are necessary could be because 1) they don’t understand the importance of security and least privilege, 2) they are planning on future releases that will require these privileges and 3) they don’t fully understand how to work with the platform and make the code function correctly.

Since mobile devices and smartphones are unique in that they have a built-in billing system, there must be ongoing education of developers with emphasis on security and privacy or additional built-in measures in the OS to enforce security over code the developers write or the permissions for which they ask.

Here is the full list of applications tested.

Bold applications represent applications bundled with iOS from Apple.

Abstract— Mobile devices, such as smartphones and PDAs have become increasingly popular with consumers and often provide essential functionality in our everyday life. Usually these mobile devices contain lots of sensitive information, such as addresses, contacts, ingoing/outgoing call logs, SMS messages, and on latest models, a calendar, emails and potentially our current position. A smartphone or mobile device today is as powerful as a desktop or laptop and while the latest models feature a complete OS, for many users these devices are “just phones”, so there is a underestimation of the risk connected to mobile device security. This makes mobile devices an interesting target for malicious users. Damages that a user can sustain are financial loss, privacy and confidentiality, slowdown of processing speed, battery life.

Index Terms—mobile, security, malware, defense

1. INTRODUCTION

Mobile devices, especially cellphones, have changed a great deal from their counterparts from the 1990s. Gone are the days of brick-sized phones, with a 1-line displays, 9 analog buttons and several KB of memory. In recent years there has been an explosion [2] of powerful mobile computing devices. These new smartphones and tablets, small enough to fit in your pocket or backpack, hold an immense amount of computing power. Information is available at a simple touch or finger flick and many users use these devices to access all sorts of data or services such as email, personal contacts, websites and even performing tasks normally reserved for a desktop system, such as video conferencing, watching movies or listening to music.

These devices are able to access the internet, download additional software from the internet, send and receive email, browse websites and send and receive SMS messages from other users. In addition to these capabilities, many subscribers use their cell phone as a primary method of communication, storing their personal contacts information (which include address, email addresses, phone numbers, etc.) as well as photographs they have taken. Additionally, many of these devices have a built-in GPS that allows the user to “geo-tag” photographs and use Location Based Services, such as FourSquare, Twitter and Facebook in addition to the basic mapping and GPS functionality.

Devices that run iOS (iPhone, iPad and iPod Touch), Android and Windows Mobile (which represent the majority of the market share) present a brand new computing paradigm in terms of availability, user interface and security. These devices are being targeted as never before by attackers [1]. Today more than 300 kinds of malware – among them worms, Trojan horses and other viruses and spyware have been unleashed against the devices [1]. Although desktop systems remain the most widely targeted platform, as mobile computing become more ubiquitous and powerful, the lines between a traditional desktop system and a mobile system will become blurred and these devices will gradually enter the virtual battlefield.

Clearly, these new capabilities mixed with the fact that users store personal information on the devices make it a prime target for attackers. There are three different basic categories of attacks that can be carried out against mobile devices and they include:

• Confidentiality attacks: Data theft, data harvesting [3]
• Integrity attacks: Phone hijacking [2]
• Availability attacks: Protocol based Denial-of-Service attacks, battery draining [1]

These three categories represent a wide spectrum of security issues and there are many different attacks that an attacker could carry out. Any of the aforementioned attacks could range in severity from “low” to “high”, which makes this particular research area a wide open problem.

2. Survey Organization

This research paper is organized in the following manner: section III discusses some notable previous work that has been in this particular field, section IV discusses threats, vulnerabilities and defenses of mobile device security and sections V and VI presents some original work of the authors that show current and potential security threats to security from mobile device applications.

3. Previous Work

There has been a significant amount of previous work done in the area of mobile security. In [2], the authors show that since many of the mobile devices are based on similar codebases as their desktop counterparts (as in the case of Windows Mobile and iOS), rootkits, or stealthy malware that affects system programs and files, can equally affect the mobile version of these systems. In addition to showing that the systems can be exploited, they also show the implications of being able to compromise these devices.

In [4], the authors demonstrated the ability to deploy “a Trojan with few and innocuous permissions, that can extract a small amount of targeted private information from the audio sensor of the phone. Using targeted profiles for context-aware analysis, Soundcomber intelligently pulls out sensitive data such as credit card and PIN numbers from both tone- and speech-based inter- action with phone menu systems.”

In [5], the authors show that even though Android has an advanced permissions system with a sandboxed execution environment, a “genuine application [can be] exploited at runtime or a malicious application can escalate granted permissions and imply that Android’s security model cannot deal with a transitive permission usage attack and Android’s sandbox model fails as a last resort against malware and sophisticated runtime attacks.”

In [3], the author shows many of the attacks that occur on iOS have to do with theft or procurement of personal or private information. For example, iOS applications would have access to a user’s address book that includes phone numbers, addresses and email addresses. Additionally, an application can also access other data such as call history, carrier information and photographs.

4. Threats, Vulnerabilities and Defenses to Mobile Device Security

A) Attacks and Vulnerabilities

Since there has been explosive growth in the mobile device market and the demand for smartphones, tablets and other integrated devices have increased dramatically. According to Nielsen [5], “in the third quarter of 2009, smartphones accounted for 40% of new phones sold in the period, up from 25% in the prior quarter. And in the third quarter, for the first time, more people accessed the Internet from smartphones than regular phones. Assuming that 150 million people will be using smartphones by mid-2011, that means 120 million will be on the mobile Internet and 90 million, or 60%, will be watching video” according to Nielsen projections based on current data trends.

In addition to a enormous user base, people are spending more time on their mobile devices than ever before. “As of the second quarter, Nielsen has previously reported that some 15 million U.S. mobile subscribers watch video on their phones for an average of three hours, 15 minutes each month” [5].

Previously attacks such as the ones detailed later in this section haven’t been seen until now. Securing mobile devices present unique challenges to researchers, attackers, defenders and users. This “uniqueness” is due to the fact that computers don’t typically have some the interfaces that mobile devices do, such as GPS hardware, or similar data storage paradigms, such as storage in the cloud. Several potential reasons mobile malware and attacks are becoming more popular is because people are using their mobile devices for more day-to-day tasks such as banking, email and web surfing. This might cause users to store more “valuable” information on these devices and this represents a prime target for attackers due to the wealth of information they could obtain. In addition, while users may have good security habits when it comes to more traditional systems, they may not realize that their mobile device could be just as vulnerable as their servers, desktops and laptops.

I) History and Current Day Data

The first known cell phone virus, Cabir (EPOC.cabir and Symbian/Cabir), occurred in 2004, written by the group 29A [1, 8]. It affected devices that had Bluetooth modules and spread using Bluetooth via OBEX (object exchange). While it was originally designed as a proof-of-concept that would affect Symbian OS devices, it could spread to any Bluetooth enabled device such as desktop computers and printers.

The virus would activate each time the phone was turned on and immediately start looking for other hosts to infect. However, Cabir required the victim user to accept the transfer before any transfer could start. If an attacker were able to masquerade as someone the victim trusted, then they could easily socially engineer their way into infecting the victim’s phone.

Since the virus was released as a proof-of-concept to shock the security community into focusing on the importance of mobile malware, it did nothing other than drain the battery of the infected device while it constantly scanned for new targets to which to send the virus. A variant of Cabir, Mabir, was released that infected phones not only via Bluetooth but also by SMS, so it could use the carrier as an attack vector. This increased the infection potential of this virus exponentially since you didn’t have to be within the 10 meter range of Bluetooth to be infected.

Other historic, notable mobile malware include Duts (a virus for the PocketPC platform), Skulls (a trojan horse that infects all applications, including SMS/MMSes) and Commwarrior (a worm that used MMS messages and Bluetooth to spread to other devices). A more complete list of mobile viruses, trojan horses, worms and malware exist at [1, 7, 8].

Clearly, since mobile devices represent the future of computing, the fact that mobile malware is becoming more prolific shouldn’t be a surprise. In fact, F-Secure reported an almost 400% increase in mobile malware within a two year period from 2005-2007 [2].

Additionally, [9] reports that the amount of Android malware jumped 37% in the third quarter of 2011 and that Android is now the “exclusive platform for all new mobile malware. While the Symbian OS remains the platform with the all-time greatest number of malware, Android is clearly today’s target.” Today, there are currently over 1,200 known malware samples [9].

In [10], the authors performed a survey of mobile malware in the wild. They determined, that between 2009 and 2011, there were 46 pieces of malware released, including 4 for iOS, 24 for Symbian and 18 for Android. They also determined that the most common malicious activities were confidentiality attacks (collecting user information, 61%) and integrity attacks (sending premium-rate SMS messages, 52%).

I) Reasons for an increase in attacks

There are a number of reasons that the community is now experiencing an increase in malware including:

• Increased computing power and storage capabilities: While many consumers may not recognize mobile devices as being equivalent in power to their larger counterparts (laptops and desktop PCs) due to their size, many smartphones, tablet devices and other PDA type devices have a rich set of hardware interfaces. Many smart phones, such as iPhones, Android-based phones, have powerful dual-core processors and a large amount of storage space to accommodate music, movies, documents and other types of media that can be consumed on the go.

In addition, the available software applications that come pre-installed on the device by the device manufacturer, such as web browsers, email clients, messaging applications allow the user to interact much more with the physical and virtual world than previous mobile devices. Furthermore, additional software applications can be downloaded, installed from the Internet and run by the user. These third-party applications are able to access this advanced hardware as well as GPS and network interfaces (3G, WiFi and Bluetooth).

This provides mobile malware and crimeware authors a much larger array of possibilities to carry out their attacks. In addition, more sophisticated hardware and software could make it easier for these attackers to “hide” their attack by ensuring that it only consumes a small portion of the resources, thereby modeling a legitimate application.

• Increased network connectivity: There is a widespread availability of 802.11 WLANs and high-speed broadband data access (3G, WiMAX). These services allow users to stay constantly connected to services such as email and messaging at home, at work and in foreign places, such as coffee shops. In addition, many applications utilize network connections to either request or send data. For example, a game application might transmit a user’s high score to a web server for storage.

Additionally, some applications rely on the fact that the phone will have a network connection to receive and send data, such as the Amazon.com shopping application. Lastly, many recent applications utilize location-based services, such as Facebook, Twitter, and FourSquare. These applications provide additional functionality if they are able to access the network and a user’s current location.

Lastly, many applications can make use of social data, such as the friends one might have on Facebook. This Facebook data could be stored within the application. While Facebook might maintain rigorous security standards on who and what can access the user’s data, other third party applications might not be so careful with the user’s data.

• Standardization of OS and interfaces: The OS is consistent on all the same family of devices, so malware applications would have more effect, being able to exploit the same security vulnerability across many devices. In addition, many manufacturers give third party developers access to the system to write applications for the platform. For example, one can freely download the Android and iOS SDK. Using this provided SDK, one could craft a virus or some other piece of malware and then submit it for inclusion in the respective application storefronts.

• Enterprise integration: Many mobile devices, such as Android, iOS and Blackberry, support standards to be integrated into an enterprise environment. For example, many of the same devices have support for Virtual Private Networks (VPNs) as well as Exchange server integration. Thieves and malware authors recognize that this will greatly enhance the infection potential if a mobile virus, trojan horse or worm is able to spread from a mobile device to a corporate environment.

Consider the case of an employee with a mobile device becoming infected while at a coffee shop and taking that device back to the corporate environment where it could spread throughout the organization. Where previously an attack might have only stolen information pertaining to the particular victim, now the attacker could have potentially obtained information on many different people as well as corporate information.

• Other reasons, social engineering and hacktivism: The number of socially-engineered attacks are becoming more prevalent and more sophisticated. [9] reports “that targeting content works based on cultural and sociological differences between geographic regions. Hacktivism have become part of the mainstream in 2011 due to groups such as Anonymous and LulzSec.”

I) Attack Vectors, Motivations and Types of Attacks

There are a number of attack vectors that exist for compromising confidentiality, integrity or availability. In fact, many of the attack vectors are the same ones that are available to desktop applications. Mobile attack vectors often spread via interfaces and services as well as interfaces unique to smart phones, including SMS and MMS.

While the motivation behind such attacks can be varied and numerous, as well as being outside the scope of this paper, [10] identifies some current and future incentives including:

• Novelty and amusement
• Financial Gain
• Political Gain
• Damage resources

In addition to numerous attack vectors and motivations, there are many different types of attacks that a malicious individual could attempt to carry out. [10] defined three main categories of attacks including 1) malware attacks 2) grayware attacks and 3) spyware attacks.

Malware attacks: “Malware attacks are attacks that gain access to a device for the purpose of stealing data, damaging the device, annoying the user, etc. The attacker defrauds the user into installing the malicious application or gains unauthorized remote access by taking advantage of device vulnerabilities. These particular types of threats provide no notice to the user and typically includes worms, Trojan horses and viruses [10].”

• RF attacks: In this particular type of attack, an attacker could compromise confidentiality, integrity and availability. For example, if an attacker were to have the correct equipment, the attacker could sniff the air (WLAN and RF) for user data that the attacker could steal if it were unencrypted, such as usernames, passwords and account numbers. This particular type of attack is made easy if: 1) the network the user is on doesn’t utilize encryption and, 2) the application transmits confidential information in plaintext.

As we’ll see later on in section V, there are a number of iOS applications today that transmit private information without encryption. Additionally, if an attacker were able to successfully carry out a “mis-association” attack where the mobile device joins the wrong access point, the attacker could attack the integrity of communications by performing a “man-in-the-middle” attack, if the data were not validated or signed. Consider an example where a victim user sent a SMS message that said “Transfer $1000 to account X.” The attacker could alter the SMS to say “Transfer $10,000 to account Y”, where Y was his account.

Lastly, an attacker could also compromise the availability of RF services. Consider an example where an aircraft was WiFi enabled (passengers could access the WiFi during flight). An attacker could carry out a “disassociation” attack, where he transmits 802.11 management frames that cause the wireless clients to disassociate from the access point. The denial-of-service would continue until the attacker stopped transmitting the packets.

• Bluetooth attacks: This particular class of attack has many different possible different attacks, similar to RF attacks, that can compromise confidentiality and integrity of data. In “blue-jacking”, an attacker’s malware application could insert contacts or SMS messages into a victim user’s mobile device. Additionally, in “blue-snarfing”, a user’s data is again under attack by allowing the attacker to steal or transfer the victim’s data. Yet another type of bluetooth attack is “blue-tracking” where an attacker could follow the victim’s movements. Lastly in “blue-bugging”, an attacker could listen in on conversations by activating the attack software and having the phone call them back. When the attacker answers the call back, the attacker would then be able to listen in to a conversation.

• SMS attacks: “SMS spam is used for commercial advertising and spreading phishing links. Commercial spammers are incentivized to use malware to send SMS spam because sending SMS spam is illegal in most countries” [10]. In addition to sending and receiving regular-rate SMS messages, SMS can also be used as an attack vector by exploiting vulnerabilities in the software stack, such as performing SMS fuzzing.

• GPS/Location attacks: In this type of attack, the attacker can access the GPS hardware to monitor the user’s movement and current location. This data can be used to create a profile of a particular user. In addition, this information could be sold to to other companies for purposes of marketing and sending advertisements to the user. A more insidious use of a GPS/location attack would allow a criminal to track when a victim leaves their residence and then the attacker could rob the person while they are gone.

• Application masquerading and personal data attacks: This particular type of attack is as simple as accessing a user’s private data and saving, sending or using it in an unauthorized manner. For example, in [3], the author shows multiple examples of information that an application could access that could potentially be sensitive or confidential, such as a user’s phonebook, keyboard and location cache and photo albums.

In addition, [3] also showed that it would be relatively trivial for an application to traverse the file system of a mobile device, recording information that it could find valuable. This can all occur in the background and the user would never know that it is happening. This data could be sent off to a server and stored for later misuse.

• Phone “Jailbreaking” and 3rd party application stores: While not encouraged by the manufacturer, (and in some cases against the carrier terms and agreements), many users prefer to “jailbreak” their mobile devices. Jailbreaking is the process of removing the security limitations that are imposed by the operating system, such as the ability to only run signed applications and install additional extensions and themes. This also allows the user to bypass application sandboxing mechanism and install applications from unofficial application stores. Users find this desirable because they can add functionality that didn’t previously exist or could get from any existing application on the application stores. However, users may not realize that this could potentially be a problem if they were to install a malicious application that would normally be killed by the OS on an un-jailbroken device. For example, one of the available exploits for iOS is when a user jailbreaks their iDevice, installs the OpenSSH package and doesn’t change the root password for the device, which is “alpine”. This would allow a hacker to login to their device as the root user and exploit the system because they would have full access to read, write and execute any command.

• Premium-rate attacks: Premium rate services can deliver valuable content to a user’s mobile device. When used in a legitimate manner, a user could receive financial information, technical support or even adult services. These services can cost as much as several dollars per message or minute. In [10], they identified 24 of the 46 pieces of malware surveyed as sending premium-rate SMS messages. In one piece of malware, [10] found that applications purporting to be a Russian adult video player sent premium-rate SMS to an adult service.

Another piece of malware, Geinimi, sent premium SMS messages to numbers specified by the remote command servers. This is potentially a very large security concern, because these premium SMS messages don’t require a user’s permission to send, so they potentially could go unnoticed until an attacker has racked up hundreds or even thousands of dollars on a victim user’s phone bill. In addition to premium-rate SMS message attacks, [10] also found that 2 of the 46 malicious applications contained premium-rate phone call attacks.

• Power management attacks: [11] describes three different classes of power management attacks. A power management attack is a form of a denial-of-service attack that affects the availability of the mobile device by draining the battery more quickly than it would under normal operation. The attacker’s goal is to maximize the difference in power consumption between active and sleep states and keep the device from sleeping.

The three classes of attacks include 1) service request attacks 2) benign power attacks 3) malignant power attacks. In service request attacks, repeated requests are made to the victim for services, typically over a network. Even if the request is ultimately not granted, the power must be expended by the device to determine whether or not to grant the service request. In benign power attacks, the mobile system executes valid, but power hungry tasks (such as displaying a hidden animated gif or executing a JavaScript). In malignant power attacks, attackers create or modify binary executables that force the mobile system to consume more power than it normally would. For example, consider an application that plays a silent audio track in the background while the application is running. This type of attack might be difficult for the user to detect because they may think that their battery can no longer hold a charge.

• Time-activated and location-activated attacks: An attacker may choose to activate attacks at certainly locations or at a pre-determined or random time in the future. When the victim arrives at the location and uses the software, it could activate whatever is the intended malignant function. This would be fairly easy to do as many applications have access to the GPS. This type of attack may also sneak by any analysis of the application as it wouldn’t run every time the application was launched, but rather only under certain conditions.

In addition to malware attacks, [10] also defines several other types of “attacks” including:

Grayware attacks: In [10], the authors classified grayware attacks as “some legitimate applications collect user data for the purpose of marketing or user profiling. Grayware spies on users, but the companies that distribute grayware do not aim to harm users. Pieces of grayware provide real functionality and value to the users. The companies that distribute grayware may disclose their collection habits in their privacy policies, with varying degrees of clarity. Grayware sits at the edge of legality; its behavior may be legal or illegal depending on the jurisdiction of the complaint and the wording of its privacy policy. Unlike malware or personal spyware, illegal grayware is punished with corporate fines rather than personal sentences. Even when the activity of grayware is legal, users may object to the data collection if they discover it. Application markets may choose to remove or allow grayware when detected on a case-by-case basis.”

For example, in 2009, bloggers raised concern over the PinchMedia LLC analytics framework. This framework provided third party application developers usage information about their users. The users were not informed and they were not given an option to opt-out [3]. In addition, Storm8 and MogoRoad also faced legal issues when it was discovered that they were collecting users’ contact information without informing them and then transmitting the collected information in plaintext [3].

Spyware attacks: The last category that [10] detailed was “Spyware attacks.” “Spyware collects personal information such as location or text message history over a period of time. With personal spyware, the attacker has physical access to the device and installs the software without the user’s knowledge. Personal spyware sends the victim’s information to the person who installed the application onto the victim’s device, rather than to the author of the application. For example, a person might install personal spyware onto a spouse’s phone. It is legal to sell personal spyware in the U.S. because it does not defraud the purchaser (i.e., the attacker). Personal spyware is honest about its purpose to the person who purchases and installs the application. However, it may be illegal to install personal spyware on another person’s smartphone without his or her authorization [10].”

For more on platform specific exploits, in the papers we surveyed, please reference [1, 2, 3, 4, 6, 10, 11, 18, 20].

A) Defenses

When defending against mobile phone security threats and mobile malware, there are two main categories: prevention and detection and recovery. The majority of the work that has been already implemented falls into the class of prevention. However, detection and recovery is becoming a more popular research topic. We discuss some of the major defenses below:

• Code analysis (static and dynamic): There are two main techniques of determining an application’s characteristics: statically and dynamically. Both have certain advantages and disadvantages.

In static analysis, many techniques may be used to determine how the program works, such as decompilation, decryption, pattern matching and static system call analysis. The central idea behind static analysis is finding signatures of malicious code in a fast and easy manner. In [3], the author describes part of the App Store review process for applications and detail that using static analysis, one can dump the strings in a binary file and check them against a black list of forbidden classes, method names and file paths. However, static analysis can be circumvented with obfuscation techniques. Additionally, some languages, such as Objective-C allow developers to lookup classes and methods by name at runtime. This feature of the language adds additional opportunity to “hide” malware functionality.

The primary disadvantage of static code analysis is that malicious code patterns have to be known in advance, making it impossible to automatically detect new malware or malicious polymorphic code without an intervention of a human expert [23].

Dynamic analysis is a set of techniques which involve running an application in a controlled environment and monitoring its behavior. Various heuristics can be used to capture behavior of the application such as monitoring file changes, network activity, processes, threads and system call tracing [23].

One large difference between the iOS App Store and the Android Marketplace is that applications that are released to the iOS App Store are reviewed by a human. This accomplishes an important goal in that it rejects applications that are in a “legal gray zone such as casino gambling or collecting personal data.” The Android Marketplace allows an author to post apps without needing any a priori review but rather it relies on “crowd-sourcing” to review the applications and post comments, either positive or negative.

• ASLR and DEP: ASLR (address space layout randomization) is a computer security method that involves re-arranging the positions of key data areas, including the position of libraries, the heap and the stack in the process’ address space. This technique makes it more difficult for an attacker to predict target addresses [21]. For example, an attacker who is attempting to overflow the stack, would first have to locate the address of the stack before they could overflow it. If an attacker were to guess incorrectly and the program terminated, the next time the program were launched, the stack address will have moved again and the attacker would have to start all over again in their attempt to find it. While the full description of how ASLR is implemented and its effectiveness is outside the scope of this paper, the reader may learn more at [21].

In addition to ASLR, DEP (data execution prevention) is another security feature that prevents an application or service from executing code from a non-executable memory-region. This protection can be enforced in hardware and/or software. Again, while the full description of how DEP is implemented and its effectiveness is outside the scope of this paper, the reader may learn more at [22].

The platforms that support ALSR and DEP include iOS and Windows Mobile. Android has plans to support ASLR but the Blackberry platform has neither.

• Application sandboxing: “Sandboxing consists of running mobile code in a restricted environment called a sandbox [17].” A sandbox can be characterized by two different mechanisms: 1) confining code, either through type checking, language properties or the use of protection domains to prevent the subversion of trusted code and 2) enforcing a fixed policy for the execution of code [17].

When applied to the real world, each application will have its own environment. Other applications should not be able to interfere with another applications environment nor should that particular application be able to interfere with other applications’ environments. Both iOS and Android implement application sandboxes. In the iOS model, each application only has read-write access to a few directories (the application’s own directory and a temporary directory) and applications cannot read or write to any other directories (including other applications and system directories) than for which it is authorized.

In the iOS sandbox, all applications share the same sandbox rules and they’re allowed any action any application could ever need [18]. Compared to the Android sandbox, “applications must explicitly share resources and data and do this by declaring permissions they need for additional capabilities not provided by the basic sandbox. These additional permissions are granted or denied by the user at install time only. [13]”

• Permission systems: In Android, there is the additional protection of a permission system. This permission system “treats all applications as potentially buggy or malicious so they are assigned a low-privilege ID that can only access their own files [19].” Depending on what the application wants to do, it can request an elevation of permissions from the user, at install-time, which the user will ultimately grant or deny.

For example, there are several different levels of permissions including Normal (permissions that protect access that could annoy but not harm the user, such as setting the wallpaper), Dangerous (permissions that protect access that potentially harm the user such as gathering private information), and System (permissions that need access to the most dangerous privileges, such as deleting applications).

In addition to the permissions system, Android also has the Intents system. Intents are typed interprocess messages that are directed to particular applications or systems services, or broadcast to applications subscribing to a particular intent type [20]. Access to the Intents system is funneled through ActivityManagerService which restricts intents only being sent by applications with the appropriate permissions and processes with a UID that match the systems [19].

iOS does have a rudimentary concept of a permissions system but not one that is as in depth as the Android model. For example, when an application attempts to access the user’s location via the GPS hardware, the application will confirm with the user that this is an acceptable action, which the user can confirm or deny. However, for the most part, Apple relies on a number of other techniques to handle security issues, such as Sandboxing, ASLR/DEP and code analysis.

• ACLs and capability lists: Both iOS and Android implement standard UNIX type permissions with users and groups. This allows the systems to implement the concept of “least privilege” where accounts and users are only able access data to which they are properly privileged to access. In iOS, files have permissions and third party applications runs as the user “mobile”, instead of root. In addition, certain operations on Android require the proper permissions, such as accessing as network interfaces. Typically, when a user “roots” or “jailbreaks” their device, they are elevating the permissions of the application to that of a more privileged account to bypass certain security features such as sandboxing.

• Code signing: Code signing is a process where executable binaries are signed digitally by software authors to guarantee that the code that has not been altered or corrupted [15]. This is an important part of the defense systems in both the Android, Blackberry, Windows Mobile and iOS security model and it is used extensively in both models for validating third party applications. An important distinction between the two is that with iOS code signing, the code must be signed with a certificate validated by a certificate authority or CA (in this case, Apple), whereas with Android, self-signed certificates are acceptable [13, 14]

In addition to validating applications, iOS also uses code signing when booting up. When a device running iOS boots up, the first significant piece of code that runs on a device running iOS is the BootROM or the “SecureROM”, which is read-only [16]. Within this BootROM, an Apple root certificate is embedded such that the firmware can be validated as being official and secure. Once the RSA signature has been checked, control is passed to the low level bootloader or the “LLB.” This module runs several setup routines and checks the signature of iBoot, which is a stage 2 bootloader for all devices running iOS. Once iBoot starts, it allows the device to go into a “recovery” or “DFU” mode that allows devices to be restored from any state. Firmware images from Apple are signed and checked when the upgrade or downgrade occurs. If the device boots normally instead of going into DFU mode, control is passed to Launchd, CommCenter, and Springboard [11].

In addition to validating system software, iOS also signs the code directory structure with SHA-1 hashes of memory pages, and a PKCS#7 signature is embedded in all binaries that are downloaded from the App Store [11]. For binaries that do not validate at run time, they are killed by the OS to prevent any binary that has been altered from running. Lastly, to provide a defense against rootkits, for system binaries, code directories hashes are cached in the kernel [11].

While regular users may not know or care whether a binary is signed properly, this particular process creates accountability and increased trust on the platform if an application can be traced back to a known source.

The astute reader will recognize that a signed binary may not necessarily contain safe or bug-free code – just code verified as coming from a particular known source and has not been subject to tampering. Also, if the system does not strictly enforce running only validated binaries, users could be tricked (through social engineering perhaps) into running code that refuses to validate. Lastly, one potential flaw with Google’s scheme is that Google does not require that a CA sign the application signing certificate – they only record the signatures for book keeping purposes. This additional level that Apple introduces could potentially reduce the amount of malware and spyware on Android.

• Data encryption: Data encryption and decryption is a very CPU and power intensive activity. Battery life on a mobile device is a very precious commodity because the usage model for a mobile device is such that the devices are meant to be used out in the field where a constant power source isn’t always available. As such, there hasn’t been any commercial implementations of automatic whole disk encryption. However, certain OSes, such as iOS, encrypts particular items such as the Keychain (a password manager and storage framework) and provides the ability to encrypt individual files. Another option for third party applications is to provide their own encryption services framework, using known models, such as PKI, and known programs, such as OpenSSL.

One interesting study of mobile device encryption was done by the authors of [28]. In their research, they explored the use of Field Programmable Gate Arrays or FPGAs, processors and ASIC hardware in the context of finding a framework for encryption on hand-held communication units. They used the IDEA encryption algorithm to show the tradeoffs in the suggested technologies. They measured their results using three different metrics: 1) performance, 2) programmability and, 3) power consumption. They determined that since power consumption is directly related to frequency, FPGAs provided the highest performance (MOPS/watt).

• Detection and recovery defenses: There has been a lot of notable work done in this category of mobile malware defense. For example, [24] presented various approaches for mitigating malware on mobile devices. The authors implemented and evaluated the suggested approaches on Google Android. The work is divided into the following three segments: a host-based intrusion detection framework; an implementation of SELinux in Android; and static analysis of Android application files.

[24] determined that to provide well-rounded protection, a security suite for mobile devices or smart phones (especially open-source ones such as Android) should include a collection of tools blending various capabilities that operate in synergistic fashion.

In [24], the author’s first approach was an innovative host-based intrusion detection system (IDS) for detecting malware on mobile devices. This framework relies on a lightweight agent (in terms of CPU, memory and battery consumption) that continuously samples various features on a device, analyzes collected data using machine learning and temporal reasoning methods and infers the state of the device. Features belonging to groups such as Messaging, Phone Calls and Applications belong to the Application Framework category and were extracted through APIs provided by the framework; features belonging to groups such as Keyboard, Touch Screen, Scheduling and Memory belong to the Linux Kernel category.

This study on anomaly detection was based on various detection algorithms. The purpose of this study is to understand how a detection algorithm, a particular feature selection method and number of top features can be combined to differentiate between benign and malicious applications which are not included in the training set, when training and testing are performed on different devices and to find specific features that yield maximum detection accuracy. Empirical results suggest that the proposed framework is effective in detecting malware on mobile devices in general and on Android in particular (accuracy of 87.4% with false positive rate of 12.6%).

The author’s second study examined the applicability of detecting malware instances using a light version of the Knowledge-based Temporal Abstraction (KBTA) method that can be activated on resource-limited devices. The new approach was applied for detecting malware on Google Android powered-devices. Evaluation results demonstrated the effectiveness of the new approach in detecting malicious applications on mobile devices (detection rate above 94% in most scenarios) and the feasibility of running such a system on mobile devices (CPU consumption was 3% on average).

This study also proposed the implementation of SELinux in Android in order to harden the Android system and to enforce low-level access control on critical Linux processes that run under privileged users. By choosing this route, the system can be better protected from scenarios in which an attacker exploits vulnerability in one of the high privileged processes.

[25] is another interesting study using malware behavior detection. This paper proposed a behavior-based malware detection system for Windows Mobile platform called WMMD (Windows Mobile Malware Detection system). WMMD uses API interception techniques to dynamically analyze an applications behavior and compare it with a malicious behavior characteristics library using model checking.

The architecture of the proposed framework consists of two modules: a Dynamic Analysis Module and a Behavior Detection Module. Both modules use the API interception technique to obtain software running information. The Dynamic Analysis Module is responsible for analyzing the program’s behavior. The Behavior Detection Module monitors the process’ real time information and compares it with abehavior signature library. Once it detects a mal-behavior, it responds to the user and offers a feedback to construct new behavior model.

All the experiments were done first in Windows Mobile Emulator in PC and then verified in a real mobile phone. The Emulator is Windows Mobile 6.0 professional version and the real mobile phone is HTC PPC6800 with Windows Mobile 6.0 OS.

To test WMMD’s effectiveness towards obfuscation or packing techniques, they used UPX packer to pack five Windows Mobile viruses, and compare WMMD to six other anti-virus software packages (Windows Mobile 6.0) which included an updated virus signatures database. Testing results revealed that all of the other anti-virus products can only detect the virus before packing and they are useless when the virus was packed. However, the WMMD can detect those viruses after packing, since WMMD checks the API call in real execution which cannot be changed. For example, WinCE.Infojack.A is a trojan that binds to popular software installation files and it will extract a file named mservice.exe to the \Windows directory and create mservice.lnk file to the \Windows\StartUp directory. It then start this mservice.exe process. It is apparent that other variants of WinCE.Infojack.A have the same behavior and thus they only need to monitor the CreateFile() and CreateProcess() APIs to check whether their arguments satisfies the specific behavior.

This evaluation on real-world mobile malware shows that behavioral detection can successfully detect malware variants which have certain behavioral patterns with existing patterns in the database, while other anti-virus product cannot detect.

In [26], the authors describe that malware doesn’t always need to be physically installed on the phone to affect the mobile device as they considered defending mobile devices against “proximity malware.” The dynamics of proximity propagation inherently depend upon the mobility dynamics of a user population in a given geographic region. Unfortunately, there is no ideal methodology for modeling user mobility. Traces of mobile user contacts reflect actual behavior, but they are difficult to generalize and only capture a subset of all contacts due to a lack of geographic coverage. Analytic epidemiological models are efficient to compute and scale well, but simplify many details. Synthetic models are flexible and provide the necessary geographic coverage, but lack the full authenticity of user mobility traces.

[26] assumed that devices have a trusted defense software component that can examine messages and files transferred between devices, securely record persistent information about these transfers, and control device hardware when necessary (e.g., disable radio communication). These assumptions may be strong, but not unreasonable given the increasing prevalence of trusted computing modules. However, if malware has the ability to disable defense software, we can predict what the result will be: unchecked propagation through a population.

The first strategy described in [26] simply uses local evidence to detect malware and prevent further dissemination by the device, such as by disabling the Bluetooth or WiFi radio. Preventing further propagation by disabling communication may inconvenience the user, but voice and messaging with the provider network remain possible. Disabling the malware prevents further propagation but makes no attempt to notify other devices or the network about the presence of malware. It serves as a useful baseline for comparison.

The second strategy described in [26] extends local detection with an active mitigation component. In this strategy, each device maintains a table S of signatures of malware files, such as an MD5 hash over the file content. After a device X infers that it is infected, it disables the malware and warns subsequent devices about it. Device X computes a content-based signature s over the file(s) that triggered the infection recognition (e.g., the hash it has used to track file transfers in the first place). When X comes into proximity contact with another device Y, X disseminates the signature s to Y . If Y is infected, it immediately disables the malware. Y then adds s to its signature table S. Whenever another device shares a file with Y, Y will check the file against the signatures in S. The device can then either delete the file, or warn the user about the file.

The third strategy described in [26] relies upon the network provider to disseminate signatures using a broadcast mechanism. In addition to standard unicast messaging, providers are also able to send data packets over broadcast at a low cost. In this strategy, whenever a device decides that it is infected, it sends the malware content to an anti-virus server in the provider network (using MMS). The server, since it presumably contains far greater processing power than the mobile devices, can compute a better quality signature. Also, due to access to anti-virus experts, the server may also be able to compute a patch that contains information on how devices may “cure” themselves and remove the infection from the device. Manual involvement in generating patches is also a possibility.

Lastly, in [27], the authors used data mining techniques to detect malware behavior. This paper proposed a technique of ontology-based behavioral analysis to develop a detection method for smart phone malware. In this experiment, a mobile environment is constructed in the laboratory. The HTC HD2 smart phone with Windows mobile 6.5 operation system was adopted as the main test platform. Then they installed and ran their mobile malware detection system (MMDS) on an HTC HD2 smart phone. Then the other smart phones sent files or messages to the HTC HD2 through MMS or SMS. The MMDS can automatically filter all files and message by extracting their behavioral characteristics. The system will determine the degree of danger of these behaviors. When users have confirmed that this message is in danger of intrusion, the system will refuse the MMS or SMS.

As a result of the experiment in [27], the proposed FPN model can detect most of new mobile malware. However, there exist two pieces of mobile malware that cannot be detected by the FPN model. Since the collection of mobile malware is difficult, one cannot gain various types of mobile malware to test. Thus, the FPN model through the behavior analysis of mobile malware based on the ontology theory may not detect the above-mentioned mobile malware.

I) Future Defense Work

The authors would also like to explore the feasibility of implementing other security features such as:

• Enhanced permissions systems: It would be worthwhile to spend time researching how one could detect over-privileged applications and adjust their privilege level for only the privileges that are needed. Additionally, research into a permission models that allow finer-grain control on application permissions would allow developers and users to control exactly what information or modules could be used or accessed. Lastly, research into dynamic analysis models to determine malicious activity would be useful as mobile platforms and applications are becoming more complicated and future attacks could possibly target push notification systems and in-application purchasing systems.

• Trusted computing modules: Trusted computing is a technology that ensures that a computer will consistently behave in an expected way and that those behaviors will be strictly enforced by hardware and software. If these modules could be included as hardware on the phone, we could ensure that hackers couldn’t deploy rootkits on phones because the boot process would be verified and secure.

• Encryption modules: Many desktop operating sytems implement some form of full hard drive encryption, such as Windows BitLocker and Mac OS X FileVault. Using a hardware encryption module, it would be possible to encrypt the entire hard drive without consuming a large amount of battery power. For example, the user application space could be encrypted in a separate volume that is mounted and decrypted at boot time. In addition to encrypting the entire user application space, applications could also provide their own separate encryption keys to do encryption of application specific data.

• Firewall modules: Since smartphones and mobile devices are now as powerful as desktop computers, and the trend of consumers using them as a replacement to desktop and laptop systems will continue, more personal and confidential data will be stored on mobile platforms. It would be valuable to invest research time into mobile firewalls and packet filtering in attempt to possibly detect whether or not data harvesting is occuring on the device and being transferred across a network interface.

5. Applications Survey

The author surveyed over 230 applications (the full list of applications ,can be found in Appendix A), including applications in the “Top” categories on the iTunes store to determine what type of information could be extracted from auditing packet streams. The results were quite surprising.

To perform this audit, the author launched one application at a time and used WireShark to capture and analyze packets. The experiment was performed on an open network that the author created. The access point was a Cisco Small Business router (WAP4410N) and was configured using a hidden SSID and MAC address authentication to prevent outside users from associating with the access point and introducing outside, extra packets. While the author realizes that hidden SSIDs and MAC address authentication are easily defeated mechanisms, it was used to prevent casual users from using the access point. The mobile devices used were an Apple iPod Touch 4G, an Apple iPad 1G and an iPhone 4, configured with iOS 5.0.1.

For reasons of classification, the authors created several different levels of potential security breaches. The levels are defined as:

• None: This level is defined as having no potential security breaches and no exposure of confidential information.

• Low: This level is defined as having a few potential security breaches or exposure of confidential information that could not directly affect the user, such as device IDs that could be used in tracking users (in iOS, these are called UUIDs).

• Medium: This level is defined as having several potential security breaches or exposure of confidential information that is potentially serious or if information is exposed such that an attacker would be able to identify the user on an individual basis, such as addresses, latitudes or longitudes, etc.

• High: This level is defined as having multiple potential security breaches or exposure of extremely confidential information, such as account numbers, PINs, and username/password combinations.

For more information on the specific application, including the version number of the application with the vulnerability, see Appendix A for a full listing.

The majority of the applications that were surveyed encrypted the exchanges of confidential or sensitive information, such as usernames, passwords and account numbers via SSL/TLS.

However, many applications performed some sort of tracking or storing of analytic information, such as passing the UUID in a call to a web service. In some of the instances, this identifying information was not encrypted. While not potentially dangerous in the sense that an attacker could use this information to “identify” a particular person, none of the applications let users know that their information such as UUID, phone OS and model, was being used or recorded, nor did they let the user “opt-out.”

The largest single potential security breach was with the Southwest Airlines application. Due to the fact that the username and password were submitted to a web server via a POST operation in plaintext, an attacker could simply sniff for this data. If an example was captured, one could use those credentials to log into a particular account and book travel, use award miles and possibly change information in the victims profile. This not only obviously worrisome from the standpoint of a potential attacker fraudulently using a victims account and credit card information, but also due to the possibility of a terrorists threats in air travel.

For example, consider the possibility of a person who is currently (and rightfully) on the Department of Homeland Security’s “No-Fly” list. If this person were able to capture a victim’s credentials and create a fake ID, he could pass through TSA security without being stopped.

Of the 253 applications surveyed, 91.7% had no risk found, 3.1% had a low risk, 2.3% had a medium risk and 2.3% had a high risk. While it would be desirable to have no applications in the “Medium” or “High” category, the number of applications the authors found presented a security risk was both surprising and far too numerous. There are over 500,000 applications on the iOS App Store, so extrapolating the results, there could be at least 15,500 applications in the “Low” risk category and 11,500 applications in the “Medium” and “High” risk category.

Overall, the number of applications with some sort of security risk is low. This is not very surprising to the authors as many of these applications are in the “Top” applications list and any potential security flaws would have already been found.

Due to the fact that iOS does not have a robust privilege system, there is no way that a user could know their information was being used in a dangerous or insecure way. While there is support for showing users there is network traffic by using a spinning “network activity indicator”, it is certainly not mandatory for them to do so. In fact, a legitimate or malware application could access the network interfaces, sending and receiving information and never alert the user on iOS.

Developers typically do not follow the principle of least privilege. If an application needs a set of privileges for functionality, they will request them up front, not just when they are needed. This is particularly dangerous because this could be an entry point for an attacker to compromise the application.

[19] performed research where they surveyed 940 Android applications and found that more than 50% required 1 extra unnecessary permission and 6% required more than 4 unnecessary permissions. The reasons that developers may request more permissions than are necessary could be because 1) they don’t understand the importance of security and least privilege, 2) they are planning on future releases that will require these privileges and 3) they don’t fully understand how to work with the platform and make the code function correctly.

Since mobile devices and smartphones are unique in that they have a built-in billing system, there must be ongoing education of developers with emphasis on security and privacy or additional built-in measures in the OS to enforce security over code the developers write or the permissions for which they ask.

6. Using Mobile Devices as Network Monitors

We also researched the possibility of using WiFi sniffing and cracking utilities on the iPhone, as well as the feasibility of releasing a spyware application into the iTunes App Store and collecting user information. As a side note, all the techniques used here could easily be ported to other mobile platforms, such as Android, but this particular research focused on iOS devices.

To be able to perform basic packet sniffing, there are several critical elements that must be performed. The first element is to be able to put the particular interface into a “promiscuous” mode, in the case of wired network interfaces or “monitor” mode, in the case of wireless network interfaces.

In normal network interface operation, the kernel will discard any packets that are not destined to the specific node. Using this “monitor” mode, we are able to capture and further analyze these packets that would normally be discarded. This was a particularly easy piece of code (see Appendix B) to write (if one understands the BSD subsystem and C). In our particular implementation, we have a utility function to toggle the “promiscuous” or “monitor” bit (IFF_PROMISC) in the command word flag (SIOCGIFFLAGS) as well as a function to list all the interfaces that the OS knows about.

In addition to being able to manipulate the firmware of the network card, we also need to be able to access the bpf files or Berkely Packet Filter devices that are located in /dev. These bpf devices “provide a raw interface to data link layers in a protocol independent fashion. All packets, even those destined for other hosts, are accessible through this mechanism [29].”

The access to these files are restricted to root only. This presents a problem for as root access is restricted for applications that will be available in the App Store. Therefore, to install and distribute this application, one must have a jailbroken iOS device.

To give the programmer a more friendly access to the raw data from the bpf device and the packets, a user space program, libpcap (Unix or Linux) or WinPCap (Windows), is bolted onto these kernel-only devices. Interestingly, the authors discovered that the SDK for Mac OS included a pre-built library, libpcap.dylib. Unfortunately, that library is not available natively for iOS, but it can be cross-compiled for the arm architecture by downloading the source from www.tcpdump.org.

Lastly, to perform useful functions with this data, one must create an interface to analyze, extract and filter packet information at the application level. While, a .pcap file could be created for later analysis, it might not be as useful as having live analysis. This can easily be done with a user space program such as tcpdump or WireShark.

This particular research shows there could be a major potential data confidentiality problem with mobile devices. Assuming that the device is jailbroken and if we were able to release an app into the Cydia App Store that a user could download, we could silently harvest and store personal and sensitive information from that particular user, in addition to any other device that is on an open wireless network (assuming that the user was connected), without alerting the user. In addition the being able to sniff packets, since we are in the Cydia App Store, we would have full use of all APIs, including Apple private APIs to harvest personal information.

One question that the authors would like to explore in future research is if we could also use the aircrack-ng suite on mobile devices.

7. Conclusion

In this paper, we examined the history of mobile threats and vulnerabilities as well as current threats and vulnerabilities against mobile devices. We also researched and examined defense mechanisms that currently exist and proposed future research topics. Additionally, we performed two experiments, one as an audit of mobile application security and the other as the feasibility of turning a mobile device into a RF sniffing and data collection device.

Appendix A: Full Application List

Bold applications represent applications bundled with iOS from Apple.

Appendix B

#include 

#include 

#include 

#include 

#include 

#include 

#include 

#include 

#include 

int go_promisc(int);

int get_iface_list(struct ifconf *);

int get_iface_names(void);

int go_promisc(int on) {

int fd;

struct ifreq *ifreq = (struct ifreq *)malloc(sizeof(struct ifreq));

get_iface_names();

strcpy(ifreq->ifr_name, "en0");

fd = socket(AF_INET, SOCK_STREAM, 0);

if(fd < 0) {

perror("opening socket");

return -1;

}

int status = ioctl(fd, SIOCGIFFLAGS, ifreq);

if(status < 0) {

perror("ioctl(SIOCGIFFLAGS)");

status = -1;

}

if(on) {

ifreq->ifr_flags |= IFF_PROMISC;

ifreq->ifr_flags |= IFF_ALLMULTI;

}

else {

ifreq->ifr_flags &= ~IFF_PROMISC;

ifreq->ifr_flags &= ~IFF_ALLMULTI;

}

status = ioctl(fd, SIOCSIFFLAGS, ifreq);

if(status < 0) {

perror("ioctl(SIOCSIFFLAGS)");

status = -1;

}

close(fd);

return status;

}

int get_iface_list(struct ifconf *ifconf) {

int sock, rval;

sock = socket(AF_INET, SOCK_STREAM, 0);

if(sock < 0) {

perror("opening socket");

return (-1);

}

if((rval = ioctl(sock, SIOCGIFCONF, (char *)ifconf)) < 0) {

perror("ioctl(SIOGIFCONF)");

}

close(sock);

return rval;

}

int get_iface_names() {

static struct ifreq ifreqs[20];

struct ifconf ifconf;

int  nifaces, i;

memset(&ifconf, 0, sizeof(ifconf));

ifconf.ifc_buf = (char*) (ifreqs);

ifconf.ifc_len = sizeof(ifreqs);

if(get_iface_list(&ifconf) < 0) {

return -1;

}

nifaces =  ifconf.ifc_len / sizeof(struct ifreq);

printf("Interfaces (count = %d)\n", nifaces);

for(i = 0; i < nifaces; i++) {

printf("\t%-10s\n", ifreqs[i].ifr_name);

}

}

int main(int argc, char *argv[]) {

if(argc != 2) {

printf("usage: promisc [ON | OFF]");

return -1;

}

get_iface_names();

if(strcmp(argv[1], "ON") == 0) {

return go_promisc(1);

}

else if(strcmp(argv[1], "OFF") == 0) {

return go_promisc(0);

}

else {

printf("usage: promisc [ON | OFF]");

return -1;

}

return 0;

}

References

[1] M. Hypponen, “Malware Goes Mobile”, November 2006, Scientific American Magazine. Pages 70–77. http://www.cs.virginia.edu/~robins/Malware_Goes_Mobile.pdf

[2] J. Bickford, O. O'Hare, A. Baliga, V. Ganapathy, and Iftode. L, “Rootkits on Smartphones: Attacks, Implications and Opportunities”. ACM, In the Workshop on Mobile Computing Systems and Applications. Annapolis, MD. Feb. 2010.

[3] N. Seriot, “iPhone Privacy”, Black Hat DC 2010. Arlington, Virginia, USA. http://seriot.ch

[4] R. Schlegel, K. Zhang, X. Zhou, M. Intwala, A. Kapadia, X. Wang, "Soundminer: A Stealthy and Context-Aware Sound Trojan for Smartphones”. In Proceedings of the 18th Annual Network & Distributed System Security Symposium (NDSS) Feb. 2011.

[5] J. Rocha. “The Droid: Is this the smartphone consumers are looking for?”. Nov. 2011. http://blog.nielsen.com/nielsenwire/consumer/the-droid-is-this-the-smartphone-consumers-are-looking-for/

[6] A. Bose. “Propagation, detection and containment of mobile malware”. 2008. http:/hd1.handle.net/2027.42/60849

[7] http://www.symbianpoint.com/types-latest-list-mobile-viruses.html

[8] A. Gostev. “Mobile Malware Evolution: An Overview, Part 1”. Sep. 2006. http://www.securelist.com/en/analysis?pubid=200119916

[9] McAfee. “McAfee Threats Report, Third Quarter 2011”. 2011. www.mcafee.com/us/resources/reports/rp-quarterly-threat-q3-2011.pdf

[10] A. Felt, M. Finifter, E. Chin, S. Hanna, D. Wagner. “A Survey of Mobile Malware in the Wild.” ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices (SPSM). Oct. 2011.

[11] T. Martin, M. Hsiao, D. Ha, J. Krishnawami. “Denial-of-Service Attacks on Battery Powered Mobile Computeres”. Proc. of Second IEEE Annual Conference on Pervasive Computing and Communications (PERCOM). 2004.

[12] K. Aras. “Jailbreaking iOS - How an iPhone breaks free.” Stuttgart Media University.

[13] Google. “Application Signing”. 2011. http://developer.android.com/guide/publishing/app-signing.html

[14] Apple. “Configuring Development Assets”. 2011. http://developer.apple.com/library/IOs/#documentation/Xcode/Conceptual/ios_development_workflow/100-Configuring_Your_Developmet_Assets/identities_and_devices.html

[15] “Code Signing.” 2011. http://en.wikipedia.org/wiki/Code_signing

[16] The iPhone Wiki. 2011. http://www.theiphonewiki.com/wiki/index.php

[17] S. Loueiro, R. Molva, Y. Roudier. “Mobile Code Security”. Institut Eurecom. 2011. http://www.eurecom.fr/~nsteam/Papers/mcs5.pdf

[18] C. Miller, “Mobile Attacks and Defenses”, IEEE Security and Privacy. Vol. 9, Issue 4, Pages 68-70, July-Aug 2011

[19] A. Felt, E. Chin, S. Hanna, D. Song, D. Wagner. “Android Permissions Demystified”. Proceedings of ACM conference on COmputer and Communications Security (CCS 2011). 2011.

[20] W. Enck, D. Octeau, P. McDaniel, S. Chaudhuri. “A Study of Android Application Security.” Proceedings of the 20th USENIX Security Symposium. 2011.

[21] H Shacham, M. Page, B. Pfaff, E. Goh, N. Modadugu, D. Boneh, “On the effectiveness of address-space randomization”. Proceedings of the ACM conference on Computer and Communications Security (CCS’04). New York. 2004.

[22] R. Hund, T. Holz, F. Freiling. “Return-oriented rootkits: bypassing kernel code integrity protection mechanisms”. Proceedings of ACM 18th Conference on USENIX Security Symposium (SSYM’09). Berkeley, CA. 2009.

[23] T. Blasing, L. Batyuk, A.D. Schmidt, S. A. Camtepe. S. Albayrak. “An Android Application Sandbox System for Suspicious Software Detection.”

[24] A. Shabtai. “Malware Detection on Mobile Devices”. Proceedings of IEEE on Mobile Data Management. Pages 289-290. 2010.

[25] S. Dai, Y. Liu, T. Wang, T. Wei, W. Zou. “Behavior based malware detection on mobile phones”. IEEE Wireless Communications Networking and Mobile Computing (WiCOM). Pages 1-4. 2010.

[26] G. Zyba, G. Voelker, M. Liljenstam, A. Mehes, P. Johansson. “Defending mobile phones from proximity malware”. Proceedings of IEEE INFOCOM. 2009.

[27] H.S Chiang, W.J. Tsaur. “Identifying smartphone malware using data-mining technology”. Proceedings of IEEE Computer Communications and Networks (ICCCN). Pages 1-6. 2011.

[28] O. Mencer, M. Mar, M.J. Flynn. “Hardware Software Tri-Design of Encryption for Mobile Communications Units.” Proceedings of IEEE Acoustics, Speech and Signal Processing. 1998.

This was all discovered after several hours of building drivers, realizing that I accidentally downloaded the x64 version of BT5 (which is why the drivers weren’t working), ndiswrapper doesn’t (and won’t ever) support monitor mode.

Disclaimer: I am not a “web guy” by nature. I have recently start experimenting more and more with it because eventually I want to become proficient with web and design. Therefore, this may be a simple tips for “web people” but a typical “desktop application” person might struggle with this.

1. Debugging in Chrome, Firefox and Safari is easy, debugging in IE is hard: Chrome, Firefox and Safari all make it extremely easy to debug issue related to HTML, CSS and Javascript (AJAX included!) because they Chrome and Safari have built in “inspectors.”  Firefox requires an additional plugin, Firebug, but it is just as useful once you have it.  IE makes this process as painful, however, in IE 8 (I haven’t verified anything before or after this version), the Tools menu has a “Developer Tools” window that attempts to do the same thing as Firebug and the built-in inspector for Chrome and Safari.  While it is kind of like using rock knives compared to the other tools, it did work and I was able to debug my CSS in IE (Javascript is much harder). Microsoft has a nice writeup on this tool here.  In fact, go ahead and use Photoshop or Illustrator to create your designs – that’s fine.  However, when it comes time, always test first in IE since it has the poorest support.  Chances are that if it works in IE first time, it’ll work in the other browsers first time as well.  Oh yeah, and get CSSEdit (sorry Windows people – you’re on  your own).
2. RAILS_ENV != RACK_ENV in Rails 3: I was having a devil of a time getting my staging server to work with my staging database.  Passenger and Apache would connect and use the proper staging files directory but it would constantly connect to the production database.  The trick is that in Rails 3 (only!!!) you need to set ENV['RACK_ENV'] = “staging” or ENV['RACK_ENV'] = “production”.  In Rails 2.x.x, you still use the RAILS_ENV to tell it which environment to use.
3. Having a good deployment strategy and capistrano recipe is key: Many months earlier, I had already put up a plain HTML site (no framworks, *gasp*) at www.davidheartsrachel.com at a really cheap host, iPage (they were $30 for the year and believe me, you get what you pay for but that was OK because it was just text and images). Everything was working there just fine there while I was developing this Rails app. Initially, (before I switched to Rails from Google App Engine), I thought that I would host the application at some place like Heroku, since their deployment looks dead simple. As it turned out, it would be much cheaper, by about $40/per month, to have my primary host (the one this site is using) and just buy some additional upgrades in RAM to my VM to suppose multiple Passenger instances.Anyways, I knew that I would need to start testing outside of a development environment. In addition to the standard development, test and production environments given to you in Rails, I converted the test environment into a local version of a production environment (cache classing, actual mail delivery, etc.), however I kept my database adapter as SQLite. I also created an additional environment called staging. The theory behind this is that I would run exactly as it would in production (using MySQL, class caching, session stored via :mem_cache, using delayed_job to deliver mail, etc.) with test fixtures and data in the database. Obviously production would be production environment with live data. This allowed me to iron out any bugs that might crop up during switching to a more production-like environment and catch any assumptions I had made in development that wouldn’t be valid in production (when switching adapters, etc.). I dedicated a subdomian to deploying my staging environment and password protected it. If you are interested in my Capistrano file, environments file and Apache vhost file, they are here (with the obvious implementation and sensitive path/password redactions).
4. Radio buttons: I was making some AJAX calls that should have selected radio buttons in the same group when the call returned based on :success or :error AJAX callback. I was having trouble getting this to happen but this code here is solid:
   // this example had a yes/no. it can be extended to any number of radio buttons
var radios = $('input:radio[name=radio_group_name]');
radios.filter('[value=val1]').attr('checked', false);
radios.filter('[value=val2]').attr('checked', true);
5. Rails UJS: Rails 3 embraces a more MVC way of thinking by abstracting the Javascript choice (whether it is jQuery, Prototype, etc.). Before the implementation was fairly specific and you sometimes had to mix specific JS code into your Ruby code. With UJS, implementation in Rails 3 takes benefit of the new HTML5 data-*@ attributes. So Rails doesn’t spit out Prototype-based Javascript inline (the old helpers are still there). Rather, the helpers just define an appropriate data attribute in the tag, and that’s where Javascript comes in. Obviously this is a more flexible and reusable pattern. You can also plug and play different frameworks without the headache of writing a lot of the same code over again (I swapped Prototype and jQuery in in several minutes – just remember to get the new rails framework Javascript files, such as rails.js, in jQuery or you’ll have weird errors that are hard to track down.  Check out jqueryrails.). Since I am far from a RoR or Javascript expert, this, this and this has a more expert breakdown (why re-invent the wheel).
6. Remote form_for (AJAX) and :onsubmit(): In Rails 2? and at least 3, form_form, :remote => true overrides the :onsubmit => ‘func();’ method to do the actual form submission.  If you want to bind something to your form before it gets submitted (or during or after!), bind the form using jQuery.bind() and then observe the AJAX callback functions to do what you need. <script type="text/javascript">
  function get_address_information() {
    // check jquery for all the possible callbacks. there is also success and error. compete get calls after both success and error
    var pars = 'param1=x&param2=y&param3=z';
    $.ajax({ type: "POST",
      url: "/get_invite_address",
      data: pars, dataType: 'script',
      beforeSend: function() {
        // do your before send checking here
      },
      complete: function(data, status) {
        // do some post processing ehre
      }
    });
  }
</script>
7. Rails is as good as everyone says (and other languages/frameworks are as bad as you think): Truly. The only drawback is the lack of internationalization. However, in terms of use, the language, setup, data access Rails is superior in every way. Rails setups up for testing (as well as performance testing) from the word go. The framework allows you to concentrate on coding while it does the heavy lifting.

Report card from here

Index Terms—capacity planning, multi-tier systems, requests, sessions, performance, autonomic computing, distributed systems

I. Introduction

The Internet has become an important and popular channel for a variety of different services, such as news aggregation, online shopping, social networking or financial brokerage and other services. Many of these popular services on the Internet today, such as Facebook, Twitter, Amazon and eBay [2, 5, 9] are composed of a generic multi-tier architecture.  Requests generated by users flow through each layer in this architecture.  Each tier in this system provides certain functionality to the following tier by executing part of the incoming request [5].  This multi-tier system includes 1) a web tier running a web server to respond to incoming requests, such as Apache 2) an application tier that is running an application container which hosts an application, such as Tomcat 3) and a backend database tier running database software such as MySQL.

Tiered Application [2]

Each of these tiers may be a single machine or may be distributed over a cluster of nodes.  In addition, these servers may be virtualized, such that multiple applications can be hosted and run on the same physical node.  Often times, to ensure acceptable performance, especially in application hosting, there are service level agreements (SLA) that are put into place to govern the desired system performance.  An SLA can specify one or multiple parameters, such as required total uptime of a system, total throughput, average end-to-end delay of requests within a certain percentile, etc.  The SLA target is maintained by the addition of resources into each tier and can be maintained by removing resources as necessary (without violating the SLA).

Traffic on a computer network is inherently non-uniform and hard to predict, mainly due to “think” time of the end user.  This “bursty” behavior is characterized by short, uneven spikes of peak congestion in the life of an application [4].  Popular services can experience dynamic and varying workloads that depend on popularity, time or date, or general demand.  In addition, Internet flash crowds can cause bursty and unpredictable workloads in each of these tiers.  For example, in November 2000 holiday season, Amazon.com experience a forty-minute downtime due to an overload [5].  These flash crowds can cause a significant deviation from the normal traffic profile of an application, affecting the performance.

The performance of the system can be characterized by the total measured end-to-end delay generated by these incoming requests and their workload each tier.  Due to the fact that the interactions between the software and hardware in each of these aforementioned tiers can be quite complex, the management process of allocating sufficient resources (CPU, disk, network bandwidth, etc.) to each tier such that they don’t saturate and become a bottleneck in the system is often a difficult, lengthy and potentially error-prone process to model and estimate for human operators.  For example, if the performance of an application tier dominates the performance of the system, that tier becomes the bottleneck.  It is non-trivial to model the process (scheduling, algorithms, memory, I/O and CPU times) and time it takes to execute the business logic in this tier, especially if resources are shared, as is common in distributed systems.  In addition to these problems, static allocation does not address the issue of Internet flash crowds and the potential they have to overload the system and jeopardize the SLA compliance.

Capacity planning and autonomic resource management plays an important role in modern data centers.  Service differentiation (Quality of Service) and performance isolation help these complex systems to adapt to changing workloads within the system as a whole and within each tier.  In this paper, we will present a survey of the large variety of models, techniques and results of autonomic resource management in large-scale, multi-tier systems.

II. Network Traffic Characteristics

To truly understand the reason for capacity planning and the difficulties it presents in modeling in terms of the interactions between the incoming requests and the hardware and software that service them, we have to understand network traffic characteristics.  Incoming session and request rates tend to fluctuate based on a variety of different factors.  These varying workloads could be characterized as “bursty”, which is defined by [4] as short, uneven spikes of peak congestion during the lifetime for the system.  These traffic patterns deviate significantly from the average traffic arrival rates.

One way to characterize burstiness as seen in [4] is to use the variable I to represent the index of dispersion.  The index of dispersion is used to measure whether a set of observed occurrences are clustered or dispersed compared to a standard model.  When I is larger, the observed occurrences of bursty traffic is more disperse.  We can see that in Figure 1, burstiness can aggregate into different periods represented by different indices of dispersion.

Figure 1

A simple way to calculate the index of dispersion for a particular is as follows:

where N_t is the number of requests completed in a time window of t seconds, where t seconds are counted ignoring server idle time, Var(N_t) is the variance of the number of completed requests and E[N_t] is the mean service rate during busy periods.  An astute reader can deduce that as t becomes large, if there are no bursty periods within t, the index of dispersion is low.

Now that bursty periods can be quantified via I, what causes burstiness?  [4] suggests that “burstiness in service times can be a result of a certain workload combination” and “burstiness in service times can be caused by a bottleneck switch between tiers and can be a result of “hidden” resource contention between the transactions of different types and across different tiers.”  That is, one server or tier may be lightly loaded during a particular period, but may become saturated in other periods where a large number of requests are processed.

This hidden resource contention and different types of transactions across different tiers can be extremely difficult to model and even more troublesome for human operators to correctly provision for.  Autonomic provisioning of system resources is needed to optimize service metrics.

III. Self-Management

As previously mentioned, the major motivation behind autonomic resource management and capacity planning is to reduce the human involvement in these activities due to their difficulty.  If the systems are able to plan for the future and react to the present without operator input, it will take the load off of system administrators and more accurately respond to the situation at hand.  In order to do this, one has to define the goals and properties of such an autonomic system.  The main properties of an autonomic system, as cited in [1] are:

• Self-configuration: autonomic systems configure themselves based on high level goals, specifying what is desired (SLAs), not how to achieve the goals.  Systems with self-configuration can install and set themselves up.
• Self-optimization: autonomic systems can optimize its use of resources either proactively or reactively in an attempt to improve service and meet SLAs.
• Self-healing: autonomic systems can detect and diagnose problems at both high and low levels (software and hardware).  Autonomic systems should also attempt to fix the problem.
• Self-protecting: autonomic systems should protect itself from attacks but also from trusted users trying to make changes.

One particularly important feature of autonomic system is the ability to exhibit “proactive” features.  For example, software or hardware agents should be able to be:

• Autonomic: Agents operative without direct intervention of humans and have some kind of control over their actions and internal state [1].”
• Social: “Agents interact with other agents via some agent-communication languages [1].”
• Reactive and proactive: “Agents are able to perceive their environment and respond in a timely fashion.  Agents do not simply act in response to their environment, but they are able to exhibit goal-directed behavior in taking the initiative [1]

In order to model these properties, IBM suggested a general framework autonomic control loop, MAPE-K as detailed in [1].  MAPE-K allows for clear boundaries in the model to classify the work that is taking place in the autonomic system.  In MAPE-K, there are managed elements, which represents any software or hardware resource that is managed by the autonomic agents.  Sensors collect information about managed elements and provide input to the MAPE-K loop such that the autonomic manager can execute the changes.  Typically the managers are additional software components configured with high-level goals which leaves the implementation of these goals up to the autonomic manager.  Effectors actually carry out the changes to the managed elements (these changes can be fine or course grained).  Each of these elements can be observed in the following sections.

In the MAPE-K loop, there are five different components: monitor, analyze, plan, execute and knowledge.  Monitoring involves “capturing properties of the environment (psychical or virtual) that are important to the self properties of the autonomic system [1].”  This information is captured from sensors in the system in two different ways, passively (using built in tools to run an analysis on the system) or actively (engineering the software to monitor and improve performance).

In the planning model of MAPE-K, the manager uses the monitoring data from the sensors to produce a series of changes to one or more managed elements.  These changes can be calculated by having the autonomic system keep state information about the managed elements and data so that adaptation plans can be adjusted over time.  An additional benefit of keeping state information is that systems are able to create an architectural model where the actual system mirrors the model and proposed changes are able to be verifying that the system integrity is still in tact before and after the proposed changes.  If any violations occur after applying the changes to the model, the changes can be aborted or rolled back to avoid damage or downtime to the system.

In the knowledge part of the model, the knowledge used to effect adaptation can come humans, logs, sensor data, or day-to-day observation of a system to observe its behavior [1].  In this part of the model, there is a large space that one can use machine learning to acquire knowledge about the system.  While [1] suggested reinforcement learning and Bayesian techniques, other authors [2] suggest K-Nearest Neighbors (KNN) and neural networks [7] with fuzzy control.  This author would suggest that decision trees could be an effective method of acquiring the knowledge to effect the proposed plans.  Also, clustering could be used as a way of identifying and classifying previously similar plans to see if they were successful or if they resulted in failure.

Finally, there are several different levels of automaticity including: Basic, Managed, Predictive, Adaptive and Fully Autonomic.

Now that we have seen a general framework for automated resource management, let’s continue to explore each component.

1. Managed Elements

Managed elements in an autonomic system consist of all the hardware and software elements that can be managed by autonomic agents.  Due to the fact that multi-tier systems can be very complex, there are multiple levels of detail that one can view the system.

The first and highest level is where the system is viewed as a black box.  At this level, we consider the end-to-end delay when the request enters the system and returns back to us.  If there is congestion or delay caused by insufficient capacity at any tier, we are unable to know which tier’s capacity is causing the problem. Typically, changes in allocation to capacity are fairly course-grained at this level.  [8] plans their allocation strategy at this level.

At the next level down, we no longer monitor system performance by a single delay metric.  We are now able to monitor the performance of individual tiers.  When congestion occurs at a single tier, we are able to target that tier and increase the allocation capacity of just that tier.  However, one must be careful not to trigger downstream allocations with the increased capacity at the original tier.  [5] plans their allocation strategy at this level.

Lastly, at the most fine-grained level, we are able to collect statistics on individual components within a tier such as CPU usage for an application on a particular node.  Figure 2 shows an example of the black-box and per-tier paradigms.

1. Autonomic Managers and Controllers

In the MAPE-K loop, the monitoring, analyzing and planning phases are done in the control plane or autonomic manager.  The manager should adjust managed elements in the following fashion: [2]

• Prompt. It should sense impending SLA violations accurately and quickly and trigger resource allocation requests as soon as possible to deal with load increases (or decreases).
• Stable. It should avoid unnecessary oscillations in provisioning resources (adding and removing).

Keeping the two guidelines in mind, there are two important decisions that every autonomic manager will be making: 1) when to provision resources and 2) when a provisioning decision is made, how much of a particular resource should be provisioned.

When To Provision

When considering the question of “when to provision”, one must consider the timescale that is being evaluated.  There are two main methods of provisioning: predictively and reactively.  Predictive provisioning attempts to stay ahead of the workload, while reactive provisioning follows to correct for short term fluctuations.

In predictive provisioning, the autonomic system is planning increased (or decreased) capacity for the future.  When evaluating real-world workloads, typically the system or application will see a peak in traffic during the middle of the day and be the minimum during the middle of the night [5].  Other factors, such as recent news (breaking important stories) or seasons (holiday shopping) can affect this distribution of traffic.  By using predictive analysis, automated systems are able to provision sufficient capacity well in advance of any potential events that would cause SLA violations.

Prediction can be done via a number of different methods including statistical analysis, machine learning, or by simply using past observations of workload.  This prediction is usually implemented in the control plane or autonomic manager of the system [2, 5, 6, 7, 8].  For example in [2], a scheduler for each application monitors the application database tier and collects various metrics, such as average query throughput, average number of connections, read/write ratios and system statistics such as CPU, memory and I/O usage, about the system and application performance.  These metrics are reported back to the autonomic manager and an adaptive filter predicts the future load based on the current measured load information.  Each of these metrics are weighted to reflect the usefulness of that feature.  In addition, a KNN classifier determines if an SLA is broken and redirects the source allocation to adjust the number of databases such that that tier is no longer in violation.  A resource allocation component decides how to map the resources dynamically.  [7] uses self-adaptive neural fuzzy controller to decide upon the allocation of resources.  Moreover, [6] uses a model estimator which automatically learns online a model for the relationship between an application’s resource allocation and it’s performance and a optimizer which predicts the resource allocation required to meet performance targets.  Lastly in [5], the authors use a histogram of request arrival rates for each hour over several days.  Using that data, the peak workload is estimated as a high percentile of the arrival rate distribution for that hour.  By using these metrics, the application is able to predict shifting workloads in a sliding window.

Most of the papers survived use the error in prediction e(k) (that is, the predicted workload or arrival rate λ_pred(t) and the observed workload or arrival rate λ_obs(t) for a time period t) or the change in error in prediction Δe(k) (e(k) −e(k −1)) as input to their algorithms to help determine the next periods allocation [5, 6, 7].  Since workloads and arrival rates often exhibit bursty overloads [4, 5], these parameters fine-tune our prediction model.

Proactive provisioning alone is not enough to make the system robust and immune to SLA violations.  For example, there may be errors in prediction if workload or arrival rate deviates greatly from previous days.  As mentioned earlier, Internet flash crowds can spike network requests have the potential to cause congestion and overload the system due to their bursty, unpredictable nature.  In these cases, the errors would lag behind the actual events and there would be insufficient capacity.

Reactive provisioning can be used to quickly correct any deviations from these unpredicted events.  Reactive provisioning is used to plan on a shorter time scale, perhaps on several-minute basis.  If anomalies are detected, reactive provisioning can quickly allocate additional capacity to the affected tiers so they don’t become a bottleneck.

In [5], the authors implement reactive provisioning by comparing the predicted session arrival rate λ_pred(t) and the observed arrival rate λ_obs(t) for a time period t.  If these two measurements differ by more than a threshold, they take corrective provisioning action.  In [2], the resource manager monitors the average latency receiver from each workload scheduler during each sampling period.  The resource manager uses a smoothened latency average computed as an exponentially weighted mean of the form WL = α ×L + (1 –α) ×WL, where L is the current query latency.  When the α parameter is larger, the system is more responsive the average to current latency.  In [6], the authors attempt to reactively provision limited resources by using an auto-regressive –moving-average (ARMA) model, where two parameters a₁(k) and a₂(k) capture the correlation between the applications past and present performance and b₀(k)  and b₁(k)  are vectors of coefficients capturing the correlation between the current performance and the recent resource allocations.  Note that if the spike in traffic is large, it may require several rounds of reactive provisioning to get the capacity to an acceptable level.

In [8], the authors consider provisioning resources based on “what-if” analysis.  They argue that most of web applications consist of services that form an acyclic directed graph, which can be formed into a decision tree.  In their model, they ask each tier to predict, online, future performance in the event it received an additional capacity unit or had one capacity unit removed.  The performance model in [8] uses the following equation to calculate response time:

where, R_server is the average response time of the service, n is the number of CPU cores assigned to the service, λ is the average request rate and S_server is the mean service time of the server.  When a threshold of service time is exceeded, re-computation of service time occurs.  These predictions are given to the parent node.  Using these predictions, provisioning agents negotiate resources with each other based on maximum performance gain and minimum performance loss.  The root node selects which services to provision across the tree when the SLA is about to violated, or de-provision, if resources can be removed without causing an SLA violation. Furthermore, in [8], the authors also consider how provisioning cache instances using the previously described “performance promises” would affect workloads in a particular tier and all children tiers due to an increased hit rate.  Figure 3 from [8] illustrates the decision process:

Figure 3

We can see that service I has k immediate children services and it aggregates it’s own performance promises as follows:

Lastly, while the majority of the papers reviewed were concerned with provisioning additional capacity, [2] also considered removing unneeded capacity.  If the average latency is below a low-end threshold, the resource manager triggers a resource removal.  The system then performs a temporary removal of the resource.  If the average latency remains below the low threshold, the resource is permanently removed.  The reason that a temporary removal is performed first is that mistakenly removing a resource is potentially a costly operation if it negatively impacts the system performance.  The main motivation behind this logic is that unused resources are wasted if they are being under-utilized in the system.

When comparing the effectiveness of reactive and proactive, Figure 4 from [5] that proper proactive provisioning can greatly reduce the time spent in violation of the SLA.

Figure 4

In addition to handling smoothly increasing workloads, the provisioning techniques from [5], as shown in Figure 5, predictive provisioning can also handle sudden load bursts effectively.

Figure 5

By combining predictive and reactive provisioning, systems are able have sufficient capacity to handle predictable workloads as well as short-term instabilities.

How Much To Provision

The question of “how much of a given resource to provision” is less straightforward than “when to provision.”  As we stated earlier, it may be necessary to go through several rounds of provisioning before the end-to-end delay is at an acceptable level.  Moreover, depending on the implementation and type of controller you use, the model determining “how much” could be different.  However, all provisioning algorithms attempt to provision enough to meet the SLA, within a time period t, without overprovisioning, because that would be a waste of resources.  Additionally, one critical factor to consider and try to avoid is that increasing the provisioning at a particular tier k might create a downstream bottleneck at tier k + n, assuming n tiers.  Several authors in the survey explore how to avoid this phenomenon.

The authors in [5] present a general framework for determining provisioning needs based on average end-to-end latency.  Suppose a system that has n tiers, denoted by T₁, T₂,…T_N and let R denote the desired end-to-end delay.  Suppose further that the end-to-end response times are broken down into per-tier response times denoted by d₁, d₂,…d_N, such that Σ d_i = R.  Lastly, assume that the incoming session rate is λ.  Typically, one would want to provision the system for the worst-case scenario, that is the peak of λ.  Individual server capacity can be modeled using M/M/1 first come, first serve (FCFS), non-preemptive queues and each request in the queue has a certain amount of work to do.

Assuming the service rate of the queue is μ, then λ ⁄ μ = ρ, would be the service ratio.  If ρ is less than 1, then the queuing delay, that is, the amount the request waits in the queue to be serviced is bounded and finite.  If ρ is equal to 1, the queue length is infinite but queuing delay is only infinite if the inter-arrival times of requests are not deterministic.  Lastly, if ρ is greater than 1, then the queuing delay is infinite.  This can express useful data, such as service time (the queuing delay and the time it takes to execute the request).  This behavior can be modeled in the queuing theory result [9]:

where d_i is the average response time for tier i, s_i is the average service time for a request at that tier and λ_i is the request arrival rate at tier i.  is the variance of inter-arrival time and the variance of service time, respectively, which can be monitored online.  Using this equation, we can obtain a lower bound on a request rate for server i.  Assuming a think-time of Ζ, then request are issued are a rate of (1 / Ζ) [5].  Therefore, the number of servers η_i needed at tier i to service a peak request rate can be computed as:

where β_i is a tier specific constant, τ is average session duration, λ_i is the capacity of a single server and λ is the request arrival rate.  The authors in [5] assume that the servers in each tier are load balanced, although other models do exist that explore provisioning without load balancing, such as [10].

In [7], the authors use a neural fuzzy controller with four layers to determine how much of a particular resource should be (re) allocated.  In their controller design, they have a neural controller with four layers (see Figure 6).

Figure 6

In layer 1, the input variables e(k) and Δe(k) are passed into the neural controller nodes.  In layer 2, each node in this layer acts as a “linguistic term” assigned to one of the input variables in layer 1, where they use their membership functions to determine the degree to which an input value belongs to a fuzzy set (i.e. negative large corresponds to the numeric value -50).  Layer 3 uses the outputs from layer 2 multiplied together to determine the firing strength of a particular rule in layer 3.  Lastly, in layer 4, the output of layer 3 is “defuzzified” into numeric output in terms of resource adjustment Δm(k).  The magnitude of adjustment is determined by the online learning of the neural fuzzy controller, described in more detail in [7].  This online learning algorithm is able to adapt quite rapidly to stationary and dynamic workloads to guarantee 95^th-percentile delay, as seen in in Figure 7 from [7].

Figure 7

In [6], the authors use the optimizer to determine changes in resource allocations for finite resources such as disk and CPU.  This is a significant departure from other models in that it is assumed that we have a (much) larger pool of resources (essentially unlimited) to allocate increased resources from.  Their optimizer uses u_ar as the resource allocation required to meet the performance target or SLA.  The optimizer’s high-level goal is to determine an allocation in a stable manner (no resource oscillations) by using a cost minimization function to find the optimum resource assignments.  In their cost minimization function:

where J_p = (yn_a(k) – 1)² serves as a penalty for deviation of the application performance from the desired target and J_p = ||ur_a(k) – u_a(k – 1)||² serves as a control cost function to improve stability of the system.  The controller will attempt to minimize the linear combination of both functions.  One important point to note in this particular allocation scheme is that allocation requests are confined by the following constraint:

where ur_irt is a tuple of requested resources for application i, resource r for tier t.  Stated another way, the allocations for all applications for a particular resource, such as CPU or disk, in a particular tier must be less than or equal to100%.  When this constraint is violated, a particular resource is experiencing contention.  It is important to note that in this allocation scheme, while there exists the potential for all application to receive adequate capacity to satisfy demand, when contention exists, applications are weighted according to their priorities in the minimization function so that most “important” application receives the largest share of the divided resources.  Clearly, because we cannot allocate more than 100% of a node’s physical CPU, disk or other resource, all applications suffer from performance degradation – the only difference is to what extent the application suffers that degradation.

Other authors of course decide how much to provision based on their controller implementation.  In [2], the authors use a k-nearest neighbors classifier to decide the number of databases to be provisioned.  Lastly, in [8], use a decision tree process to determine the maximum performance gain (and minimal performance loss) to allocate or remove additional resources.  The performance of the algorithm in [8] was quite impressive in observing it’s SLA and remaining well-provisioned without wasting resources as see in Figure 8.

Figure 8

Before looking at additional MAPE-K components, it is worth looking at solving two additional enhancements to resource provisioning: preventing downstream bottlenecks and preventing allocation oscillations.

1. A. Preventing Downstream Bottlenecks

The relationship between an incoming request and the work units at successive tiers are not necessarily 1-to-1.  A single incoming search request at the web tier may trigger more 1 more query requests at a database tier.  Therefore, downstream tier capacity has to be considered when allocating additional capacity at upstream tiers.

First, consider what would happen if additional downstream tiers were adjusted for additional incoming requests.  When an increase in allocation is triggered by an SLA violation at tier k, the capacity will increase by n.  If the downstream tier k + 1 is almost to capacity, that tier now has an additional n requests to service, which has the potential to cause an SLA violation.  When the resource manager detects a potential violation at tier k + 1, there will be another round of provisioning.  In the meantime, due to the fact that it takes a fixed amount of time to provision additional resources, tier k + 1, could be in violation.  Following a pattern, this could cause up to k provisioning events.  Clearly, this cascading chain of allocation, violation and additional provisioning, which in addition to being wasteful, increases the total time in violation of the SLA.  [5] proposes a solution by using the constant β, where if β_i is greater than one if the request triggers multiple requests at tier i or less than one if caching at prior tiers reduces the work at this tier.

1. B. Preventing Allocation Oscillations

Due to the fact that internet traffic has a bursty nature, there is a potential for a rapidly fluctuating load in the system.  Because of this fact, a necessary enhancement to the system controller should be a filter to smooth or dampen very brief load spikes.  This is not to say that we don’t want to respond to all SLA violations, but adding or removing additional capacity can potentially be a very costly and time consuming operation.

In addition, in [2], the authors keep state on the additional allocation, as either STEADY or MIGRATION state.  During the MIGRATION state, it may take a period of time after the SLA violation and associated additional capacity allocation is triggered for the migration of additional resources to occur and “warm-up.”  [2] found that latency may continued to be high or even increase during this period, and as such, sampling during this period may not be reflective of the actual system state.  Furthermore, they found that making decision based on samples from this migration and warm-up time period, may continue to add unnecessary replicas which will need to be removed later.  This is wasteful and could penalize other applications hosted on the same machine.

For additional information on each model and provisioning algorithm, refer to the related paper.

1. Effectors

The execution of the plan in the MAPE-K loop happens with the effectors. In some of the papers that were surveyed, [citation needed – 5?], the authors used request policing to ensure that the admission rate does not exceed the capacity.  Any extra sessions beyond capacity are dropped at the threshold of the system.  Once the request was admitted, it was not dropped at any intermediate tier.  Conversely, each tier could have their own request policing mechanism however, this is quite wasteful as requests that have had prior work done, may be dropped at downstream tiers.

In the majority of the papers [2, 5, 6, 7, 8], additional capacity was allocated from some pool of free resources.  If no more free resources existed or that resource reached it replication maximum, resources were re-provisioned or the sessions and requests were capped at a maximum. In [2], additional database capacity was added to the tier by replicating the existing database.  During the period of migration, requests were held and then when the migration was complete, they were “played” in stages to bring the new copy up to date with the current copy with any additional writes that may have occurred during migration.

In [5], the authors used agile server switching.  In this model, each node had a copy of the application running in a separate virtual machine (VM) however, only one application was active at a time.  When the control plane determined that it needed to allocate additional capacity, if the application running on a particular node schedule for replacement was not the application receiving the additional capacity, the sessions and request were ramped down using fixed-rate or measurement-based techniques.  Once all the sessions had been concluded for the previous application in that VM, it was suspended and the new allocation was brought up.  This allowed new applications to quickly be provisioned for additional capacity.

However, admission control may not be the best choice and [3] presented an alternative to dropping sessions by allocating additional capacity in a “degraded mode.”  Using traditional control theory (having a closed loop that reacts to sensors and actuators), we can readily adjust the load on the server to stay within bounds of process schedulability based on available resources.

The actuator or effector is responsible for performing a specific action based on controller output.  In the case of [3], they initially used admission control as a way of controlling server load.  This limits the number of clients that the server can respond to concurrently.  All other clients are rejected and while this is not a favorable outcome, it is a way of providing QoS to other clients.

However, this actuator can be adapted to provide a “degraded” level of service.  In this “degraded” level of service, the content that is served to the client receiving the degraded service is different than that which the “full service” client receives.  For example, the degraded content may have less content such as images in it.  To perform this degradation, there must be multiple versions of the content and the web server will serve the content from the appropriate file structure at request time.  There can be m different service levels where m = I + F (I is the service level where F is the fraction of clients who are served at level I).  When m = M, all clients receive the highest service and conversely when m = 0, all clients are rejected.  This allows a more general model of saying that a percentage of clients will receive degraded service, rather than specifically specifying which clients (which clients may be addressed through different functionality, such has hashing their IP address).  The server load is control through the input of the variable m.

The monitor proposed in [3] is expressed as the utilization U as a function of served request rate R and delivered byte bandwidth W (U = aR + bW, where a and b and derived constants through linear regression).  This function is able to approximate the maximum utilization rate for deadline-monotonic schedulers to meet deadlines of U < 0.58.  Using this scheme of admission and “capacity” planning, the authors in [3] were able to admit almost 3 times the number of requests to the system as shown in Figure 9 before significant service degradation occurred.

Figure 9

1. Sensors

Sensors are part of the knowledge phase in the MAPE-K loop.  The sensors that report data to the control plane can be any number of metrics.  In the papers surveyed, there were authors that used CPU usage [5, 6], bandwidth, throughput, active network connections [2, 5], I/O transfer [2, 6], cache hits [8], end-to-end delay or response time [5, 7], session arrival rate [5], request arrival rate [5], read/write ratios [2], lock ratios [2] and memory usage [2].  These parameters can be recorded and monitored online or offline, using standard system tools such as vmstat, iostat and sysstat or custom written software to record statistics.

IV. Quality of Service & Performance Isolation

Lastly, the control theory approach was extended in [3] using virtual servers to show support for performance isolation (each virtual server can guarantee a maximum request rate and maximum throughput), service differentiation (each virtual server supports request prioritization) and excess capacity sharing (conforming virtual servers under overload can temporarily exceed capacity to avoid client degradation).

With performance isolation, a virtual server is configured with a maximum request rate R_MAX and a maximum delivered bandwidth of W_MAX.  This expresses an throughput agreement to serve up to the specified levels.  If R_MAX is exceeded, then the agreement on W_MAX is revoked.

In performance isolation, the virtual server can adapt content as previously mentioned to stay within the agreement bounds.  When each of the i servers are configured, if their aggregate utilizations U*_i is less than U < 0.58, then the system capacity is planned correctly.  To perform the correct bookkeeping, load classification is done when the request is performed and then based on the classification requests served and delivered bandwidth is charged against the requested virtual server.  Lastly, based on the rates for virtual server i for requests (R_i) and delivered bandwidth (W_i), the control loop achieves the degree of content degradation necessary to keep the utilization of that virtual server at the proper level, preventing overload.

In service differentiation, the goal is to support a number of different clients at different service levels (lower priority clients are degraded first).  In [3], if there are m priority levels, where 1 is the highest priority, capacity of the utilization level of that virtual server is available to clients in priority order.  For the purposes of their research, [3] had two service levels, premium and basic, where premium was guaranteed service and basic was not guaranteed service.

Lastly, in sharing excess capacity, if the load on one server exceeds the maximum capacity C, then the server under overload may temporarily utilize the resources of under utilized virtual servers.  This requires only a simple modification to the control theory loop.

IV. Conclusion

In this survey, we presented information on burstiness in network traffic and how it affects service times.  We also showed how autonomic systems can provision themselves to handle the dynamic workloads.  Dynamic provisioning in multi-tier applications raises some interesting and unique challenges.  As distributed systems become more complex, the techniques surveyed in this paper will become more useful and relevant.

References

[1]     M.C. Huebscher, J.A McCann.  A survey of autonomic computing: Degrees, models, and applications.  ACM Computing Surveys, 40(3), 2008.

[2]     J. Chen, G Soundararajan, C. Amza. Autonomic Provisioning of Backend Databases in Dynamic Content Web Servers. Proc. of IEEE ICAC, 2006.

[3]     T. Abdelzaher, K.G. Shin, N. Bhatti. Performance Guarantees for Web Server End-Systems: A Control-Theoretical Approach. IEEE Transactions on Parallel and Distributed Systems, 13(1), 2002.

[4]     N. Mi, G. Casale, L. Cherkasova, M. Smirini. Burstiness in multi-tier applications: Symptoms, causes, and new models. Proc. of ACM/IFIP/USENIX Middleware, 2008.

[5]     B. Urgaonkar, P. Shenoy, A. Chandra, P. Goyal, and T. Wood. Agile dynamic provisioning of multi-tier Internet applications. ACM Trans. on Autonomous and Adaptive Systems, 3(1):1-39, 2008.

[6]     P. Padala, K.-Y. Hou, K. G. Shin, X. Zhu, M. Uysal, Z. Wang, S. Singhal, and A. Merchant. Automated control of multiple virtualized resources. Proc. of EuroSys, 2009.

[7]     P. Lama and X. Zhou. Autonomic provisioning with self-adaptive neural fuzzy control for end-to-end delay guarantee. Proc. IEEE/ACM MASCOTS, pages 151-160, 2010.

[8]     D. Jiang, G. Pierre, and C.-H. Chi. Autonomous resource provisioning for multi-service web applications. Proc. ACM WWW, 2010.

[9]     L. Kleinrock. Queuing Systems, Volume 2: Computer Applications. John Wiley and Sons, Inc. 1976.

[10]  B. Urgaonkar, G. Pacifici, G. Shenoy, P. Spreitzer, M. and A. Tantawi. An analytical model for multi-tier Internet services and its applications. Proc. of ACM ICMMCS.  Banff, Canada, 2005.

<sessions-enabled>true</sessions-enabled>

If you don’t you may find yourself with only read-access to the session object.

In addition, you need to make sure that all your objects that are you going to persist to the session implement the java.io.Serializable interface.  This is particularly important and this is what I failed to realize until I struggled with this for 2 hours.  The reason the object needs to be serializable is because App Engine stores session data in the datastore and memcache.  Any objects referenced by the value you put in the session must be serializable, so the entire object graph is available.  What I found interesting is that it must commit the session data in an transactionally based manner because I had also stored a String in the session and that wasn’t persisted either.  If the object isn’t serializable, the app will NOT fail in a local development machine, but will fail when deployed to the cloud.

A bit of sample code:

public void doGet(HttpServletRequest req, HttpServletResponse resp) {
  HttpSession session = reg.getSession(true);
  // passing in a boolean to getSession() will allow you to inspect if a session
  // already exists.  if you pass in true, the session will be created by default
  // whether one exists already or not.
  // if you pass in false, if a session doesn't exist, you will be return NULL and no
  // session will be created.  if there is a previously existing session, it will return
  // that session.

  String name = (String)session.getAttribute('name');

  session.setAttribute("age", 25);
}

You can check to see that everything is being stored correctly if you look on your machine, you should see a cookie for the domain your working in (dev: localhost, prod: appname.appspot.com) with the key JSESSIONID.  The value for JSESSIONID should match what is in your _ah_SESSION table in the App Engine datastore. You can visually inspect the bytes in the session as well.

There is a small gotcha between the standard J2EE HttpSession and the GAE HttpSession.  There is a difference in when services manipulate objects stored in the session, in that case your changes will be lost when another service will get the object from the session.  The fix for this is invoking setAttribute again after having modified the Person object in the session. This workaround will solve all the inconsistencies but has a pretty important trade-off, every setAttribute will trigger a new serialization and write to the datastore.

Lastly, depending on the utilization of your application, you may find yourself with more than a few sessions.  That is because when the session is written to the datastore, the expiration of the session is set to  24 * 60 * 60 * 1,000 = System.currentTimeMillis() + 86,400,000.   The _expires field is updated each time the session is active, so that could be quite a bit of data storage.  There is currently no automatic removal of expired sessions in GAE.

One last note: remember that App Engine is a distributed architecture so a difference from J2EE is that you are never guaranteed the same application server instance during request processing as the previous request.  While the object is being serialized correctly in memcache, you still have to call setAttribute() every time due to the fact that memory is not shared.

I also enjoy simple software and hardware interfaces. Things that I can just pick up and know instantly how to use without reading an instruction manual. Something so simple that Grandma could use it. Apple products always have incredibly simple, unified interfaces: “One Interface to rule them all, One Interface to find them, One Interface to bring them all, and in the darkness bind them.” Simple people living in a simple world. However, simple does NOT imply intuitive.

Ever since the dawn of the electronics age, electronics have had buttons or some sort of interface that had something you could push, pull, or manipulate in some other fashion. Now, there are rumors that iPad2 and iPhone5 will not have home buttons.  In Johnathan Geller’s post:

“We just got some pretty wild information … while it’s hard to believe at first, it does make sense. … The iPad will be losing the home button. … Instead of button taps, you will use new multitouch gestures to navigate to the home screen and … launch the app switcher.
We’re told that this change will make its way over to the iPhone as well. … Apple employees are already testing iPads and iPhones with no home buttons on the Apple campus. … Steve Jobs didn’t want any physical buttons on the original iPhone … it looks like he may soon get his wish.”

So what will be replacing the home button?  In iOS 4.3, which was seeded to developers yesterday, there are new gestures that be done with four or five fingers.  This is particularly a stretch since previous iOS releases supported UIGestureRecognizer to simplify recognizing things like taps, pinch-in and pinch-out.  You could customize the number of fingers needed to perform the gesture and this is simply an extension of existing functionality.  In iOS 4.3, a four or five finger pinch brings you to the home screen,  a swipe up or down reveals the multi-tasking bar and a swipe left or right allows switching between apps.

I am going to do what a devoted Applephite would never do and disagree with what Steve wants.  I think that this is a bad decision because, unlike Steve, regular users actually like buttons!  They like the tactile feel of pushing something, the affirmative “click” that something was done or something will happen.  Consider some software that would process information for whole minutes at a time without providing feedback in the form of a spinner or a progress bar.  Most users would get fed up and try to quit the application.  Without the feedback, user’s may not know what to do.  While these gestures may be fine for people who have been around technology their whole life (think GenY and younger), there are a lot of users who would have problems picking  up the device and straight away being able to use it.  I would even go so far to say that they may never discover some of the features without reading the manual or browsing forums for tips.  Apple has always pride themselves, on simple AND intuitive, easy to use software.  I don’t think regular users are ready for this.

To make my point even clearer, there are still people I know with iOS 4.2 that don’t know that double-tapping the home button brings up the task bar.  They literally have over 75 applications “running” and if it weren’t for me, they would have continued on into their own blissful world, not knowing of that functionality.  And that’s WITH a button.  These gestures should be included in the software in addition so user’s can have their choice of methods of how they want to interact with the device.

Usability aside, how would we perform operations such as recovering crashed phones?

Personally, I would rather see support for a 4G network and WiFi-less FaceTime support.  Although, who knows what the new feature set will be because apparently iOS 4.3 drops support for arm6 (3G).

The horizontal scroll wheel is great because it is built out of standard UI components (UIImageView and UIScrollView) and there is no subclassing to be done to make the widget work. It can be dropped into any application. There is little code to support the layout and generation of the scroll wheel, and all layout calculations are done dynamically so you can have as many or as few items in the list as desired. Also, you can easily create additional value with the audio framework and Core Animation that is extremely simple to do.

For users, it is great because it is an easy to use and understand, quick method to filter a UITableView. The component requires no explanation. A sign of good UI design is a WYSIWYG type approach where the interface is so intuitive it is a joy to use. Create components that users can’t stop playing with.

Let’s get started already. By the way, this tutorial assumes you know Objective-C, Cocoa and basic programming principles. I welcome feedback on any of the code.

ScrollWheel.h

#import <UIKit/UIKit.h>;
#import "ChannelCategory.h"
#import "NSArray+FirstObject.h"
#import "Colors.h"
#import "Debugger.h"

#define SCROLL_WHEEL_CATEGORY_CHANGED @"ScrollWheelCategoryChanged"
#define SCROLL_WHEEL_CATEGORY_ADDED @"ScrollWheelCategoryAdded"

@interface CategoryScrollWheel : UIView <UIScrollViewDelegate> {
  IBOutlet UIScrollView *_scrollView;
}

@property (nonatomic, retain) IBOutlet UIScrollView *_scrollView;

- (void) setupCategories: (NSArray *) categories;
- (void) scrollToButton: (id) sender;
- (void) rebuildCategories;
- (NSArray *) loadNewsCategories;
@end

ScrollWheel.m

#import "CategoryScrollWheel.h"

@implementation CategoryScrollWheel

@synthesize _scrollView;

- (id)initWithFrame:(CGRect)frame {
  if ((self = [super initWithFrame:frame])) {
    // Initialization code
  }
  return self;
}

- (void) rebuildCategories {
  // release the previous wheel and alloc a new one

  [_scrollView release], _scrollView = nil;
  _scrollView = [[UIScrollView alloc] init];

  // reload the categories into the scroll wheel

  [self setupCategories: [NSArray arrayWithArray: [self loadNewsCategories]]];
}

- (NSArray *) loadNewsCategories {
  NSManagedObjectContext *managedObjectContext = [(RSSAppDelegate *)[[UIApplication sharedApplication] delegate] managedObjectContext];

  NSFetchRequest *request = [[NSFetchRequest alloc] init];
  NSError *error;
  NSArray *fetchResults;
  NSSortDescriptor *sortDescriptor = [[NSSortDescriptor alloc] initWithKey:@"name" ascending:NO];
  NSArray *sortDescriptors = [[NSArray alloc] initWithObjects: sortDescriptor, nil];

  [request setEntity: [NSEntityDescription entityForName:@"ChannelCategory" inManagedObjectContext: managedObjectContext]];
  [request setSortDescriptors: sortDescriptors];

  if ((fetchResults = [managedObjectContext executeFetchRequest: request error: &amp;error]) == nil) {
    ALog("%@", error);
    [request release], request = nil;
    [sortDescriptor release], sortDescriptor = nil;
    [sortDescriptors release], sortDescriptors = nil;
    return nil;
  }

  [sortDescriptor release], sortDescriptor = nil;
  [sortDescriptors release], sortDescriptors = nil;

  if([fetchResults count] == 0) {
    // no feeds to refresh
    [request release], request = nil;
    return nil;
  }
  else {
    return fetchResults;
  }
}

- (void) setupCategories: (NSArray *) categories {
  // our variable x will keep track of how far our scroll view extends
  // we need to add a half scroll view width to get the arrow in the middle to
  // be able to center up

  double x = ([self _scrollView].frame.size.width / 2);
  double lastHoldover = 0;
  
  for(ChannelCategory *category in categories) {
    UIButton *button = [[UIButton alloc] init];
    [button setTitle: [category name] forState: UIControlStateNormal];
    [[button titleLabel] setFont: [UIFont fontWithName: @"Helvetica-Bold" size: 11]];
    [[button titleLabel] setTextColor: UI_COLOR_LIGHT_GREY];
    
    CGRect rect = CGRectMake(x, 0, [[category name] sizeWithFont: [UIFont fontWithName: @"Helvetica-Bold" size: 11]].width, 30);

    if([category isEqual: [categories firstObject]]) {
      x -= (rect.size.width / 2);
      rect.origin.x = x;
    }

    [button setFrame: rect];
    [button addTarget: self action: @selector(scrollToButton:) forControlEvents: UIControlEventTouchUpInside];

    [[self _scrollView] addSubview: button];

    x += button.frame.size.width + 10;

    if([category isEqual: [categories lastObject]]) {
      lastHoldover = (button.frame.size.width / 2);
      // bad hack to lop off the extra 10 on the last item
      x -= 10;
    }

    [button release], button = nil;
  }

  // this adds a half scrollview width to the end of the scrollview

  x += ([self _scrollView].frame.size.width / 2);
  x -= lastHoldover;

  CGSize size = CGSizeMake(x, 30);

  [[self _scrollView] setContentSize: size];
}

- (void)scrollViewDidEndDecelerating:(UIScrollView *)scrollView {
  // we add the offset and half the view together to bring it into line with the
  // arrow in the middle
  double val = [[self _scrollView] contentOffset].x + ([self _scrollView].frame.size.width / 2);

  // loop through and see which button is closest.  snap to that button
  // using scrollToButton

  UIButton *closestButton;
  double closest = 100;

  for (UIButton *view in [[self _scrollView] subviews]) {
    if ([view isKindOfClass:[UIButton class]]) {
      double calculatedValue = abs(view.frame.origin.x -  val);

      if(calculatedValue &lt; closest) {
        closestButton = (UIButton *)view;
        closest = calculatedValue;
      }
    }
  }

  CGPoint offset = closestButton.frame.origin;
  CGRect scrollViewFrame = [[self _scrollView] frame];
  offset.x -= (scrollViewFrame.size.width / 2) - (closestButton.frame.size.width / 2);

  [[self _scrollView] setContentOffset:offset animated:YES];
  [[NSNotificationCenter defaultCenter] postNotificationName: SCROLL_WHEEL_CATEGORY_CHANGED object: [[closestButton titleLabel] text]];
}

- (void) scrollToButton: (id) sender {
  CGPoint offset = ((UIButton *)sender).frame.origin;
  CGRect scrollViewFrame = [[self _scrollView] frame];

  offset.x -= (scrollViewFrame.size.width / 2) - (((UIButton *)sender).frame.size.width / 2);

  [[self _scrollView] setContentOffset:offset animated:YES];
  [[NSNotificationCenter defaultCenter] postNotificationName: SCROLL_WHEEL_CATEGORY_CHANGED object: [[sender titleLabel] text]];
}

/*
// Only override drawRect: if you perform custom drawing.
// An empty implementation adversely affects performance during animation.
- (void)drawRect:(CGRect)rect {
  // Drawing code
}
*/

- (void)dealloc {
  [super dealloc];
}

@end

ScrollWheel.xib

In my ScrollWheel.xib, I have 2 UIImageViews and a UIScrollView. Literally, it is that easy. The image view is meant to create a nice frame around the scroll view giving the illusion of “limited view.” The top UIImageView may even have a small arrow pointing down to indicate which item is currently selected. The UIScrollView is the one referenced in the code above.

So there it is – a neat, clean, compact way of taking standard UI components and turning it into a custom UI to allow for selection of categories. If you have any questions or comments on how to improve the code, please let me know!

Index Terms—machine learning, hierarchical agglomerative clustering, k-means clustering, unsupervised learning, Pearson’s coefficient, euclidean distance

1. Introduction

Clustering is a method of dividing a heterogenous group of objects into smaller, more homogenous groups that display similar characteristics to other objects in the cluster while displaying one or more dissimilar characteristics to objects in other clusters. Clustering is an unsupervised learning technique that has many applications in data mining, pattern recognition, image analysis and market segmentation. Clustering is easy to implement and fairly quick to come up with the groupings.

The main purposes of a Department of Corrections at any governmental level are to enhance public safety, to provide proper care of the inmates, to supervise inmates under their jurisdiction and to assist inmates’ re-entry into society.

There is no doubt that inmates at correctional institutions are dangerous to society. However, even after they are incarcerated at these institutions, some individuals remain ongoing offenders. While all individuals in prison have displayed some sort of deviant behavior, it is hypothesized that certain combinations of personality traits make some inmates more likely to be sexual predators and some inmates more likely to be sexual victims of these predators. At most correctional facilities, sexual contact between inmates, consensual or not, is not permitted.

Identification of those inmates likely to be sexually predatory toward other inmates would greatly assist corrections facilities in their goal of providing a safer environment for incarceration. Clustering can help with this goal by comparing a particular offender to known perpetrators and victims. After comparison, victims can be incarcerated separately from predators and receive any special needs assistance that can be offered while predators can be segregated in such a fashion as to reduce the potential for successful predatory behaviors.

1.1 Outline of Research

In this research survey, we implemented two different types of clustering algorithms – a standard “bottom-up” hierarchical method, single link clustering, and a standard “top-down” partitional algorithm, k-means clustering. We evaluated different distance measure criteria, including Euclidian distance and Pearson’s correlation coefficient. Results are discussed in section 4 after running the clustering algorithms multiple times with the provided Colorado inmate dataset.

1.2 Data

The dataset that we used was provided by Dr. Coolidge of the Department of Psychology at the University of Colorado at Colorado Springs. The dataset is publicly available at http://www.cs.uccs.edu/~kalita/work/cs586/2010/CoolidgePerpetratorVictimData.csv. This dataset pertains to scores on personality disorder tests given to inmates in the State of Colorado. Dr. Coolidge’s inventory of personality disorder tests are given to all inmates in the State of Colorado.. This dataset provided contained 100 rows (25 rows describing victims of sexual abuse and 75 describing perpetrators of sexual abuse) with 14 attributes chosen by Dr. Coolidge.

The data described different measurements of how inmates had scored on different personality disorders tests. The tests included antisocial (AN), avoidant (AV), borderline (BO), dependent (DE), depressive (DP), histrionic (HI), narcissistic (NA), obsessive-compulsive (OC), paranoid (PA), passive-aggressive (PG), schizotypal (ST), schizoid (SZ), sadistic (SA) and self-defeating (SD) markers. The scores on these individual tests are measured by T scores, which are a type of standardized score that can be mathematically transformed into other types of standardized scores. T scores have a Gaussian distribution and a score of 50 is always considered the mean and a standard deviation is always 10.

It should be noted that even though the dataset is quite small, with only 100 rows available, the quality of the data in the dataset is very good. The astute reader can appreciate the fact that incarcerated persons might not be completely truthful in their answering of the test questions for a variety of reasons, such as a lack of caring or the desire to appear more “damaged” than any other inmate. The data is cross-checked with several other validation methods to ensure that the answers provided are reasonable. Test scores that are not reasonable were discarded by Dr. Coolidge and not included in this dataset.

2. Application

We chose to write the implementation of the clustering algorithms in Java because of the ease of use of the language. Java also presented superior capabilities in working with and parsing data from files. Using Java allowed the author to more efficiently model the problem through the use of OO concepts, such as polymorphism and inheritance. Lastly, several different Java libraries were available, such as Java-ML [1] that increased the ability to analyze the clusters after the algorithms had been run.

2.1 Hierarchical Cluster Construction

In agglomerative single link clustering, clusters are constructed by comparing each point (or cluster) with every other point (or cluster). Each object is placed in a separate cluster, and at each iteration of the algorithm, we merge the closest pair of points (or clusters), until certain termination conditions are satisfied.

This algorithm requires defining the idea of a single link or the proximity of other points to a single point. For single link, the proximity of two points (or clusters) is defined as the minimum of the distance between any two points (or clusters). If there exists such a link, or edge, then the two points (or clusters) are merged together.

This is often called “the nearest neighbor clustering technique.” [4] Relating this algorithm to graph theory, this clustering technique constructs a minimum spanning tree by finding connected components, so the algorithm is quite similar to Kruskal’s or Prim’s algorithm.

MSTSingleLink (Elements, AdjacencyMatrix)
Create a set of clusters C = {t1, t2,…,tn} from Elements
Create a partial distance matrix showing the distance between all clusters in C.
k = n, where n is the number of clusters
d = 0
Begin
Ci, Cj = Closest clusters in AdjacencyMatrix
d = dis(Ci, Cj) // update dendrogram with distance threshold
dis({Ci ∪ Cj}, C) // recalculate the distance from the new cluster to all other points in C
C = C – {Ci} – {Cj} ∪ {Ci ∪ Cj} // merge the two closest clusters or points
dis(Ci, Cj) = 0
Until k = 1

Typically, for this algorithm, the termination criteria is for all elements grouped together in one cluster. A better termination criterion would be to record the distances at which the merges of individual objects and clusters take place and if there is a large jump in this distance (large being defined by the user), it might give the user an indication that the two objects or clusters should not be merged because they are highly dissimilar. Note that the running time for MSTSingleLink is O(n^2). This running time makes it impractical for large datasets. For further information on single link MST, see [2], [4].

2.2 Partitional Cluster Construction

Where a hierarchical clustering algorithm creates clusters in multiple steps, a partitional algorithm, such as k-means, creates the clusters in one step. [4] Also, in partitional clustering, the number of clusters to create must be known a priori and used as input to the algorithm.

In k-means, elements are moved among sets of clusters until some sort of termination criteria is reached, such as convergence. A possible measure for convergence could be testing to see if cluster elements have not changed clusters. Using k-means allows one to achieve a “high degree of similarity among elements, while a high degree of dissimilarity among elements in different clusters.” [4]

KMeansCluster (Elements, k)
Create a set of clusters C = {t1, t2,…,tn} from Elements
Assign intial values for means, m1, m2,…,mk
Begin
Assign each item ti to the cluster which has the closest mean
Calculate new means for each cluster
Until convergence criteria met

Note that the running time for KMeansCluster is O(tkn), where t is the number of iterations. While, k-means does not suffer from the chaining problem, it does have other problems. k-means does not handle outliers well, work with categorical data or produce any clusters shapes other than convex clusters. [4] Also, while k-means produces good results, it does not scale well and is not time-efficient. [4] While this particular provided dataset is not large, k-means could have problems with attempting to cluster millions of objects. Lastly, it is possible for k-means to find a local optimum and miss the global optimum. For further information on k-means clustering, see [2], [4].

2.3 Distance Criterion

In both algorithms, cluster formation relies on having some notion of a distance measure. Using this metric, we can determine how “similar” two elements are. Depending on the distance metric chosen, it will influence the shape of our clusters. While there are many distance measures such as Mahalanobis, hamming, city-block and Minkowski [2], [6], in our implementation, we used two different distance measures, a euclidean distance measure and Pearson’s correlation coefficient.

2.3.1 Euclidean Distance

Euclidean distance is the ordinary distance between two points that one would measure with a ruler. It is a simple distance metric and by far the most commonly used, where one has to make sure all attributes have the same scale [2]. It is defined by this equation

2.3.2 Pearson’s Correlation Coefficient

Pearson’s correlation coefficient is a measure of the linear dependence between two variables X and Y, giving a value between +1 and −1 inclusive. A value of 1 implies that a linear equation describes the relationship between X and Y perfectly, with all data points lying on a line for which Y increases as X increases. A value of −1 implies that all data points lie on a line for which Y decreases as X increases. A value of 0 implies that there is no linear correlation between the variables. [3] Although it depends on the data being analyzed, typically anything over 0.5 or below -0.5 indicated a large correlation. Pearson’s correlation coefficient can be calculated using the equation below, where X and Y bar are the means of the two variables.

For further information on Pearson’s correlation coefficient, see [3].

2.4 Testing

Testing of the clustering algorithms was performed with the entire dataset of 100 examples. For single link clustering, the algorithm was run using the Euclidean distance measure and Pearson’s correlation coefficient. For k-means clustering, the algorithm was run using the Euclidean distance measure 8 times. Each time the program was run, the number of clusters specified, k, was increased by one from 2 clusters to 10 clusters. Discussion of results are in section 4.

3. Potential Problems

There was one main problem that we encountered during implementation and testing. Our biggest problem is that with single link clustering, we observed the chaining effect. The chaining effect is where “points that are not related to each other at all are merged together to form a cluster simply because they happen to be near (perhaps via a transitive relationship) points that are close to each other.” [4] By including dissimilar objects, this can cause clusters to become skewed. A potential solution to this problem would be to either specify a maximum distance threshold, above which points (or clusters) would not be merged. This could also serve as part of the termination criteria. Another solution, would be to use a complete link distance criteria. [4] The chaining effect is most obviously seen when the output of the clustering program is a dendrogram.

4. Evaluation

For the evaluation of the results, we have chosen to use several different evaluation criteria. For single link clustering, we will qualify our results by examining the intra-cluster and inter-cluster distance, which measures the homogeneity of the clusters. In addition, we also evaluate the distance threshold at which the clusters were merged and the entropy within the cluster. Lastly, we evaluate the cluster by calculating the recall, precision and F measure of that cluster.

For k-means clustering, we evaluate the clusters using some of the same techniques (recall, precision and F measure) and we also introduce the squared sum of errors measure, and Bayesian information criterion measure.

4.1 Macro Evaluation

In single link clustering, the distance threshold that produced the best clusters was 46.23 (see Figure 4.1). Cluster 1 were inmates that exhibited personalities of victims of sexual abuse while cluster 2 were inmates that exhibited personalities of perpetrators of sexual abuse.

The remaining clusters, 3, 4, 5 and 6 were outliers and in their own clusters and their personalities exhibited behavior of both victims and perpetrators of sexual abuse. We noted that the distance threshold was much higher to merge clusters 4, 5, 6 with thresholds of 49.60, 55.22 and 59.63.

Cluster Cluster Members Intra-cluster Distance Inter-cluster Distance
1 1, 65, 4, 6, 97, 45, 49, 58, 53, 19, 18, 42, 67, 55, 62, 36, 7, 32, 54, 69, 39, 38, 89, 24, 26, 72, 8, 3, 9, 11, 95, 91, 27 15.64 90.82
2 2, 10, 16, 41, 50, 47, 51, 87, 78, 64, 68, 79, 77, 44, 84, 29, 31, 34, 63, 33, 90, 80, 40, 74, 82, 37, 43, 71, 48, 93, 96, 98, 22, 73, 52, 20, 57, 59, 46, 61, 75, 85, 66, 92, 83, 94, 88, 81, 70, 100, 60, 99, 86, 56, 13, 15, 30, 21, 14, 12, 23 8.73 79.20
3 5, 35, 25 0.17 66.25
4 17 0.0 67.19
5 28 0.0 67.19
6 76 0.0 67.20
Figure 4.1 – Index of clusters with intra-cluster and inter-cluster distance for Euclidean single link clustering

In Pearson’s coefficient clustering, the distance threshold that produced the best clusters was .035 (see Figure 4.2).
Cluster Cluster Members Intra-cluster Distance Inter-cluster Distance
1 1, 17, 14, 52, 48, 54, 27, 21, 22, 26, 77, 74, 33, 50, 57, 62, 47, 49, 79, 31, 34, 69, 67, 2, 86, 71, 100, 3, 41, 10, 29, 8, 25, 40, 83, 37, 19, 44, 46, 72, 55, 81, 88, 66, 30, 98, 38, 70, 20, 45, 36, 80, 60, 87, 13, 56, 91, 68, 51, 23, 16, 53, 89, 12, 65, 82, 94, 96, 90, 64, 58, 43 19.71 138.31
2 4, 11, 35, 84, 39, 15, 61, 63, 18, 92, 24, 76, 6, 7, 59, 95, 78 3.68 108.01
3 5, 97, 73, 93, 75 2.5 106.01
4 32, 42, 99 0.0 106.72
5 28 0.0 109.65
6 9 0.0 109.65
7 85 0.0 109.65
Figure 4.2 – Index of clusters with intra-cluster and inter-cluster distance for Pearson’s single link clustering

Based on these results, using the Euclidean distance may produce better clusters based on intra-cluster distance. Another improvement to the results may come in the form of changing the policy on finding the best distance at which to merge the clusters (i.e. using complete link or average link distance measures). If a more accurate method of distance finding were implemented, we would expect to see a more consistent result set because there would be less of an effect from the chaining problem.

4.2 Micro Evaluation

When the individual clusters are broken down and the individual members are analyzed, we achieve the following results. These recall, precision, F measure and entropy measurements assume the same clusters as above in section 4.1. In our calculations, we assign a negative value to not identifying a sexual abuse perpetrator because they would be allowed to interact with the general inmate population instead of being in administrative segregation. In addition, we also only consider the first two clusters, of either predator or victim, but no mixed classes.

Cluster Recall Precision F-measure
1 44.00% 32.35% 0.37
2 66.67% 75.75% 0.71
Overall 55.34% 54.05% 0.54
Figure 4.5 – Recall, Precision, F-Measure for Euclidean Single Link Clustering

Cluster Recall Precision F-measure
1 64.00% 22.22% 0.33
2 13.33% 70.58% 0.22
Overall 38.67% 46.40% 0.28
Figure 4.5 – Recall, Precision, F-Measure for Pearson’s Coefficient Single Link Clustering

From these results, it shows that despite the chaining effect, the Euclidean distance appears to be the superior distance measure when clustering via hierarchical agglomerative methods.

Our next test involved using k-means clustering. We ran the algorithm from 2 to 10 clusters and we measured the accuracy of the clusters using Bayesian information criteria (a criterion for model selection among a class of parametric models) and the sum of squared errors, as defined below:

where x is the observed data, n is the number of data points in x, k is the number of free parameters to be estimated, p(x|k) is the likelihood of the observed data given the number of parameters and L is the maximized value of the likelihood function.



k Clusters Members BIC Score SSE
2 Cluster 1: 2, 5, 10, 12, 14, 15, 16, 20, 21, 22, 23, 25, 29, 31, 33, 34, 35, 37, 40, 41, 43, 44, 46, 47, 50, 51, 56, 57, 59, 60, 61, 63, 66, 70, 71, 73, 75, 77, 78, 79, 80, 81, 83, 84, 85, 86, 87, 88, 90, 92, 94, 98, 99, 100
Cluster 2: 1, 3, 4, 6, 7, 8, 9, 11, 13, 17, 18, 19, 24, 26, 27, 28, 30, 32, 36, 38, 39, 42, 45, 48, 49, 52, 53, 54, 55, 58, 62, 64, 65, 67, 68, 69, 72, 74, 76, 82, 89, 91, 93, 95, 96, 97 144,666.03 223,893.45
3 Cluster 1: 1, 3, 4, 6, 7, 9, 11, 13, 18, 21, 24, 30, 32, 36, 38, 41, 42, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 58, 62, 64, 65, 67, 68, 69, 74, 78, 82, 93, 95, 96, 97, 98
Cluster 2: 2, 5, 10, 12, 14, 15, 16, 20, 22, 23, 25, 29, 31, 33, 34, 35, 37, 40, 43, 46, 51, 56, 57, 59, 60, 61, 63, 66, 70, 71, 73, 75, 77, 79, 80, 81, 83, 84, 85, 86, 87, 88, 90, 92, 94, 99, 100
Cluster 3: 8, 17, 19, 26, 27, 28, 39, 72, 76, 89, 91 142,191.31 190,017.27
4 Cluster 1: 8, 17, 26, 28, 39, 72, 76, 89,
Cluster 2: 3, 4, 6, 19, 27, 36, 42, 45, 49, 55, 62, 67, 91, 95, 97
Cluster 3: 2, 5, 10, 12, 14, 15, 16, 20, 22, 23, 25, 29, 31, 33, 34, 35, 37, 40, 43, 46, 51, 56, 57, 59, 60, 61, 63, 66, 70, 71, 73, 75, 77, 80, 81, 83, 85, 86, 88, 90, 92, 94, 99, 100
Cluster 4: 1, 7, 9, 11, 13, 18, 21, 24, 30, 32, 38, 41, 44, 47, 48, 50, 52, 53, 54, 58, 64, 65, 68, 69, 74, 78, 79, 82, 84, 87, 93, 96, 98 143,341.18 174,843.29
5 Cluster 1: 8, 17, 24, 26, 38, 65, 72, 89
Cluster 2: 2, 10, 12, 13, 15, 16, 21, 22, 23, 29, 30, 32, 33, 37, 40, 41, 44, 47, 48, 50, 51, 52, 56, 71, 73, 77, 78, 79, 84, 87, 90, 93, 96, 98
Cluster 3: 5, 14, 20, 25, 31, 34, 35, 43, 46, 57, 59, 60, 61, 63, 66, 70, 75, 80, 81, 83, 85, 86, 88, 92, 94, 99, 100
Cluster 4: 7, 28, 36, 39, 76
Cluster 5: 1, 3, 4, 6, 9, 11, 18, 19, 27, 42, 45, 49, 53, 54, 55, 58, 62, 64, 67, 68, 69, 74, 82, 91, 95, 97 141,884.73 152,909.21
6 Cluster 1: 8, 17, 24, 26, 38, 65, 72, 89
Cluster 2: 2, 5, 10, 12, 14, 20, 22, 23, 25, 29, 31, 33, 34, 35, 37, 40, 43, 46, 51, 56, 57, 59, 60, 61, 63, 66, 70, 71, 75, 77, 80, 81, 83, 85, 86, 88, 90, 92, 94, 99, 100
Cluster 3: 1, 7, 13, 15, 16, 21, 30, 32, 41, 44, 47, 48, 50, 52, 53, 54, 58, 64, 68, 69, 73, 74, 78, 79, 82, 84, 87, 93, 96, 98
Cluster 4: 4, 6, 18, 19, 36, 39, 42, 45, 49, 62, 67, 91, 97
Cluster 5: 28, 76
Cluster 6: 3, 9, 11, 27, 55, 95 141,230.52 144,776.69
7 Cluster 1: 2, 10, 12, 13, 15, 16, 21, 22, 23, 29, 30, 31, 33, 37, 40, 41, 44, 47, 48, 50, 51, 56, 73, 77, 78, 79, 84, 87, 90, 93, 94, 98
Cluster 2: 5, 14, 20, 25, 34, 35, 43, 46, 57, 59, 60, 61, 63, 66, 70, 71, 75, 80, 81, 83, 85, 86, 88, 92, 99, 100,
Cluster 3: 4, 6, 9, 11, 18, 19, 42, 45, 49, 53, 55, 58, 62, 64, 67, 68, 74, 82, 95, 96, 97
Cluster 4: 28, 76,
Cluster 5: 3, 27, 91
Cluster 6: 1, 7, 32, 36, 39, 52, 54, 69
Cluster 7: 8, 17, 24, 26, 38, 65, 72, 89 140,546.79 135,787.72
8 Cluster 1: 3, 27, 95
Cluster 2: 1, 7, 24, 30, 32, 38, 53, 54, 58, 64, 65, 68, 69, 74, 82
Cluster 3: 39
Cluster 4: 9, 11, 18, 42, 62, 67,
Cluster 5: 4, 6, 19, 36, 45, 49, 55, 91, 97
Cluster 6: 5, 10, 12, 14, 20, 22, 23, 25, 29, 31, 34, 35, 37, 40, 43, 46, 56, 57, 59, 60, 61, 63, 66, 70, 71, 75, 77, 80, 81, 83, 85, 86, 88, 90, 92, 94, 99, 100
Cluster 7: 8, 17, 26, 28, 72, 76, 89
Cluster 8: 2, 13, 15, 16, 21, 33, 41, 44, 47, 48, 50, 51, 52, 73, 78, 79, 84, 87, 93, 96, 98 140,510.19 138,410.06
9 Cluster 1: 3, 9, 11, 74, 82, 95
Cluster 2: 15, 29, 30, 31, 33, 34, 40, 41, 44, 47, 50, 73, 78, 79, 84, 87, 90
Cluster 3: 8, 17, 24, 38, 65, 72, 89,
Cluster 4: 26, 28, 39, 76
Cluster 5: 5, 14, 20, 25, 35, 46, 57, 59, 60, 61, 63, 66, 70, 75, 80, 81, 83, 85, 86, 88, 92, 100
Cluster 6: 4, 6, 18, 19, 27, 36, 42, 45, 49, 55, 62, 67, 91, 97
Cluster 7: 2, 10, 16, 21, 22, 37, 43, 51, 71, 77, 94, 98, 99
Cluster 8: 1, 7, 13, 32, 48, 52, 53, 54, 58, 64, 68, 69, 93, 96
Cluster 9: 12, 23, 56 140,889.59 126,846.88
10 Cluster 1: 28, 76
Cluster 2: 91
Cluster 3: 30, 32, 53, 54, 58, 64, 68, 69, 74, 82
Cluster 4: 5, 14, 20, 25, 35, 43, 46, 57, 59, 60, 61, 63, 66, 70, 71, 75, 80, 81, 83, 85, 86, 88, 92, 99, 100
Cluster 5: 27
Cluster 6: 9
Cluster 7: 3, 4, 6, 11, 18, 19, 42, 45, 49, 55, 62, 67, 95, 97
Cluster 8: 1, 7, 13, 21, 36, 39, 48, 52, 93, 96
Cluster 9: 2, 10, 12, 15, 16, 22, 23, 29, 31, 33, 34, 37, 40, 41, 44, 47, 50, 51, 56, 73, 77, 78, 79, 84, 87, 90, 94, 98
Cluster 10: 8, 17, 24, 26, 38, 65, 72, 89 140,789.73 117,432.09
Figure 4.6 – Cluster Members, BIC Score and SSE Score for Euclidean k-means Clustering

Cluster Recall Precision F-measure
1 48.00% 22.22% 0.30
2 17.30% 28.20% 0.21
Overall 32.65% 25.21% 0.26
Figure 4.7 – Recall, Precision, F-Measure for Euclidean k-means Clustering

In these results, we can see that, as the number of clusters increases, the squared sum of errors decreases. This is because we are including fewer dissimilar items in each of the clusters, so they more accurately represent the true nature of that cluster. I would also expect the precision, recall, and F measure to increase as the number of clusters increase as well. However, it would become harder to interpret the actual “class” of the clusters as k increases as we were instructed to disregard the class of each instance in the dataset. Additionally, we might achieve better results if we used decision trees to identify the most influential personality test markers and then used a subset of those markers for clustering.

Based on all results, while not highly accurate, prison officials could obtain good insight into what attributes are the most important in regards to who might be on-going offenders.

5. Conclusion

In this research project, different methods of constructing clusters were explored. Additionally, different distance measures were implemented and then analyzed to see how they affected the accuracy of the clusters created.

While the results of this project show only a maximum of 67% accuracy, clustering is still a valid machine learning technique. With an more advanced algorithm and an increased size dataset, clustering may be able to predict predators and victims at a much better rate.

References

Abeel, T.; de Peer, Y. V. & Saeys, Y. Java-ML: A Machine Learning Library, Journal of Machine Learning Research, 2009, 10, 931-934

Alpaydin, E. Introduction to Machine Learning, Second Edition. The MIT Press, Cambridge, MA. 2010.

Coolidge, F. Statistics A Gentle Introduction, 2nd edition. SAGE Publications, Inc. 2006.

Dunham, M. Data Mining: Introductory and Advanced Topics. Prentice-Hall. 2002.

Saha, S., Bandyopadhya, S. Performance Evaluation of Some Symmetry-Based Cluster Validity Indexes. IEEE Transactions on Systems, Man and Cybernetics – Part C: Applications and Review. Vol. 39, No. 4. July 2009.

Jain, A.K., Murty, M.N., Flynn, P.J. Data clustering: a review. ACM Computing Surveys. Vol. 31, No. 3. Sept. 1999.

Abstract—A common problem for many web sites is appearing as a high ranking in search engines.  Many searchers make quick decisions about which result they will choose and unless a web site appears within a certain threshold of the top rankings, the site may have a low throughput from search engines. Search engine optimization, while widely regarded as a difficult art, provides a simple framework for improving the volume and or quality of traffic to a web site that uses those techniques.  This paper is a survey of how search engines work broadly, SEO guidelines and a practical case study using the University of Colorado at Colorado Springs’ Engineering and Science department home page.

Index Terms—search engine optimization, search indexing, keyword discovery, web conversion cycle, optimized search, organic search, search marketing

1.  Introduction

Search Engines are the main portal for finding information on the web.  Millions of users each day make searches on the internet looking to purchase products or find information.  Additionally, users generally only look at the first few results.  If a website is not on the first couple pages it will rarely be visited.  All these factors give rise to many techniques employed to raise a website’s search engine ranking.

Search engine optimization (SEO) is the practice of optimizing a web site so that the website will rank high for particular queries on a search engine.  Effective SEO involves understanding how search engines operate, making goals and measuring progress, and a fair amount of trial and error.

Discussed in the following sections is a review of search engine technology, a step by step guide of SEO practices, lessons learned during the promotion of a website, and recommendations for the Engineering and Applied Science College of the University of Colorado at Colorado Springs.

In this survey, we spend our research efforts determining how SEO affects organic search results and do not consider paid inclusion.  For more information on paid inclusion SEO, see [1].

2.  A Review of Search Engine Technology

Search engine optimization techniques can be organized into two categories: white hat, and black hat.  White hat practices seek to raise search engine rankings by providing good and useful content.  On the other hand, black hat techniques seek to trick the search engine into thinking that a website should be ranked high when in fact the page provides very little useful content.  These non-useful pages are called web spam because from the user’s perspective they are undesired web pages.

The goal of the search engine is to provide useful search results to a user.  The growth of web search has been an arms race where the search engine develops techniques to demote spam pages, and then websites again try to manipulate their pages to the top of popular queries.  Search engine optimization is a careful balance of working to convince the search engine that a page is relevant and worthwhile for certain searches while making sure that the page is not marked as web spam.

The accomplishment of this objective requires understanding of how a search engine ranks pages, and how a search engine marks pages as spam.  Although the exact algorithms of commercial search engines are unknown, many papers have been published giving the general ideas that search engines may employ.  Additionally, empirical evidence of a search engine’s rankings can give clues to its methods.  Search engine technology involves many systems such as crawling, indexing, authority evaluation, and query time ranking.

2.1  Crawling

Most of the work of the search engine is done long before the user ever enters a query.  Search engines crawl the web continually, downloading content from millions of websites.  These crawlers start with seed pages and follow all the links that are contained on all the pages that they download.  For a url to ever appear in a search result page, the url must first be crawled.  Search engine crawlers present themselves in the HTTP traffic with a particular user-agent when they are crawling which websites can log.

In the eyes of search engine crawlers not all websites are created equal.  In fact some sites are crawled more often than others.  Crawlers apply sophisticated algorithms to prioritize which sites to visit.  These algorithms can determine how often a webpage changes, and how much is changing. [7]  Additionally, depending on how important a website is, the crawler may crawl some site more deeply.  One goal of a search engine optimizer is to have web crawlers crawl the website often and to crawl all of the pages on the site.

Search crawlers look for special files in the root of the domain.  First they look for a “robots.txt” file which tells the crawler which sites not to crawl.  This can be useful to tell the crawler not to crawl admin pages because only certain people have access to these pages.  The robots.txt file may contain a reference to an XML site map.  The site map is an XML document which lists every page that is a part of the website.  This can be helpful for dynamically generated websites where the sites may not all necessarily have links.  An example of this is on the Wikipedia site.  Wikipedia has a great search interface, but crawlers do not know how to use the search or what to search for.  Wikipedia has a site map which lists all the pages it contains.  This enables the crawler to know for sure that it has crawled every page.  When optimizing a website, it is important to make it easy for a site to be completely crawled.  Crawlers are timid to get caught in infinite loops of pages or download content that will never be used in search result pages. [5]

2.2  Indexing

After the search engine has crawled a particular page, it then analyzes the page.  In its most simple form the search engine will throw away all the html tags and simply look at the text.  More sophisticated algorithms look more closely at the structure of the document to determine which sections relate to each other, which sections contain navigational content, which sections have large text, and weight the sections accordingly. [3]  Once the text is extracted, information retrieval techniques are used to create an index.  The index is a listing of every keyword on every webpage.  This index can be thought of as the index in a book.  The index tells which page the subject is mentioned; however, in this case the index tells which website contains particular words.  The index also contains a score for how many times the word appears normalized by the length of the document.  When creating the index, the words are first converted into their root form.  For example the word “tables” is converted to “table”, “running” is converted to “run”, and so forth.

In order for a page to appear in the search engine results page, the page must contain the words that were searched for.  When promoting a website, particular queries should be targeted, and these words should be put on the page as much as possible.  However, this kind of promoting is abused.  Indeed, some sites are just pages and pages of keywords with advertisements designed to get a high search engine ranking.  Other tactics include putting large lists of keywords on the page, but making the keywords only visible to the search engine.  In fact some search engines actually parse the html and css of the page to find words that are being hidden to the user.  These kinds of tactics can easily flag a site as being a spam site.  Therefore one should make sure that the targeted keywords are not used in excess and text is not intentionally hidden.

One problem that many pages have is using images to display text.  Search engines do not bother trying to read the text on images.  This can be damaging to a site especially when the logo is an image.  A large portion of searches on the internet are navigational searches.  In these searches, people are looking for a particular page.  For these kinds of searches, it is hard to rank high in the search engine result page when the company brand name is not on the page except in an image.  One alternative is to provide text in the alt field of the image tag.  This may not be the best option however because the text in the alt field is text that the user does not normally see and is therefore prone to keyword abuse; hence, suspicious to the search engine.

2.3  Authority Evaluation

One of the major factors that sets Google apart from other search engines is its early use of PageRank as an additional factor when ranking search result. [3]  Google found that because the web is not a homogeneous collection of useful documents, many searches would yield spam pages, or pages that were not useful.  In an effort to combat this, Google extracted the link structure of all the documents.  The links between pages are viewed as a sort of endorsement from one page to another.  The idea is that if Page A links to Page B, then the quality of Page B is as high or higher than that of Page A.

The mathematics behind this model is a model of a random web surfer.  The web surfer starts on any page on the internet.  With the probability of 85% the random web surfer will follow a link on the page.  With the probability of 15%, the web surfer will go to a random page on the internet.  The PageRank of a given page is the probability that the user will be visiting that page.

One of the problems with this model is that any page on the internet will have some PageRank, even if it is never linked to.  This baseline page rank can be used to artificially boost the PageRank of another page.  Creating thousands of pages that all link to a target page can generate a high PageRank for the target page.  These pages are called link farms.

Many techniques have been proposed to combat link farms.  In the “Trust-rank” algorithm [4], a small number of pages are reviewed manually to determine which pages are good.  These pages are used as the pages that the web surfer will go to 15% of the time.  This effectively eliminates the ability to create link farms because a random page on the internet must have at least one page with trust-rank linking to it before it has any trust-rank.   Another technique is to use actual user data to determine which sites are spam.  By tracking which sites a user visits and how long the visits are, a search engine can determine which are the most useful pages for a given query. [6]

2.4  Result Ranking

With traditional information retrieval techniques, the results are ranked according to how often the query terms appear on a page.  Additionally, the query terms are weighted according to how often they occur in any document.  For example, the word “the” is so common in the English language that it is weighted very low.  On the other hand the word “infrequent” occurs much less often, so it would be given a higher weight.
During the beginning of the World Wide Web, search engines simply ranked pages according to the words on the page compared to the page’s length and relative frequency of the words.  This algorithm is relative simple to fool.  One can optimize landing pages to contain high frequencies of the targeted query terms.  To combat this, the relevance score and the authority score are combined to determine the final ranking.  In order for a document to appear in the search engine result page it must match the keywords in the query, but the final ranking is largely determined by the authority score.

3. Guidelines to SEO

Equipped with the knowledge of how search engines index and rank web sites, one has to tailor content in such a way that gives the best opportunity for being ranked highly.  There are many guidelines to SEO, but we attempt to distill the most important ones into our survey here.  For a more complete picture of SEO best practices and information see, [1].

3.1  Refine The Search Campaign

The first thing that the reader must consider is why are they going to do SEO on their web site?  In a broad sense, the purpose is to get more traffic volume or quality to one’s web site, but one needs to break this down into smaller subgoals.  We will look at defining the target audience to determine why people use search in the first place and then we will examine how one can leverage the searchers intent to design a web site that will best serve the audience.

3.2  Define The Target Audience

Typically searchers aren’t really sure on what they are looking for.  After all, that is why they are performing a search in the first place.  When designing a web site, the webmaster must be cognizant of the searcher’s intent.

Generally, a searcher’s intent can be broken down into 3 different categories.  Know which category a particular searcher might fall into is important in how one designs your web site and which keywords they might choose.

• Navigational searchers want to find a specific web site, such as JPMorgan Chase and might use keywords such as “jp morgan chase investments web site.”  Typically, navigational searchers are looking for very specific information “because they have visited in the past, someone has told them about it because they have heard of a company and they just assume the site exists.  Unlike other types of searchers, there is only one right answer.” [1]  It is important to note that typically navigational searchers just want to get to the home page of the web site – not deep information.  With navigational searches, it is possible to bring up multiple results with the same name (i.e. A Few Guys Painting and A Few Guys Coding).
• Informational searchers want information on a particular subject to answer a question they have or to learn more about a particular subject.  A typical query for a informational searcher might be “how do I write iPhone applications.”  Unlike navigational searchers, informational searchers typically want deep information on a particular subject – they just don’t know where it exists.  Typically, “information queries don’t have a single right answer.” [1]  By far, this type of search dominates the types of searches that people perform and therefore the key to having a high ranking in informational searches is to choose the right keywords for your web site.
• Transactional searchers want to do something, whether it is purchase an item, sign up for a newsletter, etc. A sample transactional search query might be “colorado rockies tickets.”  Transactional queries are the hardest type of query to SEO for because “the queries are often related to specific products.” [1]  The fact that there are many retailers or companies that provide the same services only complicates matters.  In addition, it can be hard to dipher between whether the searcher wants information or a transaction (i.e. “Canon EOS XSi”).

After understanding the target searcher, one has to consider how the searcher will consume the information they find.  According to Hunt, et. al., “nearly all users look at the first two or three organic search results, less time looking at results ranked below #3 and far less time scanning results that rank seventh and below.  Eighty-three percent report scrolling down only when they don’t find the result they want within the first three results.”

So what does this mean for people doing SEO?  Choosing the correct keywords and descriptions to go on your site is probably the most important action one can take.  Searchers choose a search result fairly quickly, and they do so by evaluating 4 different pieces of information, included with every result, which include: URL, title, the snippet, and “other” factors.

Graph 3.1 – Identifying the percentages of the 4 influencing factors of a search result. [1]

3.3  Identify Web Site Goals

Now that we have identified the types of searchers that are looking for information, lets consider why one wants to even attempt SEO in the first place.  For that, one needs to identify some goals, including on deciding the purpose of your web site.  Generally, there six types of goals including:

• Web sales.  This goal is selling goods or services online.  Note that this could be a purely online model (as in the case of Amazon.com or Buy.com) or it can be a mix between online and brick and mortar stores.  Web sales can be broken down further in specifying 1) whether or not the store is a retailer that sells many different products from different manufactures or if is a manufactures’ site and 2) what type of delivery is available (instant or traditional shipping).  In all cases, the ultimate desire is to increase the number of sales online.  This type of site would benefit from optimizing their site with transactional searchers in mind.
• Offline sales. This goal involves converting web visitors into sales at brick and mortar stores. While this type of goal would benefit from transactional optimization, the benefits from SEO are much harder to calculate because there isn’t as good a measure on sales success.  The most important concept that a webmaster of an offline sales site can remember is to always emphasize the “call-to-action”, which the thing that you are trying to get someone to do, in this case, convert from web to physical sale.
• Leads. Leads are conceptually the same as offline sales, however leads are defined by when the customer switches to the offline channel.  Customers who do some research online and then switch to offline are leads while customers who know model numbers, prices, etc. are offline sales.  With leads, the search optimization and marketing strategy needs to be different because customers who are leads are typically informational searchers instead of transactional searchers as in the case of web and offline sales.  Therefore, people who want to optimize for leads need to attract customers who are still deciding on what they want.
• Market awareness. If a goal for one’s web site is market awareness, this is an instance of where paid placement could help boost your page rank quicker than organic results simply due to the fact that the product or service isn’t well know yet.  With market awareness, the web site mainly exists to raise awareness, so you would want to the site for navigational and informational searchers.
• Information & entertainment. These sites exist solely to disseminate information or provide entertainment.  They typically don’t sell anything, but might raise revenue through ad placement or premium content, such as ESPN Insider.  Sites that focus on information and entertainment should focus almost exclusively on optimizing their site for informational searchers.
• Persuasion.  Persuasion websites are typically designed to influence public opinion or provide help to people.  They usually not designed to make money.  In order to reach the most people, sites like these should be designed for a informational searcher.

3.4  Define SEO Success

A crucial element of undertaking SEO is to determine what should be used to measure the “success” of the efforts performed.  Depending on what type of goal the webmaster or or marketing department has defined for the website (web or offline sales, leads, market awareness, information and entertainment and persuasion), there are different ways to measure success.  For example, the natural way for a site that does web or offline sales to determine success is to count the conversions (i.e. the ratio of “lookers” to “buyers”).  If a baseline conversion ratio can be established prior to SEO, you can measure the difference between the old rate and the new rate after SEO.

This logic can be applied to any of the goals presented.  For example, with market awareness, you could measure conversion by sending out surveys to consumers or perhaps including some sort of promotion or “call-to-action” that would be unique to the SEO campaign.  If a consumer were to use that particular promotion or perform that action, it would give an indication of a conversion.

Another way to determine success is to analyze the traffic to the web site and there are a couple of different metrics that can be used for this including “page views and visits and visitors.” [1]  Using the most basic calculation with page views, you could determine a per-hour, per-day, per-week, per-month or even per-year average count of visitors to your website.  Using some tracking elements (such as cookies or JavaScript), you could determine the rise (or fall) or visitors to the site before and after SEO was performed.  The amount of information that this simple metric would provide would be invaluable because you could also determine peak visiting hours, which pages had the most hits at a specific time, visitor loyalty (unique vs. returning visitors), visitor demographics (location, browser and operating system, etc.) and even the time spent per visit.

3.5  Decide on Keywords

Keywords are the most important element of SEO.  It is important that during SEO, one focuses on the keywords that searchers are most likely to use while searching.  When choosing keywords, it is important to consider several different factors such as keyword variations, density, search volume and keyword “competition.”

Keyword variations play an important role in choosing keywords because one has have to consider how an average searcher may try to find information.  For example, certain keywords such as “review or compare” might be able to be used interchangeably in a search query.  In addition, some keywords are brand names that people automatically associate with products such as “Kleenex” for tissues and “iPod” for MP3 player.  If someone is able to have more variations on a certain sequence of keywords, they have a higher likelihood of being found.

Search volume also an big factor in determining which keywords to choose because one wants to choose keywords that people are actually searching for.  If a particular keyword only has 3,000 queries a month, it is much better to use a keyword that has 20,000 queries a month because 1) the searchers are associating the higher volume keyword with whatever the subject is instead of the lower volume keyword and 2) using the higher volume keyword will reach a larger audience.  However, this is not to say that low-volume keywords have no value.  In fact, the opposite is true.  Mega-volume keywords, such as brand names, often used by companies to compete to have the highest ranking.  If one were able to achieve the same or nearly same results with lower contention keywords, the SEO process would be much easier because fewer companies target them making it easier to achieve a high ranking with them.

Lastly, keyword density is an important factor.  When deciding  where to rank your page, the search engines have it figured out that anywhere from 3%-12% of the page is a good density for keywords, with 7% being optimal.  If it detects a higher percentage than this, the search spider might consider the page spam, simply because one is trying to stuff as many high volume keywords into the page as possible without any relevant context or content to the user.  Typically, this results in a lower ranking, no ranking or sometimes the page is even removed from the index or blocked.

3.5.1  Create Landing Pages for Keyword Combinations

The last step in assessing a site under consideration for SEO would be to identify pages on your site that you want your specific keyword queries to lead to when a searcher enters that query.  For example, P&G might want “best laundry detergent” to lead to their Tide home page.  Landing pages are the pages that these queries lead to and are “designed to reinforce the searchers intent.” [1]  Each of the keywords or phrases identified in section 3.4 must lead to a landing page and those pages must be indexed.  If there isn’t a landing page already for some of those keywords, then they must be created.

3.6  Page Elements That Matter And Don’t Matter

Now that the target audience, goals, keywords and landing pages are identified, it is crucial to consider the design of the web page.

• Eliminate popup windows.  The content in popup windows are not indexed by spiders.  If important content, navigation  or links are contained inside the popup window, then it will not be seen by the spider.  The content needs to be moved outside the popup window.
• Pulldown navigation.  Pulldown navigations suffers from the same problem as popup windows.  Since spiders cannot see these elements (mouse-over or click on them), they cannot index the content, which creates a large problem if the navigation is done with pulldown menus.  Either the pulldown menu must be done in a compatible way or the site has to allow for some other alternative means of navigation.
• Simplifying dynamic URLs. Pages that use dynamic content and URLs must be simplified for the spiders to crawl.  The nature of a dynamic URL means that a spider could spend an infinite amount of time attempt to crawl all the possible URLs and that would produce a lot of duplicate content.  To deal with this, spiders will only crawl dynamic URLs if the URL has less than 2 dynamic parameters, is less than 1,000 characters long, does not contain a session identifier and every valid URL is linked from another page.
• Validating HTML.  Robots are very sensitive to correctly formed documents.  It is important that the HTML is valid for the spider to get a good indication on what the page is truly about
• Reduce dependencies.  Some technologies, such as Flash, make it impossible for the spider to index the content inside them.  The spider cannot view this particular content, so any important keywords or information inside it, is lost.  The content should be moved outside to allow indexing to take place.
• Slim down page content.  A spider’s time is valuable and typically spiders don’t crawl all the pages of a bloated web site.  Google and Yahoo! spiders stop at about 100,000 characters. [1]  The typical cause of HTML page bloat is embedded content such as styling and other content such as JavaScript.  A simple way to solve this is to link to Cascading Style Sheet (CSS) so that you are able to make use of re-useable styles.  Another way to reduce JavaScript bloat is to use a program to obfusticate long files which replace long variable names such as “longVariableName”, which much shorter versions such as “A.”
• Use redirects.  From time to time, pages within a web site move.  It is important to make accurate use of the correct type of redirects within your site so that when spiders attempt to visit the old URL, they are redirected to the new URL.  If they find that the server returns a 404 (Unavailable) for the old URL, the spider might remove that particular page from the index.  Instead, the proper way to indicate that the page has moved permanently is to use a server-side redirect, also known as a “301 redirect.”  This is returned to the spider when it attempts to navigate to the old URL and it is then able to update the index with the new URL.  A sample implementation might look like the following:

Redirect 301 /oldDirectory/oldName.html http://www.domain.com/newDirectory/newName.html

Note that spiders cannot follow JavaScript or Meta refresh directives. [1]  Additionally one can use a “302 redirect” for temporarily moved URLs.  See Fielding et. al for more information on “302 redirect.”

• Create site maps. Site maps are important for larger sites because “it not only allows spiders to access your site’s pages but they also serve as very powerful clues to the search engine as to the thematic content of your site.” [1]  The anchor text used for the link could provide some very good keywords to the the spider.
• Titles and snippets.  Together, the title and the snippet that the search spider extracts account for a large part of how they index a particular page.  The title, the most important clue to a spider on the particular subject of a page, is the most easily fixed element.  The title is a great place to use keywords that were previously decided on.  For example, the title element for StubHub, a ticket brokerage site is “Tickets at StubHub! Where Fans Buy and Sell Tickets.” Additionally, the snippet, or summary that the spider comes up with to describe their result is important as well.  Typically, the spider uses the first block of text that they run across to use for a snippet.  For example, for WebMD, Googlebot uses the following snippet “The leading source for trustworthy and timely health and medical news and information. Providing credible health information, supportive community, …”  In both of these examples, it is clear that having essential keywords present in both are highly correlated to having high page ranks.
• Formatting heading elements.  Using traditional HTML subsection formatting elements, such as <h1>, <h2>, <h3>, etc. to denote important information can help give context clues to spiders on what text is important on a particular page.

3.6.1  The Importance of Links

Links, both internal and external, play a big role in SEO and in page ranking.  Search engines place a certain value on links because they can use these link to judge the the value of the information.  Similar to how scientific papers have relied on citations to validate and confer status upon an authors work, links apply status and value to a particular site.  “Inbound links act as a surrogate for the quality and “trustworthiness” of the content, which spiders cannot discern from merely looking at the words on the page.”  [1]

Several factors can influence how an algorithm ranks the link popularity on a particular page including 1) link quality, link quantity, anchor text and link relevancy.

Using the 4 link popularity factors from above, search engines use a theory of hub and authority pages to create link value.  Hub pages are web pages that link to other pages on a similar subject.  Authority pages are pages that are linked to by many other pages on a particular subject.  Therefore, search engines usually assign a high rank to these pages because they are most closely related to a searcher’s keywords. Using this model, it is easy to see why the harder an inbound link is to get, the more valuable it might be in terms of value.  For more information on inbound and outbound link importance and their value, see Hunt et. al.

3.7  Getting Robots To Crawl The Site

In addition to all of the goals and HTML elements above, it is crucial to have a “robots.txt” file that will allow the robot to crawl the site.  This file can give instructions to web robots using the Robots Exclusion Protocol.  Before any indexing occurs at a site, the robot will check the top-level directory for the “robots.txt” file and will index the site accordingly.  A sample “robots.txt” file might look like the following.

User-agent: *
Disallow:

This instructions applies to all robots or “User-agents” and they are to index the whole site because nothing is listed under the “Disallow” directive.  It is possible to tailor this file to individual robots and server content.  While the protocol is completely advisory, it is highly recommended to improve the search quality of what the robots index.  Note that the robot does not have to obey the “robots.txt” file, however most non-malicious robots do obey the instructions.

3.8  Dispelling SEO Myths

• SEO is a one time activity.  SEO is a continual activity that must be revisited from time to time to ensure that the most up-to-date keywords and content are being used.
• SEO is a quick fix process for site traffic.  Generating high quality organic traffic that will help with conversion is a slow process.  For each change that one makes to the web page, spiders must re-index that page and then calculate the new rankings.  It could take several months to years depending on your goal of achieving a higher ranking, passing a competitor in rankings or achieving the top results spot.
• META tags help with page ranking. META tags were abused by keyword spammers by packing in as many highly searched keywords as possible, early in the days of developing search engines and many spiders now give META tags little to no credence.

4.  Promotion Of AFewGuysCoding.com

In order to test the authors hypothesis, we chose to perform our SEO experiment with AFewGuysCoding.com.  A Few Guys Coding, LLC is a small, author-owned company that provides contract engineering services mainly for mobile platforms (iPhone, iPod Touch, iPad, Android) but also does web and desktop applications.   This web site has never had SEO performed on it and was not designed with any such considerations in mind.

4.1  Initial Investigation For AFewGuysCoding.com

In order to determine what keywords we should focus on, we created a survey and asked the following question to people who are not engineers: “Suppose you were a manager of a business that had a great idea for a mobile phone application. You knew that you had to hire an iPhone, iPod or iPad developer because you didn’t directly know anyone who could do this work for you. What words, terms or phrases might you consider searching for in Google to find this person?”  Table 4.1 represents the range of responses that were provided.  In analyzing this data, the words provided were stemmed to account for different endings.  In addition, stop words, or words that are ignored because they don’t provide any significance to the query, were also filtered out and not considered.

Looking at the results and given the question posed to the survey audience, the percentages of the top three results did not surprise the authors.  What did surprise the authors were the relative high number of results for the keywords “technician”, “help”, “inventor/invention”, “creator” and “market/marketing” and the relatively low number of results for “mobile”, “phone”, “software” and “programmer.”

After careful considering, the authors chose to incorporate some of the keywords suggested by the survey audience into the web page.

Table 4.1 – Responses from a search audience regarding possible keywords for AFewGuysCoding.com

4.2  Baseline Rankings

Prior to performing SEO on AFewGuysCoding.com, the Google page rank was 1/10.  The website was indexed by major search engines, such as Google, Yahoo!, Bing, AOL and Ask, but was not crawled on a regular basis.

Before performing an SEO activities, the amount of traffic that was coming from search engines was a little over 7% overall.  Most traffic was coming from direct referrals, for example when the user entered the address from a business card they had gotten (see graph 5.1).

In addition, the titles and snippets of the page were not as good as they could be (i.e. they didn’t include keywords or readily extractable content for the snippet).  For example, the title of the home page for AFewGuysCoding.com was “Welcome to A Few Guys Coding, LLC.”

Moreover, the web site was a transition from an older web site, www.davidstites.com, so many of the links already in Google referred to old pages that no longer existed.  When those pages were clicked on in the Google results, it took the user to a “404 Unavailable” page.

Graph 5.1 – Sources of traffic for AFewGuysCoding.com before SEO activities

In addition to the work done with keyword densities and titles, the author also started a blog and Twitter account, (http://blog.afewguyscoding.com, http://www.twitter.com/afewguyscoding), that addressed computer science and programming topics.  In the blog, we linked to other parts of the main site when a topic was referenced in a blog post that A Few Guys Coding dealt with, such as iPhone applications.  A summary feed of this blog was placed on the main page AFewGuysCoding.com web site and the authors noticed an increase in crawler traffic after the spider determined the content was changing frequently enough to warrant additional visits to the site to re-calculate PageRank.

4.3  Rankings After SEO

After an initial pass of SEO was performed to the web site, the traffic from search engines increased dramatically and the page rank increased 1 point to 2/10.  The authors believe that this increase was largely from changing the titles of the web pages themselves, changing keyword densities and tying keywords to landing pages.   Using Google Analytics, over a two month period, traffic for AFewGuysCoding.com was up 81.24%.

They word densities are not quite as high as 3%-11%, however, it is a marked improvement from before, when the important keyword densities were all 1% and below.  See Table 5.2.

Table 5.2 – Keyword densities for certain pages on AFewGuysCoding.com

Graph 5.2 – Sources of traffic for AFewGuysCoding.com after SEO activities (April 2010)

Table 5.3 – Page View Overview for top 8 most visited pages in April, 2010

Graph 5.3 – Number of visitors per day for April, 2010 against Time on Site goal achievement rate

Another action that was taken by the authors was using Apache mod_rewrite and a redirect file, we we’re able to direct the spider to update their index to the new pages (from the older site) using a “301 redirect”.  We were able to transform the URL using mod_rewrite to match the current top-level domain.  This ensured that pages were not removed from the crawlers index due to being status 404.

Lastly, the authors set several goals for the web site (besides the increase in PageRank), including a “Time on Site” measure that would help measure SEO success.  If a particular user stayed on the site for longer than 5 minutes and/or had 10 or more page views, we considered this criteria meeting the goal.  See Graph 5.3 for a comparison of visitors to goals met.

4.3  Conclusions

Clearly, making simple changes to content (such as keywords and titles) can have a large effect on search engine ranking and the amount and quality of traffic that is directed to a site that has had SEO.  The difference in search engine traffic over a month represented an 400% increase. It might be reasonable to infer that an additional increase of 3-4% in keyword density might generate search engine referrals up to 50%-60%. The authors would like to see the effects of continuing SEO at a 3, 6, 9 and 12 month period.

5.  Recommendations To UCCS

Based on the research that we performed, we would like to make some suggestions to the UCCS EAS webmaster that would allow the UCCS EAS site to be ranked higher than it currently is.  Currently, the URL http://eas.uccs.edu/ has a Google PageRank of 4/10. By following the suggestions below, it may be possible to raise the Page Rank 1 or 2 points to 5/10 or 6/10.  We have broken down our recommendations into a four step process: define the scope and goals, understand the decision making process of a potential EAS student, create content, and build links.

5.1  Define the Scope and Goals

The first step is to define the scope and goals of the Search Engine Optimization efforts.  One question to ask is: Is this effort only related to getting more EAS students, or is it to promote UCCS as a whole?  How much effort can be put into this project?  The answers to these questions will determine the scope of the project.  One potential goal is to recruit more students to the college, but a new student may have attended even without the SEO campaign.  Consequently, a plan for measuring how students became interested in the college will be important.

5.2  Understanding the Decision Making Process

Key to understanding how to boost traffic through SEO practices is understanding the content target users are searching for.  This can be discovered through surveys or interviews with students that have gone through the process of choosing a college to attend.  This can be current UCCS students or students of other universities.  From my own experience I would guess that a typical decision making process would involve finding answers to these questions: Should I go to college? What is the best school for me?  Why should I go to UCCS?  Which major should I pick? How do I apply?  Where can I find help with my application?  Each of these questions is a good area for creating good content.

5.3  Create Content

Once the decision making process is understood, content can be created to answer the questions that people are searching for.  Special attention should be made to use keywords that people would search for when they have that question.  A quick look at the current EAS site reveals that there is little content related to recruitment.  Additionally, the current pages have few keywords that are searched frequently.  Additional recommendations are listed below that highlight some of the deficiencies of the current EAS pages.

• Eliminate non-indexable content. The Adobe Flash content on the main landing page, http://www.eas.uccs.edu/ is unable to be indexed by spiders and robots.  All the information contained in that Flash element is lost.  Additionally, any images that contain content, are non-indexable as well.
• Remove “expandable” navigation.  Spiders are unable to “click” on these individual sections to expand them and are therefore unable to crawl the linked pages.  The navigation should be reworked so that all links are accessible without needing to perform any special interface actions.
• Choose better titles for individual pages.  Despite the actual pages content or purpose, all pages under the main landing page have the title “UCCS | College of Engineering and Applied Science.”  These titles should change depending on the subject or content of the page so that spiders and robots are able to create a more accurate index of the page.
• Better use of heading formatting.  Headings should use valid heading tags such as <h1>, <h2>, <h3>, etc. so that the robots that crawl the site can extract main ideas and content from the page for the index.
• Check and adjust keywords and keyword densities.  The densities of the keywords on the page are low should reflect what the page is about.  For example, on the application page for EAS, the keywords concerning admission and applying are low.  In fact, out of the top 18 keywords on the page only 4 or 5 have anything to do with admission and the densities are low, ranging from .79% to 2.37%.

• Include a site map.  A site map would help ensure that all the pages that were meant to be accessible to a web visitor are also accessible to a spider crawling the content.

5.4  Build Links

Lastly, it is important to build the authority score by getting other pages to link to the content pages.  Links can be built within organizations that already have relationships with the university.  For example the City of Colorado Springs, engineering organizations, and businesses that recruit from the EAS college could all be great sources of links.  Publishing press releases to news organizations of new websites could also be helpful in generating links.  Another form of link building could be simply link to the target pages heavily from other pages on the EAS site.

Another thing to consider is that people understand when content is on the EAS site it is going to be biased towards the EAS college.  Prospective students are not going to trust the document as much as if it were coming from a third party site.  One way to overcome this is to write content for other websites on the web.  This provides two benefits.  First it creates a seemingly unbiased source of information that can be slanted towards recruiting as EAS, and the content can link back to the EAS website providing a good link for building the authority score of a page.

By following these guidelines, the EAS college can succeed in generating more traffic on its recruitment pages and therefore seeing more students attend the college.

6.  Conclusion

In this research project, we have investigated the implementation of search engines.  We have also presented different elements that affect search ranking.  Based on our research and case study with AFewGuysCoding.com, we have provided recommendations to the UCCS EAS department to improve their PageRank within Google and other major search engines.  Indeed, search engine optimization is an important technique that any web master must master so that their site can be indexed as high in the rankings as possible.

References

1. Hunt, B, Moran, M.  Search Engine Marketing, Inc,. Driving Search Traffic to Your Company’s Web Site, 2nd ed. (2008). IBM Press.
2. Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., Berners-Lee., T.  RFC 2616 – HTTP/1.1 Status Code Definitions.  1999 [Online]. http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
3. S. Brin and L. Page. The Anatomy of a Large-Scale Hypertextual Web Search Engine. Computer Networks and ISDN Systems. 1998. pp. 107-117.
4. Z. Gyongyi, H. Garcia-Molina, and J. Pedersen. Combating Web Spam with TrustRank. Very Large Data Bases. Vol. 30, pp. 576-587. 2004.
5. O. Brandman, J. Cho, H. Garcia-Molina, N. Shivakumar. Crawler-Friendly Web Servers. ACM SIGMETRICS Performance Evaluation Review. Vol. 28, No. 2, pp. 9-14. 2000.
6. G. Rodriguez-Mula, H. Garcia-Molina, A. Paepcke.  Collaborative value filtering on the Web. Computer Networks and ISDN Systemes. Vol. 30, No. 1-7, pp. 736-738. 1998.
7. J. Cho, H. Garcia-Molina.  The Evolution of the Web and Implications for an Incremental Crawler.  Very Large Data Bases. pp. 200-209. 2000.

Knn.java – This is the main driver of the code. To do the classification, we are essentially interested in finding the distance between the particular instance we are trying to classify to other instances.  We then determine the classification of the instance we want from a “majority vote” of the other k closest instances.  Each feature of an instance is a separate class that essentially just stores a continuous or discrete value depending on if you are using regression or not to classify your neighbors.  The additional feature classes and file reader are left to the reader as an exercise.  Note that it would be fairly easy to weight features using this model depending on if you want to give one feature more clout than another in determining the neighbors.

The nice visualization of the algorithm is provided by Kardi Teknomo. As you can see, we take the number of k closest instances and use a “majority vote” to classify the instance.  While this is an extremely simple method, it is great for noisy data and large data sets.  The two drawbacks are the running time O(n^2) and the fact that we have to determine k ahead of time.  However, despite this, as shown in the previous paper, the accuracy can be quite high.

import java.util.*;

public class Knn {
	public static final String PATH_TO_DATA_FILE = "coupious.data";
	public static final int NUM_ATTRS = 9;
	public static final int K = 262;

	public static final int CATEGORY_INDEX = 0;
	public static final int DISTANCE_INDEX = 1;
	public static final int EXPIRATION_INDEX = 2;
	public static final int HANDSET_INDEX = 3;
	public static final int OFFER_INDEX = 4;
	public static final int WSACTION_INDEX = 5;
	public static final int NUM_RUNS = 1000;
	public static double averageDistance = 0;

	public static void main(String[] args) {
		ArrayList instances = null;
		ArrayList distances = null;
		ArrayList neighbors = null;
		WSAction.Action classification = null;
		Instance classificationInstance = null;
		FileReader reader = null;
		int numRuns = 0, truePositives = 0, falsePositives = 0, falseNegatives = 0, trueNegatives = 0;
		double precision = 0, recall = 0, fMeasure = 0;

		falsePositives = 1;

		reader = new FileReader(PATH_TO_DATA_FILE);
		instances = reader.buildInstances();

		do {
			classificationInstance = extractIndividualInstance(instances);

			distances = calculateDistances(instances, classificationInstance);
			neighbors = getNearestNeighbors(distances);
			classification = determineMajority(neighbors);

			System.out.println("Gathering " + K + " nearest neighbors to:");
			printClassificationInstance(classificationInstance);

			printNeighbors(neighbors);
			System.out.println("\nExpected situation result for instance: " + classification.toString());

			if(classification.toString().equals(((WSAction)classificationInstance.getAttributes().get(WSACTION_INDEX)).getAction().toString())) {
				truePositives++;
			}
			else {
				falseNegatives++;
			}
			numRuns++;

			instances.add(classificationInstance);
		} while(numRuns < NUM_RUNS);

		precision = ((double)(truePositives / (double)(truePositives + falsePositives)));
		recall = ((double)(truePositives / (double)(truePositives + falseNegatives)));
		fMeasure = ((double)(precision * recall) / (double)(precision + recall));

		System.out.println("Precision: " + precision);
		System.out.println("Recall: " + recall);
		System.out.println("F-Measure: " + fMeasure);
		System.out.println("Average distance: " + (double)(averageDistance / (double)(NUM_RUNS * K)));
	}

	public static Instance extractIndividualInstance(ArrayList instances) {
		Random generator = new Random(new Date().getTime());
		int random = generator.nextInt(instances.size() - 1);

		Instance singleInstance = instances.get(random);
		instances.remove(random);

		return singleInstance;
	}

	public static void printClassificationInstance(Instance classificationInstance) {
		for(Feature f : classificationInstance.getAttributes()) {
			System.out.print(f.getName() + ": ");
			if(f instanceof Category) {
				System.out.println(((Category)f).getCategory().toString());
			}
			else if(f instanceof Distance) {
				System.out.println(((Distance)f).getDistance().toString());
			}
			else if (f instanceof Expiration) {
				System.out.println(((Expiration)f).getExpiry().toString());
			}
			else if (f instanceof Handset) {
				System.out.print(((Handset)f).getOs().toString() + ", ");
				System.out.println(((Handset)f).getDevice().toString());
			}
			else if (f instanceof Offer) {
				System.out.println(((Offer)f).getOfferType().toString());
			}
			else if (f instanceof WSAction) {
				System.out.println(((WSAction)f).getAction().toString());
			}
		}
	}

	public static void printNeighbors(ArrayList neighbors) {
		int i = 0;
		for(Neighbor neighbor : neighbors) {
			Instance instance = neighbor.getInstance();

			System.out.println("\nNeighbor " + (i + 1) + ", distance: " + neighbor.getDistance());
			i++;
			for(Feature f : instance.getAttributes()) {
				System.out.print(f.getName() + ": ");
				if(f instanceof Category) {
					System.out.println(((Category)f).getCategory().toString());
				}
				else if(f instanceof Distance) {
					System.out.println(((Distance)f).getDistance().toString());
				}
				else if (f instanceof Expiration) {
					System.out.println(((Expiration)f).getExpiry().toString());
				}
				else if (f instanceof Handset) {
					System.out.print(((Handset)f).getOs().toString() + ", ");
					System.out.println(((Handset)f).getDevice().toString());
				}
				else if (f instanceof Offer) {
					System.out.println(((Offer)f).getOfferType().toString());
				}
				else if (f instanceof WSAction) {
					System.out.println(((WSAction)f).getAction().toString());
				}
			}
		}
	}

	public static WSAction.Action determineMajority(ArrayList neighbors) {
		int yea = 0, ney = 0;

		for(int i = 0; i < neighbors.size(); i++) { 			Neighbor neighbor = neighbors.get(i); 			Instance instance = neighbor.getInstance(); 			if(instance.isRedeemed()) { 				yea++; 			} 			else { 				ney++; 			} 		} 		 		if(yea > ney) {
			return WSAction.Action.Redeem;
		}
		else {
			return WSAction.Action.Hit;
		}
	}

	public static ArrayList getNearestNeighbors(ArrayList distances) {
		ArrayList neighbors = new ArrayList();

		for(int i = 0; i < K; i++) {
			averageDistance += distances.get(i).getDistance();
			neighbors.add(distances.get(i));
		}

		return neighbors;
	}

	public static ArrayList calculateDistances(ArrayList instances, Instance singleInstance) {
		ArrayList distances = new ArrayList();
		Neighbor neighbor = null;
		int distance = 0;

		for(int i = 0; i < instances.size(); i++) {
			Instance instance = instances.get(i);
			distance = 0;
			neighbor = new Neighbor();

			// for each feature, go through and calculate the "distance"
			for(Feature f : instance.getAttributes()) {
				if(f instanceof Category) {
					Category.Categories cat = ((Category) f).getCategory();
					Category singleInstanceCat = (Category)singleInstance.getAttributes().get(CATEGORY_INDEX);
					distance += Math.pow((cat.ordinal() - singleInstanceCat.getCategory().ordinal()), 2);
				}
				else if(f instanceof Distance) {
					Distance.DistanceRange dist = ((Distance) f).getDistance();
					Distance singleInstanceDist = (Distance)singleInstance.getAttributes().get(DISTANCE_INDEX);
					distance += Math.pow((dist.ordinal() - singleInstanceDist.getDistance().ordinal()), 2);
				}
				else if (f instanceof Expiration) {
					Expiration.Expiry exp = ((Expiration) f).getExpiry();
					Expiration singleInstanceExp = (Expiration)singleInstance.getAttributes().get(EXPIRATION_INDEX);
					distance += Math.pow((exp.ordinal() - singleInstanceExp.getExpiry().ordinal()), 2);
				}
				else if (f instanceof Handset) {
					// there are two calculations needed here, one for device, one for OS
					Handset.Device device = ((Handset) f).getDevice();
					Handset singleInstanceDevice = (Handset)singleInstance.getAttributes().get(HANDSET_INDEX);
					distance += Math.pow((device.ordinal() - singleInstanceDevice.getDevice().ordinal()), 2);

					Handset.OS os = ((Handset) f).getOs();
					Handset singleInstanceOs = (Handset)singleInstance.getAttributes().get(HANDSET_INDEX);
					distance += Math.pow((os.ordinal() - singleInstanceOs.getOs().ordinal()), 2);
				}
				else if (f instanceof Offer) {
					Offer.OfferType offer = ((Offer) f).getOfferType();
					Offer singleInstanceOffer = (Offer)singleInstance.getAttributes().get(OFFER_INDEX);
					distance += Math.pow((offer.ordinal() - singleInstanceOffer.getOfferType().ordinal()), 2);
				}
				else if (f instanceof WSAction) {
					WSAction.Action action = ((WSAction) f).getAction();
					WSAction singleInstanceAction = (WSAction)singleInstance.getAttributes().get(WSACTION_INDEX);
					distance += Math.pow((action.ordinal() - singleInstanceAction.getAction().ordinal()), 2);
				}
				else {
					System.out.println("Unknown category in distance calculation.  Exiting for debug: " + f);
					System.exit(1);
				}
			}
			neighbor.setDistance(distance);
			neighbor.setInstance(instance);

			distances.add(neighbor);
		}

		for (int i = 0; i < distances.size(); i++) {
			for (int j = 0; j < distances.size() - i - 1; j++) { 				if(distances.get(j).getDistance() > distances.get(j + 1).getDistance()) {
					Neighbor tempNeighbor = distances.get(j);
					distances.set(j, distances.get(j + 1));
					distances.set(j + 1, tempNeighbor);
				}
			}
		}

		return distances;
	}

}

So, it really bugs (read: annoys) me when users criticize the application that they are using without providing any helpful feed back, what they don’t like or even what went wrong!  Help us help you – we can’t do anything unless you provide us some information.  We aren’t expecting you to debug the application yourself or even send us crash logs – just a general, accurate description of the problem.  However, don’t think that the responsibility is completely on the consumer.  Developers have to at least attempt to easily facilitate the collection of this information.

The good example the Apple App store.  For those of you who aren’t familiar with the rating systems on there, you can rate the app after you download it with a 1-5 star designation.  You also can enter a comment along with your rating.  If you break down what kind of reviews people leave, there are several main categories: 1) the faithful user who expounds the virtues of the application and leaves a glowing review, 2) an impossible to please user that will give your app a 3 star rating for giving the old “college try” (i.e. “This app is good but would be better if it had “x, y and z” feature) 3) a user that leaves a good comment, but poor star rating or vice versa 4) a user that reports bugs and gives a low star rating (i.e. “I updated the app and every time I attempt to launch it, the app crashes!  please fix!!!!” and 5) the type of user who’s review consists of no helpful information and a low rating (i.e “this app sucks” or “the worst application ever”)

To give you an idea of how broken the system is, here is a small sampling of some of the typical reviews found from the 5 categories above.  Note: These comments are taken in the original form and have not been edited.

• “You get what you pay for.  A BIG FAT ZERO.  WHAT A JOKE.” (1 star)
• “This app is garbage.” (1 star)
• “Pure crap.  Sorry devs, but this is bad.” (1 star)
• “The best app ever.” (5 stars)
• “A great way to save money.” (1 star)
• “I love the idea. But every time load the app it thinks for 30 seconds and returns to the home screen.  Good luck guys!.” (3 stars)

If you browse the comments for any given app, you can see they are all over the place.  Let me clear however – good apps deserve good ratings and bad apps deserve bad ratings.  However, you are able to rate apps without even downloading them to try them.  As proof of how ridiculous this is, the night before the iPad launched, iTunes users were able to go to the iTunes app store and download iPad apps.  Now, absolutely no one had an iPad to test the applications with, however the ratings were still pouring in.  How could end users possibly evaluate the application without even running?

Another part of the system that is broken is that the only time that the iPhone prompts you to rate applications is when you delete them off your phone.  Typically, if you’re deleting something, something went horribly, horribly wrong (i.e. the application crashes all the time, won’t run, you don’t have a use for it anymore).  At this point, the user’s rating will most likely be negative, which will end up affecting the rating of your application.

Lastly, there is no way to disable ratings, edit ratings or reset ratings.  The only way they get “reset” is if you release a new version, but even then the previous ratings don’t disappear – the iTunes App store only separates the ratings into “This version” and “Previous versions.”  I am not saying that you should be able to “hide” the previously poor versions, but they shouldn’t affect the overall rating because every piece of software has known and unknown bugs.  The “overall” rating is the lifetime rating of the application.  Anyone who understands statistics averages fairly well knows that once you get some bad ratings that bring your overall rating down, it takes a LOT more ratings to improve the average.  For example, suppose we had 10,000 ratings that gave us an overall of 2.5/5 stars.  It would require another 5,000 5-star ratings to raise the app rating 1/2 star to 3/5 stars.  It only takes longer to raise your rating with additional lower ratings.  This is stacked against the developer and isn’t fair.

So how can we improve the process so that both the consumer and the developer is satisfied, where the consumer gets to voice their opinion and the developer gets to fix the product. First of all, lets remove the functionality that allows someone to rate an app as they are removing it.  Let’s also can the the ability to rate apps without even downloading them.  If we were able to prompt the user for a rating after they had used the app n times or after some total accumulated usage time t (tracked by iTunes) then we might get a far more unbiased opinion.  After meeting these requirements, we could then unlock the ability to rate the app or iTunes could even prompt us for our thoughts (similar to how Amazon and eBay do after you’ve placed an order and they follow up with you to check on the service).

Another idea is that even though this might deter a few people from leaving ratings, good AND bad, have a required comment field that would allow someone to elaborate on why an app deserves a particular rating.  If the user can’t take the 2 minutes to fill out a simple rating form, then they can’t be too serious about rating the application.  This could even be taken one step further by allowing the Apple or the developer to create a questionnaire with different fields depending on the rating.  eBay has a very good of example of this by listing categories of behavoirs that are encouraged. In the same vein, you could also introduce categories.  Obviously this last addition is probably overkill because it is more complicated to implement.

Lastly, it might be helpful to have a rating system within a rating system where other users could click a “thumbs up” or “thumbs down” to indicate whether another user’s review was particularly helpful or completely off base.

While no rating system is absolutely perfect, we want to get the best information possible about a potential purchase. While a rating system should be easy for the user to express their opinion, it shouldn’t be so simple as to give them all the power or reduce the rating to a simple binary “Good” or “Bad” rating.

Of course, the other alternative to all this is that, we as developers could just produce bug-free software the first time we have a app release.

I know that using FUSE is not the fastest method of backing up, so you’re mileage may vary depending on your tolerance levels and needs.  The actual download site for FuseOverAmazon is here.  Also, I am using rsync because I believe that incremental (differential) backups are far more efficient and cost/time saving than full backups every week.

1.  The first step is to install all the dependencies we’ll need for FUSE:

sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev

Next, install the most recent version of s3fs. As of now the most recent is r191, but here is a link to the downloads section so that you can check to see which version is the most up-to-date. I chose to put my src download in /usr/local/src.

wget http://s3fs.googlecode.com/files/s3fs-r191-source.tar.gz
tar -xzf s3fs*
cd s3fs
make
sudo make install
sudo mkdir /backup/s3
sudo chown yourusername:yourusername /backup/s3

2. Scripting your backup plan:

You’ll need to create a bucket on the S3 cloud.  If you haven’t done this already, you can use an online tool like JetS3t (my favorite).  I would recommend that you create a separate bucket for each logical site you are going to backup.  For example, I backup each one of my repositories in Unfuddle in a different bucket.  That makes restoring easier.  You might also want to consider replicating to multiple locations, if you don’t trust that Amazon can keep your data safe or even use a separate service provider like JungleDisk, Mozy or Backblaze.

Using gvim or TextMate (or some other text editor), we are going to automate mounting the volume, perform a sync and unmount the volume.  The reason I unmount is for safety.  If somehow the hard disk becomes corrupted, I have a bit of time to prevent the script from running and replicating the bad data.  If the volume is constantly mounted, that may not be the case.  It is also easy to wipe out the volume if you aren’t careful.

The following will be the script in your backup script, s3fs-backup.sh (or whatever you name yours):

#!/bin/bash

/usr/bin/s3fs yourbucket -o accessKeyId=yourS3key -o secretAccessKey=yourS3secretkey /mnt/s3
/usr/bin/rsync -avz --delete /home/username/dir/you/want/to/backup /mnt/s3
/usr/bin/rsync -avz --log-file=log.file --delete --exclude /sys --exclude /mnt --exclude /proc --exclude /tmp / /mnt/s3 #exclude some directories
mail -s "backup complete with log" user@host.org < log.file #email yourself the log
mv log.file log.file.`date +"%Y%m%d%H%M%S"` # move the file to a log with a datetime stamp
/bin/umount /mnt/s3

There some directories that I don’t want to backup, one being proc, because the that directory is manged by the OS while the system is running. You don’t want to restore this directory. Also, even though rsync is smart enough to recognize cycles, we don’t want to backup our /mnt/s3 directory. We exclude those here. Note, the –delete option. This will delete any files that have been removed on the ’source’. Lastly, note that we can increase/decrease the verbosity of the script and email ourselves a transcript of the backup session so we know that it actually took place – not a bad way to keep tabs.  After we are finished emailing ourselves (the potentially massive log file), we rename it to keep track of our backups on the server as well. There are many more options with rsync, so check out the man pages for the command to customize your script.

chmod 755 s3fs-backup.sh

Before you run the entire script, you might want to use the line above to change the permissions on the script you just saved.  You can verify the integrity of the script by running each command individually, which isn’t a bad idea after editing it for your own situation because mistakes do happen.  A quick check after the S3 volume (df -h) is mounted will show 256T available for your own personal use.

The most important part is automating the backup process.  If you forget and you lose your most recent data, then what was the point!?  We are going to use good ol’ fashioned *nix cron daemon to handle this process for us. There are two options for creating your crontab.  You can either put this script (or a softlink) to it in your cron.hourly, cron.daily, cron.weekly, cron.monthly folder or you can directly edit the crontab file to have more control over when the script runs.  I personally run mine every hour and every week on Sunday.  Here is a nice cron reference to customize your schedule.

crontab -e
* * * * * /path/to/s3fs.sh # this runs it hourly
0 0 0 0 0 /path/to/s3fs.sh # this runs it every week on sunday

A note about speed: The initial backup could take a long time.  The server up-stream speed is the limiting factor on how long this takes.  While rsync is a great program, using FUSE is not the speediest option in the world. There is another solution out there called ‘s3sync.’

To run the script initially and create your first back-up (if you can’t wait), simply run this command: sudo ./s3fs.sh.

One last nice thing is that this can be adapted to run anywhere, other servers, your home computers, etc.  If you can install Ruby and the dependencies above, you can have ultra cheap backups without a lot of hassle.

That’s it!

Index Terms—machine learning, recommender systems, supervised learning, nearest neighbor, classification

1. INTRODUCTION

Recommendations are a part of everyday life. An individual constantly receives recommendations from friends, family, salespeople and Internet resources, such as online reviews. We want to make the most informed choices possible about decisions in our daily life. For example, when buying a flat screen TV, we want to have the best resolution, size and refresh rate for the money. There are many factors that influence our decisions – budget, time, product features and most importantly, previous experience. We can analyze all the factors that led up to the decision and then make a conclusion or decision based on those results. A recommender system uses historical data to recommend new items that may of be interest to a particular user.

Coupious is a mobile phone application that gives it’s user coupons and deals for businesses around their geographic location. The application runs on the user’s cell phone and automatically retrieves coupons based upon GPS coordinates. Redemption of these coupons is as simple as tapping a “Use Now” button. Coupious’ services is now available on the iPhone, iPod Touch, and Android platforms. Coupious is currently available in Minneapolis, MN, West Lafayette, IN at Purdue University and Berkley, CA. Clip Mobile is the Canada based version of Coupious that is currently available in Toronto.

Using push technology, it is possible to integrate a mobile recommendation system into Coupious. The benefit of this would be threefold: 1) offer the customers the best possible coupons based on their personal spending habits – if a user feels they received a “good deal” with Coupious, they would be more likely to use it again and integrate it into their bargain shopping strategy, 2) offer businesses the ability to capitalize on their market demographics – the ability to reach individual customers to provide goods or services would drive home more revenue and add value to the product and 3) adding coupons to the service would immediately make the system more useful to a user as it would present, desirable, geographically proximate offers without extraneous offers.

1.1 OUTLINE OF RESEARCH

In this research project, we evaluated the batch k-nearest neighbors algorithm in Java. We chose to write the implementation of the decision tree in Java because of the ease of use of the language. Java also presented superior capabilities in working with and parsing data from files. Using Java allowed the author to more efficiently model the problem through the use of OO concepts, such as polymorphism and inheritance. The kNN algorithm was originally suggested by Donald Knuth in [9] where it was referenced as the post-office problem, where one would want to assign residences to the nearest post office.

The goal of this research was to find a solution that we felt would be successful in achieving the highest rate of coupon redemption. Presumptively, in achieving the highest rate of redemption required learning what the user likes with the smallest error percentage. Additionally, we wanted to know if increasing the attributes used for computation, would effect the quality of the result set.

1.2 DATA

The data that we used is from the Coupious production database. Currently, there are approximately 70,000 rows of data (“nearby” queries, impressions and details impressions) and approximately 3,400 of those represent actual coupon redemptions. The data is an aggregate from March 25th, 2009 until February 11, 2010. The results are from a mixture of different cities where Coupious is currently in production.

From a logical standpoint, Coupious is simply a conduit through which a user may earn his discount and has no vested interest in whether or not a user redeems a coupon in a particular session. However, from a business standpoint, Coupious markets the product based on being able to entice sales through coupon redemption. Therefore, for classification purposes, sessions that ended in one or more redemptions will be labeled +1 and sessions that ended without redemption will be labeled -1.

1.3 PREVIOUS WORK

While there hasn’t been any previous work in the space of mobile recommendation systems, there has been a large amount of work in the recommender systems space and classification. In [1], [2] and [3], direct marketing is studied using collaborative filtering. In [3], the authors use SVMs and latent class models to model predictions of whether or not a customer would be likely to buy a particular product. The most direct comparison of work would be in [1] and [8], where SVMs and linear classifiers are used to cluster content driven recommender systems.

2. APPLICATION

Broadly, recommender systems can be grouped into two categories, content-based and collaborative based. In content-based systems, the recommendations are based solely on the attributes of the actual product. For example, in Coupious, attributes of a particular coupon redemption includes the distance from the merchant when the coupon was used, the date and time of the redemption, the category of the coupon, the expiry and the offer text. These attributes are physical characteristics of the coupon. Recommendations can be made to users without relying on any experience-based information.

In collaboration systems, recommendations are provided based on not only product attributes but the overlap of preferences of “like-minded” people, first introduced by Goldberg et. al [5]. For example, a user is asked to rate how well they liked the product or give an opinion of a movie. This provides the algorithm a base line of preference for a particular user, which allows the algorithm to associate product attributes with a positive or negative response. Since Coupious does not ask for user ratings, this paper will focus exclusively on content-based applications.

Many content-based systems have similar or common attributes. As stated in [3], the “central problem of content-based recommendation is identifying a sufficiently large set of key attributes.” If the attribute set is too small, there may not be enough information for the program to build a profile of the user. Conversely, if there are too many attributes, the program won’t be able to identify the truly influential attributes, which leads to poor performance [6]. Also, while the label for a particular feature vector with Coupious data is +1 or -1, many of the features in the data are multi-valued attributes (such as distance, date-time stamps, etc.), which maybe hard to represent in a binary manner, if the algorithm requires it.

In feature selection, we are “interested in finding k of the d dimensions that give us the most information and accuracy and we discard the other (d – k) dimensions [4].” How can we find the attributes that will give us the most information and accuracy? For Coupious, the attribute set is quite limited. For this research, we explicitly decided the features that will contribute to our recommendations. In all cases, all attributes were under consideration while the algorithm was running and we never partitioned the attributes for different runs to create different recommendations.

2.1 THE kNN ALGORITHM

As shown in [7], “Once the clustering is complete, performance can be very good, since the size of the group that must be analyzed is much smaller.” The nearest neighbor algorithm is such a classification algorithm. K-nearest neighbors is one of the simplest machine learning algorithms, where an instance is classified by a majority vote by the closest training examples in the hypothesis space. This algorithm is an example of instance-based learning, where previous data points are stored and interpolations is performed to find a recommendation. An important distinction to make is that kNN does not try to create a model when it are given test data, but rather performs the computation when tested. The algorithm works by attempting to find a previously seen data point that is “closest” to the query data point and then uses its previous output for prediction. The algorithm calculates its approximation using this definition:

The algorithm is shown below

KNN (Examples, Target_Instance)

• Determine the parameter K = number of nearest neighbors.
• Calculate the distance between Target_Instance and all the Examples based on the Euclidian distance.
• Sort the distance and determine nearest neighbor based on the Kth minimum distance.
• Gather the Category Y of the nearest neighbors.
• Use simple majority of the category of nearest neighbors as the prediction value of the Target_Instance.
• Return classification of Target_Instance.

This algorithm lends itself well to the Coupious application. When a user uses the Coupious application, they want it to launch quickly and present a list of coupons within 10-15 seconds. Since this algorithm is so simple, we are able to calculate coupons that the user might enjoy fairly quickly and deliver them to the handset. Also, because Coupious doesn’t know any personal details about the user, the ability to cluster users into groups without the need for any heavy additional implementation on the front or back ends of the application is advantageous.

There are several possible problems with this approach. While this is a simple algorithm, it doesn’t take high feature dimensionality into account. If there are many features, the algorithm will have to perform a lot of computations to create the clusters. Additionally, each attribute is given the same degree of influence on the recommendation. In Coupious, the date and time the coupon was redeemed has the same amount of bearing on the recommendation as does the current distance from the offer and previous redemption history. The features may not scale compared to their importance and the performance of the recommendation may be degraded by irrelevant features. Lastly, [6] describes the curse of dimensionality, which describes how, if a product has 20 attributes, 2 of which are actually useful, how classification may be completely different when considering each of the 20 attributes but identical when only considering the 2 relevant attributes.

2.2 ATTRIBUTE SELECTION

While implementing the k-nearest neighbors algorithm, we decided to evaluate instances based upon seven different attributes, including coupon category, distance, expiration, offer, redemption date, handset and handset operating system, and upon the session result, a “hit” or “redemption.” The offer and expiration attributes required extra processing before they were able to be used for clustering. In both of these attributes, a “bag of words” (applying a Naive Bayes classifier to text to determine the classification) implementation was used to determine what type of offer the coupon had made.

For the OFFER attribute, we split the value space into 4 discrete values, PAYFORFREE (an instance where the customer had to pay some initial amount to receive a free item), PERCENTAGE (an instance where the customer received some percentage discount), DOLLAR (an instance where a customer received a dollar amount discount) and UNKNOWN (an instance where the classification was unknown). While the PERCENTAGE and DOLLAR values essentially compute to the exact same values when calculated, consumers tend to react differently when seeing higher percentage discounts versus dollar discounts (i.e. 50% off instead of $5 discount, even though they may equate to an identical discount, if the price were $10).

For the expiration attribute, some text parsing was done to determine what type of expiration the coupon had (DATE, USES, NONE or UNKNOWN). If the parsing detected a date format, we used DATE and if the parsing detected that it was a limited usage coupon (either limited by total uses across a population, herein known as “global” or limited by uses per customer, herein known as “local”), we designated that the coupon was USES. If the coupon was valid indefinitely, we designated that the coupon expiration was NONE. We did not record the distance of dates in the future, by the type of limited usage (global or local) or by the number of uses left.

An important detail to note is that in our implementation of kNN, if an attribute was unknown, we declined using it for computation of the nearest neighbor because the UNKNOWN attribute was typically the last value in a Java enumeration and therefore would be assigned a high integer value which would skew the results unnecessarily if used for computation.

For the remaining attributes (OFFERTIME, CATEGORY, DISTANCE, HANDSET, which considers handset model and OS and ACTION), the value space of each attribute was divided over the possible discrete values for that attribute according to Table 2.2.

Table 2.2

In the case of the handset OS, iPhone and iPod were classified together as the iPhone OS as they are the same OS with different build targets. It is important to note that kNN works equally well with continuous valued attributes as well as discrete-valued attributes. For further discussion on using kNN with real-valued attributes, see [10].

3. PROBLEMS

There were several problems that were encountered while implementing kNN. The first problem was “Majority Voting.” Majority voting is the last step in the algorithm to classify an instance and is an inherent problem with the way the kNN algorithm works. If a particular class dominates the training data, it will skew the votes towards that class since we are only considering data at a local level, that is the distance from our classification instance to the nearest data points. There are two ways to solve this problem: 1) balancing the dataset, or, 2) weighting the neighboring instances. Balancing is the simplest technique where an equal proportion of either class are present in the training data. A more complicated, but more effective, method is to weight the neighboring instances such that neighbors that are further away have a smaller weight value than closer neighbors.

Determining “k” beforehand is a troublesome problem. This is due to the fact that, if “k” is too large, the separation between classifications becomes blurred. This could cause the program to group two clusters together that are, in fact, distinct clusters themselves. However, a large “k” value does reduce the effect of noise by including more samples in the hypothesis. If “k” is too small, we might not calculate a good representation of the sample space because our results were too local. In either case, if the target attribute is binary, “k” should be an odd number to avoid the possibilities of ties with majority voting.

Also, kNN has a high computation cost because it has to consider the distance of every point from itself to another point. The time-complexity of kNN is O(n), which wouldn’t scale well to hypothesis spaces with millions of instances.

Discretizing non-related attributes, like category, presented a unique challenge for the author. When considering continuous attributes, such as distance, it was easy to discretize this data. In the case of Coupious, the distance ranges were already defined in the application so we just had to translate those over to our classification algorithm. However, some attributes, such as category, while related at an attribute level (in that they were all categories of coupon), had no real values. In this case, we simply assigned them increasing integer values

Another problem that we encountered is the sparsity problem. Since we implemented a content-driven model, the degree of accuracy relied upon how much data we had about a particular end user to build their profile. If the customer only had one or two sessions ending in no redemptions, it might not be possible to achieve any accuracy about this person. We dealt with this problem by artificially creating new records to supplement previous real records.

Coupious relies on the GPS module inside the smart phone to tell us where a user is currently located. From that position, the user gets a list of coupons that are close to the user’s location. However, there are no safeguards in place to guarantee that a redemption is real. A curious user may attempt to redeem a coupon when he is not at the actual merchant location. In the data, we can account for this by calculating the distance from the merchant at the time of the redemption request. However, this GPS reading may be inaccurate as the GPS model can adjust it’s accuracy to save power and battery life.

Lastly, accuracy is somewhat limited because of the fact that we are using a content-only model. There is no way to interact with the user to ask if the recommendations are truly useful. To achieve this additional metric would require major changes to the application and the backend systems that are outside the scope of this research paper.

Despite the multiple problems with kNN, it is quite robust to noisy data as indicated in section 4, which makes it well-suited for this classification task, as the author can only verify the reasonableness of the data, not the integrity.

4. TESTING

Testing of the kNN logic was carried out using a 3-fold cross validation method, were the data was divided into training and test sets such that | R | = k X | S |, where R is the size of the training set, k is the relative size and S is the size of the test set.

For each test run, we chose 4,000 training examples and 1 test example at random. We attempted to keep labeled classes in the training set as balanced as possible by setting a threshold n. This threshold prevented the classes from becoming unbalanced by more than a difference of n. If the threshold n was ever reached, we discarded random selections until we were under the threshold again and thereby balance the classes. Discussion of results are in section 5.

5. EVALUATION

Even though kNN is a simplistic algorithm, the classification results were quite accurate. To test the algorithm, we performed 10,000 independent runs where an equal number of “hit” and “redemption” rows were selected at random (2,000 of each so as to keep the inductive bias as fair as possible). An individual classification instance was chosen at random from that set of 4,000 instances and was then classified according to its nearest neighbors.

5.1 MACRO EVALUATION

When evaluating the results, there was one main factor that affected the recall and it was the size of the k neighbors considered in the calculation (see Table 5.1). In the 10,000 independent runs using random subsets of data for each test, the overall recall for the kNN algorithm was 85.32%. The average F-measure for these runs was 0.45.

However, if one considers a subset of results with k-size less than or equal to 15, the average recall was much higher – 94%. After a k-size greater than 30, we see a significant drop-off of recall. This can be attributed to the fact that the groups are becoming less defined because, as k-size grows, the nodes that are being used are “further” away from the classification instance, and therefore the results are not as “good.”

5.2 MICRO EVALUATION

When broken down into smaller sets of 1,000 runs, the recall and F-measure vary greatly. In 16 runs, we had a range from 29.40% recall to 98.20% recall. The median recall was 92.70%.

Table 5.1

There is a large gap between our maximum and our minimum recall. This can be attributed mainly to our k-size. Based on these results, we would feel fairly confident that we could make a useful prediction, but only if we had a confidence rating on the results (due to the high variability of the results).

These results could provide good insight into what k-size or neighbors are the most influential in suggesting a coupon. This would allow us to more carefully target Coupious advertising based on the user.

When could one consider these results valid? If the average distance of a classification instance to the examples is below a threshold such that the classification truly reflects its neighbors, we could consider the results valid. Another form of validating the results could be pruning the attribute set. If we were able to prune away attributes that didn’t affect the recall in a negative manner, we would be left with a set of attributes that truly influence the customers purchasing behavior, although this task is better suited for a decision or regression tree. An improvement to this kNN algorithm might come in the form of altering the k-size based upon the population or attributes that we are considering.

6. CONCLUSION

In this research project, we have implemented the kNN algorithm for recommender systems. The algorithm for the nearest neighbor was explored and several problems were identified and overcome. Different techniques were investigated to improve the accuracy of the system. The results of this project show an overall accuracy of 85.32%, which makes kNN an excellent, simple technique for implementing recommender systems.

REFERENCES

[1] Zhang, T., Iyengar, V. S. Recommender Systems Using Linear Classifiers. Journal of Machine Learning Research 2. (2002). 313-334.
[2] Basu, C., Hirsh, H., Cohen, W. Recommendation as Classification: Using Social and Content-Based Information in Recommendation.
[3] Cheung, K. W., Kowk, J. T., Law, M. H., Tsui, K. C. Mining Customer Product Ratings for Personalized Marketing
[4] E. Alpayden., Introduction to Machine Learning, 2nd ed. MIT Press. Cambridge, Mass, 2010.
[5] D. Goldberg, D. Nichols, B.M. Oki, D. Terry, Collaborative filtering to weave an information tapestry, Communications of the ACM 35. (12) (December 1992) 61 – 70.
[6] T.M. Mitchell, Machine Learning, New York. McGraw-Hill, 1997.
[7] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Recommender Systems for Large-Scale E-Commerce: Scalable Neighborhood Formation Using Clustering,” Proc. Fifth Int’l Conf. Computer and Information Technology, 2002.
[8] D. Barbella, S. Benzaid, J. Christensen, B. Jackson, X. V. Qin, D. Musicant. Understanding Support Vector Machine Classifications via a Recommender System-Like Approach. [Online]. http://www.cs.carleton.edu/faculty/dmusican/svmzen.pdf
[9] D.E. Knuth. “The Art of Computer Programming.” Addison-Wesley. 1973.
[10] C. Elkan. “Nearest Neighbor Classification.” University of California – San Diego. 2007. [Online]. http://cseweb.ucsd.edu/~elkan/151/nearestn.pdf

But why? Apple initially avoided an app model supporting multiple apps running at once to help preserve battery life and simplify the user experience.

So, Apple decided instead to opt for a “compromise”, push notifications to enable third party apps to appear to respond to outside updates (such as incoming messages or news alerts) even when they were not actually running. It would give the “illusion” of allowing 3rd party multitasking by displaying messages to the user when the app needed their attention. Other OSs, such as Android and Blackberry have full support for 3rd party multitasking, at the expense of battery life and perhaps a more complicated user experience.

So what is one to do? After writing the initial draft of this post, the news of iPhone OS 4.0 supporting multitasking came out a few days later. But, I don’t care about rumors – I want results. Just as in “Jerry Maguire”, show me the money. Oh, and for the sake of argument, let’s assume that our hardware is a single processor and our OS is using kernal threads and round-robin scheduling.

First, let’s consider the model for most modern OS and how they implement multitasking. Multitasking is actually a simple slight of hand. A process is simply an instance of a program or executable that is mapped into memory. We all know that we can have multiple programs or processes running at a time. But does the computer work with all the processes all at the same time? No. Similar to how a human works and processes information, a processor works with whatever data it is handed, but is only capable of working with particular process at a time. The processes are scheduled in a round-robin queue and when their turn comes up, they get a small piece of time with the processor. After their time, or quantum, expires, or the process blocks, the process is removed from the running context and placed back in the queue to wait for their turn again. This context switching happens so fast that a normal user cannot tell that this is the true model of what is happening and they believe that all programs are executing concurrently, as in the processor sharing model. The model described above has a whole array of issues relating to memory management (including page faulting), security and protection. The model I would like to describe below would have the same issues, but I will only talk about security and protection.

Armed with a very basic idea of how processes work, how can we adapt this to a more mobile OS? What are the areas that we can adjust.

Quantum – BAD
First, we can adjust the quantum, so there is less overhead associated with servicing each process and hence less power consumption. But Is this a good solution? Absolutely not. Quanta that are too short cause unnecessary thrashing with all the context switching that is occurring and if in an extreme case, the time servicing a context switch, may be longer than the actual time the process receives from the processor. What about variable quanta? Not a good idea either. The process would receive non-uniform amounts of time, making scheduling more difficult and perhaps a “jittery” response in the UI.

Scheduling – BAD
What if we change the type of scheduling we do? What if we try FCFS (first come, first served) or SJF (shortest job first)? That is not a good solution either because with FCFS, the process will continue to run and hog the processor until it is finished and in the case of applications, they will never finish until the user terminates the app. However, the solution proposed will draw inspiration from this category.

Notification – BAD
What if we eliminate the context switching and interrupts? We can set bits to indicate the app needs service. This idea is the absolute worst of them so far. What happens if two background applications request OS services in the same off round-robin cycle. If you proposed using shared memory for notification, you just lost an “interrupt.” What about not using shared memory? What a pain! Now you have to setup dynamic data structures for each app that runs and terminates. Ok, enough with the freshmen ideas. Are you ready to have a heap of knowledge dumped on you?

Let’s revisit the scheduling idea and draw inspiration from a multi-level feedback queue. In this proposed solution, there would be the same blocked and ready queues that any scheduler would have. However, there would be a foreground queue and a background queue. The foreground queue would contain the currently “focused” app, i.e. the process that the user is actively using. The background queue would contain all the remaining ready processes. The foreground queue would receive processor cycles as normal and the background queue would receive a fraction of the cycles to the tune of 1/n processes, with n capped at some upper bound to ensure that all processes still receive decent service. Giving the background processes some processor time ensure they can still continue to work. When the user wants to switch to a different process, the foreground process is removed and placed in the background queue. The rest of the process paradigm would stay the same, where a process could interrupt and request services from the OS

There are security implications of allowing multiprocessing on the iPhone, and the major one is security and protection. When the user presses the home button on the iPhone, the application receives a delegate message from the OS, applicationWillTerminate. From that point, the application has 5 whole seconds to exit, before the OS steps in and kills the process. With multiple processes, we now have to consider how multiple apps will behave together. Obviously the iPhone OS implements a virtual address space, so that separate processes have separate chunks of transparent memory. However, some additional security measures might need to be taken so that other processes are unable to access anything but their own VAs.

As much as I wish I could test this hypothesis, I can’t because I have no access to the iPhone firmware. So I will leave it up to the folks at 1 Infinite Loop, but this is just my .02.

Entropy.java – In Entropy.java, we are concerned with calculating the amount of entropy, or the amount of uncertainty or randomness with a particular variable. For example, consider a classifier with two classes, YES and NO. If a particular variable or attribute, say x has three training examples of class YES and three training examples of class NO (for a total of six), the entropy would be 1. This is because there is an equal number of both classes for this variable and is the most mixed up you can get. Likewise, if x had all six training examples of a particular class, say YES, then entropy would be 0 because this particular variable would be pure, thus making it a leaf node in our decision tree.

Entropy may be calculated in the following way:

import java.util.ArrayList;

public class Entropy {	
	public static double calculateEntropy(ArrayList<Record> data) {
		double entropy = 0;
		
		if(data.size() == 0) {
			// nothing to do
			return 0;
		}
		
		for(int i = 0; i < Hw1.setSize("PlayTennis"); i++) {
			int count = 0;
			for(int j = 0; j < data.size(); j++) {
				Record record = data.get(j);
				
				if(record.getAttributes().get(4).getValue() == i) {
					count++;
				}
			}
				
			double probability = count / (double)data.size();
			if(count > 0) {
				entropy += -probability * (Math.log(probability) / Math.log(2));
			}
		}
		
		return entropy;
	}
	
	public static double calculateGain(double rootEntropy, ArrayList<Double> subEntropies, ArrayList<Integer> setSizes, int data) {
		double gain = rootEntropy; 
		
		for(int i = 0; i < subEntropies.size(); i++) {
			gain += -((setSizes.get(i) / (double)data) * subEntropies.get(i));
		}
		
		return gain;
	}
}

Tree.java – This tree class contains all our code for building our decision tree. Note that each level, we choose the attribute that presents the best gain for that node. The gain is simply the expected reduction in the entropy of Xachieved by learning the state of the random variable A. Gain is also known as Kullback-Leibler divergence. Gain can be calculated in the following way:

Notice that gain is calculated as a function of all the values of the attribute.

import java.io.*;
import java.util.*;

public class Tree {
	public Node buildTree(ArrayList<Record> records, Node root, LearningSet learningSet) {
		int bestAttribute = -1;
		double bestGain = 0;
		root.setEntropy(Entropy.calculateEntropy(root.getData()));
		
		if(root.getEntropy() == 0) {
			return root;
		}
		
		for(int i = 0; i < Hw1.NUM_ATTRS - 2; i++) {
			if(!Hw1.isAttributeUsed(i)) {
				double entropy = 0;
				ArrayList<Double> entropies = new ArrayList<Double>();
				ArrayList<Integer> setSizes = new ArrayList<Integer>();
				
				for(int j = 0; j < Hw1.NUM_ATTRS - 2; j++) {
					ArrayList<Record> subset = subset(root, i, j);
					setSizes.add(subset.size());
					
					if(subset.size() != 0) {
						entropy = Entropy.calculateEntropy(subset);
						entropies.add(entropy);
					}
				}
				
				double gain = Entropy.calculateGain(root.getEntropy(), entropies, setSizes, root.getData().size());
				
				if(gain > bestGain) {
					bestAttribute = i;
					bestGain = gain;
				}
			}
		}
		if(bestAttribute != -1) {
			int setSize = Hw1.setSize(Hw1.attrMap.get(bestAttribute));
			root.setTestAttribute(new DiscreteAttribute(Hw1.attrMap.get(bestAttribute), 0));
			root.children = new Node[setSize];
			root.setUsed(true);
			Hw1.usedAttributes.add(bestAttribute);
			
			for (int j = 0; j< setSize; j++) {
				root.children[j] = new Node();
				root.children[j].setParent(root);
				root.children[j].setData(subset(root, bestAttribute, j));
				root.children[j].getTestAttribute().setName(Hw1.getLeafNames(bestAttribute, j));
				root.children[j].getTestAttribute().setValue(j);
			}

			for (int j = 0; j < setSize; j++) {
				buildTree(root.children[j].getData(), root.children[j], learningSet);
			}

			root.setData(null);
		}
		else {
			return root;
		}
		
		return root;
	}
	
	public ArrayList<Record> subset(Node root, int attr, int value) {
		ArrayList<Record> subset = new ArrayList<Record>();
		
		for(int i = 0; i < root.getData().size(); i++) {
			Record record = root.getData().get(i);
			
			if(record.getAttributes().get(attr).getValue() == value) {
				subset.add(record);
			}
		}
		return subset;
	}
	
	public double calculateSurrogates(ArrayList<Record> records) {
		return 0;
	}
}

DiscreteAttribute.java

public class DiscreteAttribute extends Attribute {
	public static final int Sunny = 0;
	public static final int Overcast = 1;
	public static final int Rain = 2;

	public static final int Hot = 0;
	public static final int Mild = 1;
	public static final int Cool = 2;
	
	public static final int High = 0;
	public static final int Normal = 1;
	
	public static final int Weak = 0;
	public static final int Strong = 1;
	
	public static final int PlayNo = 0;
	public static final int PlayYes = 1;
	
	enum PlayTennis {
		No,
		Yes
	}
	
	enum Wind {
		Weak,
		Strong
	}
	
	enum Humidity {
		High,
		Normal
	}
	
	enum Temp {
		Hot,
		Mild,
		Cool
	}
	
	enum Outlook {
		Sunny,
		Overcast,
		Rain
	}

	public DiscreteAttribute(String name, double value) {
		super(name, value);
	}

	public DiscreteAttribute(String name, String value) {
		super(name, value);
	}
}

Hw1.java – This class is our main driver class

import java.util.*;

public class Hw1 {
	public static int NUM_ATTRS = 6;
	public static ArrayList<String> attrMap;
	public static ArrayList<Integer> usedAttributes = new ArrayList<Integer>();

	public static void main(String[] args) {
		populateAttrMap();

		Tree t = new Tree();
		ArrayList<Record> records;
		LearningSet learningSet = new LearningSet();
		
		// read in all our data
		records = FileReader.buildRecords();
		
		Node root = new Node();
		
		for(Record record : records) {
			root.getData().add(record);
		}
		
		t.buildTree(records, root, learningSet);
		traverseTree(records.get(12), root);
		return;
	}
	
	public static void traverseTree(Record r, Node root) {
		while(root.children != null) {
			double nodeValue = 0;
			for(int i = 0; i < r.getAttributes().size(); i++) {
				if(r.getAttributes().get(i).getName().equalsIgnoreCase(root.getTestAttribute().getName())) {
					nodeValue = r.getAttributes().get(i).getValue();
					break;
				}
			}
			for(int i = 0; i < root.getChildren().length; i++) {
				if(nodeValue == root.children[i].getTestAttribute().getValue()) {
					traverseTree(r, root.children[i]);
				}
			}
		}
		
		System.out.print("Prediction for Play Tennis: ");
		if(root.getTestAttribute().getValue() == 0) {
			System.out.println("No");
		}
		else if(root.getTestAttribute().getValue() == 0) {
			System.out.println("Yes");
		}

		return;
	}
	
	public static boolean isAttributeUsed(int attribute) {
		if(usedAttributes.contains(attribute)) {
			return true;
		}
		else {
			return false;
		}
	}
	
	public static int setSize(String set) {
		if(set.equalsIgnoreCase("Outlook")) {
			return 3;
		}
		else if(set.equalsIgnoreCase("Wind")) {
			return 2;
		}
		else if(set.equalsIgnoreCase("Temperature")) {
			return 3;
		}
		else if(set.equalsIgnoreCase("Humidity")) {
			return 2;
		}
		else if(set.equalsIgnoreCase("PlayTennis")) {
			return 2;
		}
		return 0;
	}
	
	public static String getLeafNames(int attributeNum, int valueNum) {
		if(attributeNum == 0) {
			if(valueNum == 0) {
				return "Sunny";
			}
			else if(valueNum == 1) {
				return "Overcast";
			}
			else if(valueNum == 2) {
				return "Rain";
			}
		}
		else if(attributeNum == 1) {
			if(valueNum == 0) {
				return "Hot";
			}
			else if(valueNum == 1) {
				return "Mild";
			}
			else if(valueNum == 2) {
				return "Cool";
			}
		}
		else if(attributeNum == 2) {
			if(valueNum == 0) {
				return "High";
			}
			else if(valueNum == 1) {
				return "Normal";
			}
		}
		else if(attributeNum == 3) {
			if(valueNum == 0) {
				return "Weak";
			}
			else if(valueNum == 1) {
				return "Strong";
			}
		}
		
		return null;
	}
	
	public static void populateAttrMap() {
		attrMap = new ArrayList<String>();
		attrMap.add("Outlook");
		attrMap.add("Temperature");
		attrMap.add("Humidity");
		attrMap.add("Wind");
		attrMap.add("PlayTennis");
	}
}

Node.java – Node.java holds our information in the tree.

import java.util.*;

public class Node {
	private Node parent;
	public Node[] children;
	private ArrayList<Record> data;
	private double entropy;
	private boolean isUsed;
	private DiscreteAttribute testAttribute;

	public Node() {
		this.data = new ArrayList<Record>();
		setEntropy(0.0);
		setParent(null);
		setChildren(null);
		setUsed(false);
		setTestAttribute(new DiscreteAttribute("", 0));
	}

	public void setParent(Node parent) {
		this.parent = parent;
	}

	public Node getParent() {
		return parent;
	}

	public void setData(ArrayList<Record> data) {
		this.data = data;
	}

	public ArrayList<Record> getData() {
		return data;
	}

	public void setEntropy(double entropy) {
		this.entropy = entropy;
	}

	public double getEntropy() {
		return entropy;
	}

	public void setChildren(Node[] children) {
		this.children = children;
	}

	public Node[] getChildren() {
		return children;
	}

	public void setUsed(boolean isUsed) {
		this.isUsed = isUsed;
	}

	public boolean isUsed() {
		return isUsed;
	}

	public void setTestAttribute(DiscreteAttribute testAttribute) {
		this.testAttribute = testAttribute;
	}

	public DiscreteAttribute getTestAttribute() {
		return testAttribute;
	}
}

FileReader.java – The least interesting class in the code

import java.io.*;
import java.util.ArrayList;
import java.util.StringTokenizer;

public class FileReader {
	public static final String PATH_TO_DATA_FILE = "playtennis.data";

    public static ArrayList<Record> buildRecords() {
		BufferedReader reader = null;
		DataInputStream dis = null;
		ArrayList<Record> records = new ArrayList<Record>();

        try { 
           File f = new File(PATH_TO_DATA_FILE);
           FileInputStream fis = new FileInputStream(f); 
           reader = new BufferedReader(new InputStreamReader(fis));;
           
           // read the first record of the file
           String line;
           Record r = null;
           ArrayList<DiscreteAttribute> attributes;
           while ((line = reader.readLine()) != null) {
              StringTokenizer st = new StringTokenizer(line, ",");
              attributes = new ArrayList<DiscreteAttribute>();
              r = new Record();
              
              if(Hw1.NUM_ATTRS != st.countTokens()) {
            	  throw new Exception("Unknown number of attributes!");
              }
              	
			  @SuppressWarnings("unused")
			  String day = st.nextToken();
			  String outlook = st.nextToken();
			  String temperature = st.nextToken();
			  String humidity = st.nextToken();
			  String wind = st.nextToken();
			  String playTennis = st.nextToken();
			  
			  if(outlook.equalsIgnoreCase("overcast")) {
				  attributes.add(new DiscreteAttribute("Outlook", DiscreteAttribute.Overcast));
			  }
			  else if(outlook.equalsIgnoreCase("sunny")) {
				  attributes.add(new DiscreteAttribute("Outlook", DiscreteAttribute.Sunny));
			  }
			  else if(outlook.equalsIgnoreCase("rain")) {
				  attributes.add(new DiscreteAttribute("Outlook", DiscreteAttribute.Rain));
			  }
			  
			  if(temperature.equalsIgnoreCase("hot")) {
				  attributes.add(new DiscreteAttribute("Temperature", DiscreteAttribute.Hot));
			  }
			  else if(temperature.equalsIgnoreCase("mild")) {
				  attributes.add(new DiscreteAttribute("Temperature", DiscreteAttribute.Mild));
			  }
			  else if(temperature.equalsIgnoreCase("cool")) {
				  attributes.add(new DiscreteAttribute("Temperature", DiscreteAttribute.Cool));
			  }
			  
			  if(humidity.equalsIgnoreCase("high")) {
				  attributes.add(new DiscreteAttribute("Humidity", DiscreteAttribute.High));
			  }
			  else if(humidity.equalsIgnoreCase("normal")) {
				  attributes.add(new DiscreteAttribute("Humidity", DiscreteAttribute.Normal));
			  }
			  
			  if(wind.equalsIgnoreCase("weak")) {
				  attributes.add(new DiscreteAttribute("Wind", DiscreteAttribute.Weak));

			  }
			  else if(wind.equalsIgnoreCase("strong")) {
				  attributes.add(new DiscreteAttribute("Wind", DiscreteAttribute.Strong));

			  }
			  
			  if(playTennis.equalsIgnoreCase("no")) {
				  attributes.add(new DiscreteAttribute("PlayTennis", DiscreteAttribute.PlayNo));
			  }
			  else if(playTennis.equalsIgnoreCase("yes")) {
				  attributes.add(new DiscreteAttribute("PlayTennis", DiscreteAttribute.PlayYes));
			  }
			    		    
			  r.setAttributes(attributes);
			  records.add(r);
           }

        } 
        catch (IOException e) { 
           System.out.println("Uh oh, got an IOException error: " + e.getMessage()); 
        } 
        catch (Exception e) {
            System.out.println("Uh oh, got an Exception error: " + e.getMessage()); 
        }
        finally { 
           if (dis != null) {
              try {
                 dis.close();
              } catch (IOException ioe) {
                 System.out.println("IOException error trying to close the file: " + ioe.getMessage()); 
              }
           }
        }
		return records;
	}
}

**EDIT:**

playtennis.data is only a simple text file that describes the learned attribute “play tennis” for a particular learning episode. I modeled my playtennis.data file off of Tom Mitchell’s play tennis example in his book “Machine Learning.” Essentially what it contains is attributes describing each learning episode: outlook (sunny, overcast, rain), wind (strong, weak) and humidity (high, normal) and play tennis (yes, no). Based off this information, one can trace the decision tree to derive your learned attribute. A sample decision tree is below. All you have to do is create a text file (tab delimited) that describes each particular situation (make sure you match the columns in the text file to the parsing that occurs in the Java).

**EDIT 2:**

After so many requests, I have worked up a small playtennis.data file based on the Tom Mitchell “Machine Learning” book. This follows his example data exactly and can be found here (rename the file to “playtennis.data” before running as WordPress wouldn’t let me upload a .data file due to “security restrictions.” A small note of caution: the above code was put together very quickly for purposes of my own learning. I am not claiming that it is by any means complete or fault tolerant, however, I do believe that the entropy and gain calculation is sound and correct.

<div id="parent">
  <div id="child">Child node content
    <script>
      document.getElementById('parent').innerHTML = 'New child';
    </script>
   </div>
</div>

The content will still be replaced but with a warning in the console. To get rid of the warning just move the script out to the parent container:

<div id="parent">
  <div id="child">Same child node content</div>
  <script
    document.getElementById('parent').innerHTML = 'New child';
  </script
</div>

The problem I was having was related to writing out the markup associated with FancyZoom div content. The same effect can be acheived by using body.onload or jQuery(document).ready, although I did notice that if you use onload() or .ready, if the page is not cached locally, you might have a FOUC (flash of unformatted content).

If you want more information about this problem head over to Microsoft’s support site.

1.  INTRODUCTION

A myocardial infarction, commonly referred to as a heart attack, is a serious medical condition where blood vessels that supply blood to the heart are blocked, preventing enough oxygen from getting to the heart. The heart muscle dies from the lack of oxygen and impairs heart function or kills the patient. Heart attacks are positively correlated with diabetes, smoking, high blood pressure, obesity, age and alcohol consumption. While prognosis varies greatly based on underlying personal health, the extent of damage and the treatment given, for the period of 2005-2008 in the United States, the median mortality rate at 30 days was 16.6% with a range from 10.9% to 24.9% depending on the admitting hospital.[9] That rate improves to 96% at the 1 year mark if the patient survives the heart attack.

Physicians would like to be able to tell their patients their possible rate of survival and predict whether or not a certain treatment could help the patient. In order to make that prediction, we can use decision trees to model whether or not the patient has a good chance of survival. Using regression trees, we can map the input space into a real-valued domain using attributes that cardiologists could examine to determine the patient’s chances of survival after a given timeframe.

In building a decision tree, we use the most influential attribute values to represent the internal nodes of the tree at each level.  Each internal node tests an attribute value, each edge corresponds to an attribute value and each leaf node leads to a classification – in our case, deceased or alive.  We are able to traverse the tree from the root to classify an unseen example.  The tree can also be expressed in the form of simple rules.  This would be helpful when explaining the prognosis to the patient.

1.1  OUTLINE OF RESEARCH

In this research survey, we implemented a decision tree using an adapted ID3 algorithm.  We evaluated different splitting criterion as well as different approaches to handling missing attributes in the dataset.  In addition, the author considers different approaches to handling continuous valued attributes and methods to reduce decision tree over-fitting.  Lastly, results are discussed in section 4 after running the decision tree multiple times with the echocardiogram dataset.

1.2  DATA

The data that we used is from the University of California at Irvine machine learning repository.  The dataset that we chose was the 1989 echocardiogram dataset.  This dataset contained 132 rows, with 12 attributes, 8 of which were actually usable for decision tree construction (the remaining four were references for the original contributor of this dataset).  This dataset had missing values and all of the attributes were continuous valued.

The data described different measurements of patients who had suffered from acute myocardial infarction at some point in the past.  The attributes included “AGE-AT-HEART-ATTACK” (the patients’ age in years when the heart attack happened), “PERICARDIAL-EFFUSION” (binary choice relative to fluid around the heart), “FRACTIONAL-SHORTENING” (a measure of contractility around the heart where lower numbers are abnormal), “EPSS” (E-point septal separation which is another measure of contractility where larger numbers are abnormal), “LVDD” (left ventricular end-diastolic dimension, where larger numbers are abnormal), “WALL-MOTION-INDEX” (a measure of how many segments of the left ventricle are seen moving divided by the number of segments seen) and “ALIVE-AT-ONE” (a binary choice where 0 represents deceased or unknown and 1 is alive at one year).

It is important to note that not all rows could be used for learning.  Two attributes, “SURVIVAL” and “STILL-ALIVE” must be analyzed as a set.  SURVIVAL described the number of months the patient had survived since the heart attack.  Some of the rows described patients who survived less than a year and are still alive based on STILL-ALIVE (a binary attribute, 0 representing deceased and 1 representing alive).  These patients cannot be used for prediction.

It has previously been noted that “the problem addressed by past researchers was to predict from other variables whether or not the patient will survive at least one year.  The most difficult part of this problem is correctly predicting that the patient will not survive.  This difficulty seems to stem from the size of the dataset.” [1]  In implementing the decision tree logic, we have found that this is the case as well.

2.  APPLICATION

We chose to write the implementation of the decision tree in Java because of the ease of use of the language.  Java also presented superior capabilities in working with and parsing data from files.  Using Java allowed the author to more efficiently model the problem through the use of OO concepts, such as polymorphism and inheritance.

2.1  DECISION TREE CONSTRUCTION

The decision tree is constructed using the ID3 algorithm originally described by Quinlan [4] and shown in Mitchell [2] with an adaptation by the author to handle numeric, continuous valued attributes, missing attributes and pruning.  ID3 is a simple decision tree algorithm.  The algorithm is shown below

ID3 (Examples, Target_Attribute, Attributes)

• Create a root node for the tree

• If all Examples are positive, return the single-node tree root, with label = +.

• If all Examples are negative, return the single-node tree root, with label = -.

• If the number of predicting Attributes is empty, then return the single node tree root, with label = most common
  value of target attribute in the examples.

• Otherwise Begin

  • A = The Attribute that best classifies Examples.

  • Decision tree attribute for root = A.

  • For each possible value, vi, of A

    • Add a new tree branch below root, corresponding to the test A = vi.

    • Let Examples(vi) be the subset of examples that have the value vi for A

    • If Examples(vi) is empty

      • Below this new branch, add a leaf node with the label = most common target value in the examples.

    • Else

      • Below this new branch add the subtree ID3 (Examples(vi), Target_Attribute, Attributes - {A})

• End

• Return root

The ID3 algorithm “uses information gain as splitting criteria and growing stops when all instances belong to a single value of target feature or when best information gain is not greater than zero.” [6]  For further information on ID3, see [2], [4], [6], [7].

2.2  SPLITTING CRITERION

A decision tree is formed by having some concrete concept of splitting data into subsets which form children nodes of the parent.  In our implementation of the decision tree, we decided to use a univariate impurity-based splitting criterion called entropy.

Entropy is the measure of impurity or chaos in the data.  If all elements in a set of data belong to the same class, the entropy would be zero and if all the elements in a dataset were evenly mixed, the entropy would be one.  Entropy may be measured with the following equation

where is the number of positive training examples in T and is the number of negative training examples in T.  For further discussion on entropy, see Mitchell [2] or Alpaydin [3].

The ID3 algorithm can use this measure of entropy of each attribute set of values to determine the best gain, or expected reduction in entropy due to splitting on A or the difference in entropies before splitting and after splitting on A.

At each level of the tree, the “best” attributes can be found to create the maximum reduction in entropy, called information gain.  These two calculations represent the preference to create the smallest tree possible for several different reasons – mainly that a short hypothesis that accurately describes the data is unlikely to be coincidence.  For further discussion on entropy and gain, see Mitchell [2], Alpaydin [3], Steinberg [5] and Lior et al [6].

2.3  MISSING VALUES

In the dataset, there are many missing values.  The missing values show up mainly for the attributes ALIVE-AT-ONE (43.2%), EPSS (10.6%) and LVDD (7.5% of the data).  In addition, 66.3% of the data had a row with at least 1 missing attribute value.

Clearly, this dataset is not ideal for predicting attributes.  Fortunately, decision trees are robust to noisy data.  The strategy of replacing the missing attribute values with the most common value among training examples was suggested in [2].  We decided to implement this idea to deal with these missing values and initiate those values with a surrogate value.  The surrogate values were calculated using the average of all of the attributes for continuous-valued data or the most common attribute value for discrete valued data.  Our replacement was done only after all the data had been read in from the dataset instead of using a moving average while the data was still being read.

There are two main reasons we chose this method.  First, this is an extremely simple method that doesn’t require a lot of calculation.  Since decision trees cannot back up once they have made a splitting decision, we would never have to worry about the data changing without needing to regenerate the tree.  Second, this allowed the program to describe a finer grain representation of the true average value of the particular attribute for the global dataset.

2.4  HANDLING CONTINUOUS AND DISCRETE ATTRIBUTES

In ID3, handling discrete attributes are quite simple.  For each attribute, a child node is created and that branch is taken when the tree is traversed to evaluate a test set.  To handle continuous data, some partitioning of the attribute values must take place to discretize the possible range.  For example, if we had the attributes shown in the following table:

We may consider creating a discrete variable that corresponds to Temperature > 40 and Temperature < (48 + 60) / 2 that would produce a true or false value.

For our implementation, we decided to take the average of all of the values and make that our discrete split value, where all instances less than the average branch went to the left node and all the instances greater than the discrete value branch went to the right node.  The reason that we chose this approach is because it was a simple implementation over other methods that included binary search, statistical search and weighted searches. [2] [6]  This causes our tree to look closer in form to a CART tree rather than an ID3 tree, as CART can only produce trees that are binary trees.  CART uses this approach of a single partitioning value when using univariate splitting criteria. [5]  The main reason that we are able to implement this approach is the data is such that it is increasingly “abnormal” the larger the value becomes.

2.5  TESTING

Testing of the decision tree logic was carried out using the PlayTennis example shown in [2].  This example used only discrete values.  After verifying the decision tree could accurately classify these examples, the program was adapted to use continuous valued attributes.

Testing performed was done with a 3-fold cross validation method, were the data was divided into training and test sets such that | R | = k X | S |, where R is the size of the training set, k is the relative size and S is the size of the test set.

For each test run, we chose 88 training examples and 44 test examples at random. We attempted to keep labeled classes in the training set as balanced as possible by setting a threshold n. This threshold prevented the classes from becoming unbalanced by no more than a difference of n. If the threshold n was ever reached, we discarded random selections until we were under the threshold again in order to balance the classes.  Discussion of results are in section 4.

2.6  DECISION TREE PRUNING

In the process of building the decision tree, the accuracy of the tree is determined by the training examples.  “However, measured over an independent set of training examples, the accuracy first increases and then decreases.” [2]  This is an instance of over-fitting the data.  To prevent this over-fitting condition, the tree is evaluated and then cut back to the “essential” nodes such that the accuracy does not decrease with real-world training examples.

In our implementation of pruning, we had no stopping criteria to prevent over-fitting.  Our implementation let the over-fitting occur and then we used a post-pruning method called Reduced-Error pruning, as described by Quinlan. [7]  In this algorithm, the tree nodes are traversed from bottom to top while the procedure checking to see if replacing it with the most frequent class improves the accuracy of the decision tree.  Pruning continues until accuracy decreases and the algorithm ends with the smallest, accurate decision tree.

Further discussion of pruning may be found in [2], [3], [5] and [6].

3.  POTENTIAL PROBLEMS

There are some problems that we encountered during implementation.  The first problem is that ID3 does not natively support missing attributes, numeric attributes or pruning.  The algorithm had to be adapted to support these features.  In adapting the algorithm, less than efficient methods were chosen.  In my implementation, we split at the mean of the attribute values.  This would cause the surrogate values and the split points to be the same value.  In addition, this method does not handle outliers in the data well and forces everything towards the center of the sample set.  Implementing a better search algorithm or multivariate splitting might allow the author to see improved accuracy.  Another alternative would be to use the C4.5 algorithm [8], which is an evolution of ID3 adds full support for these requirements.

Another problem that we experienced is having enough training data without missing attributes to build an effective decision tree.  With the large missing attribute rate, it would be hard to get a good handle on any sort of trends in the dataset.  A possible solution may be to use a weighted method to assign the most probable value at the point where we encounter the missing value.

4.  EVALUATION

For the evaluation of the results, we have chosen not to assign value to the results so that a result of ALIVE-AT-ONE equal to 1 is positive and a result of ALIVE-AT-ONE equal to 1 as well instead of negative.  One may choose to assign intrinsic values to the instances, but one could easily evaluate these results without them.  We have chosen to only focus on the results rather than make a determination of the “goodness” of a particular outcome.  Since we have chosen this method of evaluation, we would not have any false positives or true negative results, therefore we will not be reporting any precision calculations.

4.2  MACRO EVALUATION

In 10,000 independent runs using random subsets of data for each test, the overall recall for the decision tree was 66.82%.  The average F-measure for these runs was 0.38.  See figure 4.1.  In the majority of the runs, the most influential attributes where SURVIVAL, EPSS and LVDD, where each of these attributes appeared in 100% of the decision trees created.

4.1 MIRCO EVALUATION

When broken down into smaller sets of 1,000 runs, the recall and F-measure vary greatly.  In 10 runs, we had a range from 27.27% recall to 93.18% recall.  The median recall was 84.09%.

Figure 4.1

There is a large gap between our maximum and our minimum recall.  This can be attributed to several issues, including poor data in the data set and less than efficient splitting choices.  The accuracy of the decision tree depends partly on the data with which you decide to train.  The data that we used was missing many attributes, with almost half (43.2%) of those attribute values being the target attribute.  In the absence of a value, the most common value was substituted which, in the case of this data set, would heavily skew the results towards predicting death. It should be noted that overly pessimistic results and overly optimistic results can each present their own dangers to the patient.

Another improvement to the results may come in the form of changing the policy on attribute splitting and missing value handling.  If a more accurate method of splitting were implemented (multivariate criterion, using a better search method, etc.), we would expect to see a more consistent result set.

Based on these results, we would feel fairly confident that we could make a useful prediction, but only if we had a confidence rating on the results (due to the high variability of the results).

Based on these results, while not highly accurate, they could provide good insight into what attributes are the most important in regards to classification.  In other words, the decision tree would describe the attributes that should be used for other machine learning programs, such as clustering, artificial neural networks or support vector machines.  We could be confident that these attributes are the most important because they were chosen through entropy and gain calculations while constructing the decision tree.

5.  CONCLUSION

In this research project, different methods of constructing decision and regression trees were explored.  Additionally, different methods of node splitting, missing attribute substitution and tree pruning were investigated.  While the results of this project show only a 66% accuracy, decision trees are still a valid machine learning technique.  With an augmented decision logic and a better data set, decision trees may be able to predict discrete or continuous data at a much better rate.

References

1. Salzberg, Stephen.  University of California, Irvine Machine Learning Data Repository.  1989.  [Online]. http://archive.ics.uci.edu/ml/datasets/Echocardiogram
2. Mitchell, Tom M.  Machine Learning WCB McGraw-Hill, Boston, MA.  1997
3. Alpaydin, Ethem.  Introduction to Machine Learning, Second Edition.  The MIT Press, Cambridge, MA.  2010.
4. Quinlan, J. R. Induction of Decision Trees. Mach. Learn. 1, 1 (Mar. 1986), 81-106.
5. Steinberg, Dan.  CART: Classification and Regression Trees. Taylor and Francis Group. pp 179-201, 2009.
6. Rokach, Lior and Maimon, Oded. Top-Down Induction of Decision Tree Classifieres – A Survey.  IEEE Transactions on Systems, Man and Cybernetics – Part C: Applications and Reviews.  Vol. 35, No. 4.  pp 476-487, November 2005.
7. Quinlan, J.R.  Simplifying Decision Trees. International Journal of Man-Machine Studies.  Vol. 27, pp. 221-234, 1987.
8. Quinlan, J.R.  C4.5: Programs for Machine Learning.  San Francisco, CA: Morgan Kaufmann, 1993.
9. Krumholz H et al. Patterns of hospital performance in acute myocardial infarction and heart failure – 30-day mortality and readmission. Circulation: Cardiovascular Quality and Outcomes. [Online] http://circoutcomes.ahajournals.org/cgi/content/abstract/2/5/407. 2009.

1. Enter mobile.united.com into the browser of your mobile device to check in for any flight within the U.S. starting 24 hours before scheduled departure.
2. If you are flying through one of the airports listed below, you can select to receive your boarding pass via email. Otherwise, visit any EasyCheck-in® kiosk to print your boarding pass at the airport.
3. To use your device as your boarding pass, scan the barcode on your screen at security and at the gate when boarding.

You’ll also be able to check flight status and get up-to-the-minute arrival and departure information, including times and gate information at the following airports (with United):

• Chicago O’Hare (ORD)
• Dallas – Fort Worth (DFW)
• Denver (DEN)
• Las Vegas (LAS)
• Los Angeles (LAX)
• New York LaGuardia (LGA)
• San Francisco (SFO)
• Washington Dulles (IAD)

Why This Is Awesome?

• It is so easy! Being able to check-in to your flight on the way to the airport and walking past the ticket counters without ever needing to print a boarding pass will quickly become my favorite part of flying.  The whole idea of printing your boarding pass at home for your “convenience” is really not a “convenience” for most customers.  It is a cost saving measure by the airlines.  There is less interaction required for the customers and less paper and ink tickets going out that the airline has to pay for.
• This might mean reduced ticket prices? But probably not, since the airline industry barely scrapes by with a profit anyway.  Although, you would wonder if this might reduce the number of people the airline industry employs since there would be less of a need to interact with customers.

Why Will This Flop?

• What happened to security? Is this really secure?  I have a really hard time believing that the TSA is going to let this fly (yes, I know, a terrible pun).  The first time that a TSA agent sees this, they are probably going to reject this person outright.  Depending on what the boarding pass looked like, how hard would to fake something that looked like this.  Not very, if you were technically skilled.  The TSA agents would need a scanner that would interface with the United ticket database to verify that these are actual, genuine tickets.

What I Would Like To See

• Dedicated apps.  If you’re a business, having your own app for smart phones is de rigueur.  This is would create value among customers because they would love the convenience of being able to walk straight into security with their carryon and get right to their gate.  You could allow for a much richer customer experience, such as choosing your seats by using not only a seat map, but an actual photo of that seat to let you see if you would like the window placement, leg room, TV location, etc.  It might benefit the airlines as well because customers could move to standby for other flights, upgrade to first class, check on missing baggage – all without ever needing to call or speak to anyone.

There are other airlines such as Continental and Delta (planned) migrating to online boarding passes.  Everything considered, this is excellent use of smart phones and rather overdue.  This is an classic example of writing apps to create value rather than writing apps to drive revenue.

Skills that are being taught are softening.  Good old fashioned coding and algorithm design is giving way to using libraries of code that essentially do all the work for the programmer.  But wait – isn’t code reuse a fundamental of good, quality programming.  Yes, it is!  However, this should not be the starting point for students to learn.  When the libraries are used, the logic behind the code is lost.  Good design principles dictate that when using an API, we shouldn’t care what’s inside the “black box” but rather just accept that if we feed the API good and proper data, it will perform as expected and we’ll get values out but students who are learning should care because they aren’t just learning to program, they are learning to think.  Another example from Java: what is the complexity of .contains()?  While this should be obvious to most engineers that it is O(n), universities that teach that it is acceptable to use APIs and libraries all the time are doing a major disservice to their students.

OO languages are becoming the de facto first course.  This is a backwards approach because OO allows for advanced design with encapsulation, inheritance and polymorphism.  We have to get back to the basics by learning to code as a function of learning computer science where one can appreciate architecture designs, compilers, memory management, language theory and the relationship between hardware and software.  Knowing a procedueral programming language is an absolute necessity and the  Linux Kernel, Git and  Apache Server are all products of procedural languages.  Not bad company at all.

My first programming language that I learned at Purdue was C.  This was not the common path for most CS majors – the majority of people started with Java.  Learning C first created a foundation that was later built upon by taking the required Java course.  As a friend recently put it – it is like learning to drive a car or learning how to engineer a car.  Anyone can drive a car and some people love it so much, they become cab drivers.  That is why they have “X Technology for Dummies” books.  But, if you want to dive into the internals of the car, look at how the engine works for example, you need the solid education of understanding the principles behind the application.

Being a “good developer” does not mean that you are practicing computer science, because in reality, coding is only a small fraction of what computer science is about.  Programming languages are simply a means to an end.