Augmented Reality - D200

From New Media Business Blog

Jump to: navigation, search

This Wiki will discuss augmented reality and the ways it is currently being applied in our lives through work, shopping, travelling, entertainment, and so forth. The Wiki will begin by talking about what augmented reality is, how it is different from virtual reality, and the four main types that are currently being utilized across the world. After, we will dive deeper into some common applications of augmented reality through retail, tourism, hospitality, education, training, navigation, surgery, gaming, and military. Here, we will provide some basic information on the utilization of augmented reality in this field along with some common examples, future directions, benefits and risks, and our own thoughts. To finish, we will summarize our analysis by talking about how all of these applications relate to one cohesive whole.


What is Augmented Reality?

Augmented Reality

Augmented Reality is any technology that superimposes digital images and data over physical objects within someone's view of the real world. This improves how the individual can interact with their environment, visualize information, and efficiently follow business processes. Companies that utilize augmented reality often see productivity and quality gains which can be seen in industries such as manufacturing, logistics, product development, service, marketing, and training [1].

The origin of the word "augmented" is augment which means to make something greater or enhance something by adding to it. In the case of augmented reality, The technology enhances a user's real-world experience by adding touch feedback, graphics, and sounds to their perspective. Overall, Augmented reality has seen a huge growth in popularity in recent years as it has been applied to many aspects of people's lives as will be discussed later [2].

How is Augmented Reality Different From Virtual Reality?

There is a common misconception that augmented reality and virtual reality are the same thing; however, while similar, there are many differences between the two. Augmented reality, as mentioned, uses technology to overlay digital data or images over objects in the real world, whereas virtual reality uses technology to create a completely simulated or artificial environment that is completely separate from the real world. A simple way to think about it would be that augmented reality enhances the world, while virtual reality reconstructs it [3].

What are the Types of Augmented Reality?

There are four primary types of augmented reality technologies which each have their benefits, drawbacks, and applications in business. The full list can be found below, and each will be discussed in-depth in the later sections [4]

  • Marker Based AR
  • Markerless AR
  • Projection Based AR
  • Superimposition Based AR

Marker Based AR

Marker Based AR

Marker based augmented reality requires that a marker be used to identify where the AR data or image will be overlayed. This means that the particular marker and its details are hard programmed into the application which allows the device using the application (e.g. IPhone) to sense it with its camera and distinguish it from other real-world objects. The most common type of marker is a QR code which is often put over objects such as tires, walls, tables, and books; however, other symbols may be used as long as they have simple, recognizable patterns [5].

There are many benefits to using marker based AR over other types of augmented reality. First, it does not require cameras on personal electronic devices to have a gyroscope. While many devices do have gyroscopes, many older IPhone models (mostly from before 2010) and a variety of Windows and Android devices do not. Thus, all devices can bypass this limitation and use marker based AR applications with ease [6]. Second, these markers help to determine more accurate placement, positioning, and orientation of the overlayed data and images which can make a person's field of view seem more realistic [7]. Lastly, marker based AR applications require less processing power and development effort to create which means that time and resources may be saved [8].

Markerless AR

Markerless AR

Markerless augmented reality does not require that a marker be identified, and instead only requires a fairly flat surface for the AR technology to overlay data and images. The application will recognize things that are not directly programmed into the application such as colours, patterns, distance, dimensions, and other features. This means that an algorithm will be used to identify whether an object may be a table, hot dog, car, or more [9].

While markerless AR is much more costly and difficult to implement, it provides a lot of great opportunities in many different industries such as navigation, education, gaming, healthcare, and retail. For example, if a consumer wants to browse a store's catalogue, they would have to scan a marker in a physical catalogue to see each product with marker based AR; however, with markerless AR, they would be able to surf the catalogue completely with overlayed digital images without the need for physical markers [10]. Furthermore, markerless AR often uses a velocity meter, accelerometer, digital compass, and GPS which helps the device to provide location-based data which can be useful for navigation applications or displaying information on points of interest nearby [11].

Projection Based AR

Projection Based AR (Le Petit Chef)

Projection based augmented reality projects artificial light onto the physical world which users may or may not interact with depending on the application. The technology is particularly different from marker based and markerless AR because it does not require a personal electronic device (e.g. smartphone, Ipad) or wearable technology to see the overlayed data or images. This means that multiple users can share the same AR experience which allows for more seamless integration of AR into the real world. However, there are some disadvantages such as the technology often having a fixed placement in a certain location. Also, the technology behind this is very complex as it often uses light cameras, 3D sensing systems, and depth cameras all to get the perfect projection mapping [12].

While projection based AR currently has less application in business, there are still some ways it is currently being utilized. For example, one of its earliest uses was with the Xbox Kinect which tracked movements to act as interactions within video games. This was a state of the art technology at the time which created a form of gaming that was not previously seen [13]. Another way projection based AR has been implemented is through its ability to simplify complex processes and improve training procedures. This will be discussed in a later section; however, the basic idea is that it uses audio and visual prompts to guide the users through the operational processes. This combination of human intelligence, physical equipment, and technology increases the users' speed, precision, and productivity [14]. Lastly, through visual mapping using a Panasonic projector, a restaurant has implemented "Le Petit Chef" which is animation of a tiny chef cooking customers' dinners, which is projected onto their dinner plates [15].

Superimposition Based AR

Superimposition Based AR

Superimposition based augmented reality either partially or fully replaces the view of real-world objects with that of a similar overlayed AR objects. While very similar to markerless AR, the main difference between the two is that markerless adds a completely new object to the environment while superimposition replaces an existing object in the environment. This means that object recognition is extremely important for superimposition AR since the object cannot be replaced if it is not recognized [16].

Currently, superimposition based AR is a very consumer-facing technology with its applications in retail. For example, by using the Ikea virtual catalogue, users can replace their view of a couch in their living room with that of an Ikea couch to compare dimensions, looks, and appeal [17]. Moreover, consumers can superimpose clothing onto their bodies using different clothing applications, which will be discussed in a later section.

Common Application of Augmented Reality

There are seven primary industries and areas that are currently utilizing augmented reality technologies for a variety of reasons. The full list can be found below, and each will be discussed in-depth in the later sections.

  • Retail
  • Tourism & Hospitality
  • Education & Training
  • Navigation
  • Surgery
  • Gaming
  • Military

Augmented Reality in Retail

For the past few years, we are seeing more examples of augmented reality (AR) in various retail industries, including cosmetics, apparel, furniture, etc. for marketing and/or practical purposes. With the help of the launches of Apple's ARKit and Google's ARCore, augmented reality is no longer a future possibility for retail -- it's here. Statistics show that 75% of consumers now expect retailers to offer an augmented reality experience.

Cosmetics Industry

Cosmetics are often a tricky purchase for online shoppers since it is so personalized and may require a tangible trial to ensure customer satisfaction. Augmented reality is becoming increasingly popular within the cosmetics industry to boost e-commerce sales by allowing consumers to find the perfect product virtually. Cosmetics giant L’Oréal acquired Modiface, a beauty try-on simulation, in March 2018. [18] Modiface creates augmented reality tech for beauty brands, which offers the ability to perform beauty try-on simulations on live video, and to track the face and facial features with precise detail [19].

Sephora Visual Artist
The makeup retailer, Sephora, introduced “Sephora Visual Artist” an app developed in partnership with Modiface. Customers are now able to instantly virtually try on lip colours, eyeshadows, and false eyelash styles. The simulation also offers virtual tutorials that show users how to create winged eyeliner, apply contour, etc. by overlaying the tutorial on their face [20]. On the other hand, beauty brand Charlotte Tilbury chose to use AR to focus on improving the in-store shopping experience. The company developed an AR mirror called Charlotte's Magic Mirror that is "the world’s first full view interactive touchscreen" [21] where shoppers can virtually try-on pre-set makeup looks. The product aims to create a realistic visualization of how each look will tailor to the customer's face.

Although the application of augmented reality in the beauty industry is on the rise, the tech still requires lots of fine-tuning. Virtual makeovers are interesting and engaging for shoppers by still far from being "realistic" looking. Beauty companies must find ways to figure out how to render different colours and shades correctly and blend in with the users' natural setting.

3D Products at Home

AR with Furniture
Most people often find it hard to get a sense of what a piece of furniture may look like in their living space. Furniture is undoubtedly one of the harder industries to break into the e-commerce market because customers like to see tangible products before buying, as well as the high shipping costs for heavy items. An online photo of a piece of furniture often does not do the product justice as it is hard for customers to visualize it without feeling the fabric, testing its comfort, and seeing its colour in real life. The introduction of augmented reality for 3D products at home gives shoppers a smoother online furniture shopping experience without having to pull out measuring tapes and having to visit physical stores.

Online furniture retailer, Wayfair, introduced an augmented reality feature for its Android mobile app that allows customers to visualize furniture in their own home before purchasing. [22]Leveraging the technology of Google's ARcore, customers are able to simply hold up their smartphones and point it to their living room, and "try-out" the furniture virtually.

With millennials now being the largest demographic (37%)[23] in the furniture-buying market, it is essential for companies to capitalize on the large e-commerce opportunities to gain competitive advantage. With the help of augmented reality, furniture retailers can better target the tech-savvy and time-crunched millennial audience.

Using Virtual Fitting Rooms

With the e-commerce industry’s continuous rapid growth, many apparel companies are switching their focus to selling their products online. However, one of the main customer complaints of online shopping for clothes is the lack of tangible merchandising experience that physical stores provide, which sometimes results in misfitted clothes. According to Heikki Haldre, Chief Executive and Founder of, almost 1 in 4 garments are being returned, and 70% of these returns are caused by wrong sizing.[24]

The introduction of augmented reality in online fitting rooms help alleviate this drawback of online shopping. Apparel retailer, L.K. Bennett, introduced a virtual fitting room on its website that allows customers to input their own height, weight, and key measurements to solve the issue of incorrect sizing and returns. Customers are able to try on clothes along with options to change patterns and to get a 360-degree view of the fitting and look.[25]

In summary, virtual fitting rooms allow the online shopping experience to be more engaging by improving the confidence in the customer's choices and make the experience more fun in general. Though it has been proven to reduce returns and increase sales, researchers are still unsure whether this technology will be accurate enough to make a lasting change in customers' shopping behaviour [26].

Build Brand Awareness

AR is becoming increasingly common as a marketing strategy for brands to build brand awareness by standing out amongst competitors. For example, shoe company Airwalk used an app developed by Goldrun to create an "invisible" pop-up shop in LA and New York using GPS location and augmented reality technology [27]. Once customers download the Goldrush app, they are directed to the "invisible" stores in Venice Beach in LA and Washington Square Park in New York. Once they locate the virtual shoes within the app at the GPS-tagged location, they gain access to the e-commerce store to purchase the exclusive shoe line [28]. This engaging shopping experience was a success for Airwalk as it resulted in $5 million in earned media and the busiest weekend for its e-commerce store in history [29]

Aside from improving the online shopping experience for customers, companies are also using AR effects to create a unique and interactive shopping experience in physical stores. Popular fast-fashion retailer Zara introduced an AR campaign, placing AR displays in 120 stores around the world for two weeks in April 2018 [30]. Zara experimented with temporarily replacing the mannequins with AR versions of real-life internationally-known fashion models that can be viewed by customers both inside and outside the store through the Zara AR app. With a completely empty storefront display, shoppers passing by were intrigued and curious to point their smartphones at the display to view the AR display of top models cat-walking in Zara clothing[31].
AR with Zara

In conclusion, AR is undoubtedly one of the driving forces behind sales and marketing innovations over the next few decades. Innovative businesses will leverage the AR technology to impress their customers by improving their shopping experience which will lead to increased business opportunity and sales.

Final thoughts

AR enhances the customer experience. Through AR, customers can try on products before buying them. By simply sitting at their home, they can try different variations of the products in terms of sizes and colors. Once they are satisfied with their selection, the exact item is delivered to their doorstep. It basically offers the convenience of online shopping with the experience of physically interacting with the product. AR also reduces the need for in-store staff because stores can now cater to customers' needs to try out different products without in-store assistance. AR can further cut cost by reducing in-store inventory. With the use of AR apps, retail stores can reduce in-store inventory, thus minimizing costs for shipping and storing a large inventory.However, the reduction in employment opportunities in retail due to the introduction of AR technology and other sales innovations is undoubtedly an issue that will need to be addressed. With proper strategies in place, retail jobs must be restructured to fit the current market needs.

Augmented Reality in Tourism & Hospitality

As of 2018, Canada’s tourism industry was able to provide 1.8 million jobs across the country and this number continues to grow. Currently, the tourism market is very competitive as every vendor tries to profit off of the growing market. Consumers have almost unlimited access to information online, as well as social media showcasing the latest travel destinations [32]. While there are popular tourist locations such as Paris, Vancouver and Hong Kong that provide a plethora of information, there are also some places such as Greece that might not be as technologically advanced and miss out on potential business opportunities by not advertising in innovative technologies. Also, as Google’s search algorithm combines many factors, most of the websites that reach the top are typically popular locations, which means that smaller businesses in less popular destinations have a harder time competing within the market and may not have the advertisements to improve their standings[33]. Therefore, it is essential that businesses and countries invest in augmented-reality to help grow their revenues and yearly tourist numbers.

Overall we will be focusing on five ways Augmented Reality Improves Tourism & Hospitality which are listed below.

  • Guide Tourists
  • Educate
  • Break Communication Barriers
  • Improve Customer Experience

Guide Tourists

Premier Inn Map Application
Skin & Bones Museum Application
Kabaq Dining Experience Application

As previously discussed, a challenge smaller businesses face within tourism is not being located in popular tourist destinations or not having the funding to advertise as aggressively as larger organizations. However, with Augmented Reality, a smaller, local business can differentiate themselves within the market to attract new customers who want an engaging, unique experience. Furthermore, Augmented Reality can provide customers with information while they are travelling by being adaptable to any change in plans and being adaptable to a tourist's travelling style.

These benefits can be seen with the Premier Inn who implemented augmented reality through the use of "The Hub". This is a map on all of the hotel rooms' walls which users can scan using their phones and see all the local attractions around them come to life. Whether it be a shopping center, service, or restaurant the user will be able to gain more information about specific locations which will make their travel experience more convenient and optimal as they can tailor their schedule around seeing attractions that seem interesting from the map. Also, this Augmented Reality function allows for local businesses to potentially partner with the Premier Inn so their business shows up on the map first which will increase both organizations' revenues [34].

Educating Tourists with Augmented Reality

Tourists often may visit a location they know is famous, but might not have full background information on what makes it important to the locals. Therefore, through the use of augmented reality, these tourists can learn more about these popular tourist sites such as the Heritage (background information, creator, significance to locals), fun facts, and interesting Stories (Historic moments that involved monument). Therefore, a method the Government could implement into enticing more tourists to visit their country is to partner with popular travel industry websites to build an application that enhances tourist attractions with information the allow users to learn in real-time [35].

Furthermore, there has recently been a significant decrease in attendance to traditional museums across the world. Especially in the United States. [36]. However, Augmented Reality can enhance a users experience at a museum by involving interactive elements such as a visual tour guide. This would make the trip more memorable while educating them at the same time [37]. An example of this being executed today is the Skin and Bones application that The Smithsonian National Museum of Natural History has created [38]. The purpose of this application is to show visitors how the prehistoric animal would have behaved if they were alive today through the use of videos, stories, activities and interactive sounds. Also, by pointing your phone at a set of bones, a recreation of the animal will be overlayed to give you an accurate view of what it would have looked like[39].

Breaking Communication Barriers in Augmented Reality

Even though English is the most common language in the world, there are still parts of the world where English speakers are uncommon and rare to find. Travelling to a foreign country where there is a lack of natives that are able to speak your language can deter tourists from visiting these areas which can cause their tourist revenues to drop. Through the use of augmented reality, it could be possible to help bridge this communication gap by providing foreign tourists with the ability to better understand the country that they are visiting or navigate around the country [40].

One application that can help with this is the creation of "Tunnel Vision" which uses augmented reality to help users better understand the New York subway system (which can be argued as being one of the most confusing transit systems in the world). Through its version of the New York MTA Subway Map, users can locate the entrances and exits easier, view the number of people in a given area so they can avoid large crowds and busy stations, and can access schedule data to receive live updates on a train's location. Moreover, this application would allow foreigners to navigate and adjust to a new transit system and view a translatable interactive guide which would help to break the communication barrier [41].

Improving Customers Experience with Augmented Reality

Combining the previous benefits we discussed for Augmented Reality in tourism, it can be noted that AR makes travelling more interesting and convenient by showing information in a more entertaining fashion. Through our discussion, we can also note how Augmented Reality can be easily applied to any type of company within the tourism sector. So far we’ve only shown examples of AR within hotels and attractions and not restaurants; however, there is an AR application that enhances a users dining experience called “Kabaq” [42]. The full list of features this application provides is listed below.

  • View the dish from all angles (can see a realistic version of their order as opposed to a photo)
  • Manage expectations (allows for users to see portion size and reduce misunderstandings and customer dissatisfaction)
  • Allows for customization and flexibility of orders (how any adjustments they would make look like i.e. remove sauce or add a side of vegetables)
  • Collects analytics to further understand consumers: most popular dish choices, spot any trending food preferences, allows for a more in depth recommendation by staff and can inform restaurant on what future dishes to create

Final Thoughts

So far Augmented Reality looks enticing for many Governments or businesses to adopt in order to increase their revenues; especially with how easy and adaptable it is for companies within the tourism and hospitality industry. However, how profitable it is to implement and sustain Augmented Reality into a business is questionable as this would require statistics on whether customers are visiting due to the AR applications and whether these applications are factoring into repeat visits. Overall, we see the AR industry to grow extensively in tourism and hospitality as countries and companies try to remain competitive by offering customers engaging, innovative experiences.

Augmented Reality in Education & Training

Augmented reality can be particularly effective in improving the quality and efficiency of education and training in various situations for employees and students alike. By adding layers of digital information to what we can see with the naked eye, augmented reality actively engages the human brain by enhancing the education and training experience with added sound, videos, and graphics.

Researchers have reported on how augmented reality can positively impact a learning experience:

  • increased content understanding
  • learning spatial structure and function
  • learning language associations
  • long-term memory retention
  • improved physical task performance
  • improved collaboration
  • increased learner's motivation

With all these benefits of AR for education listed above, there is no doubt that future educators will start incorporating into the classroom and workplace training.

Augmented Reality Classroom

Whether it be interactive paintings in hallways or interactive "trigger" texts and images in textbooks using an AR app, augmented reality is meant to boost the quality of education by improving student engagement. Some examples of data that can be presented in AR in lessons include[43]:

  • a short bio of a person
  • fun facts
  • historical data about sites or events
  • visual 3d models
  • extra data & graphs
When school teachers use AR animated content in classroom lessons, it is a fast and relatively inexpensive way to enhance the interaction between students and teachers and improve students' understanding of complex and/or abstract ideas.

Because the application of augmented reality allows teachers to involve students in the process with 3D models, it can help students see school in a completely different perspective. An interesting case is when Canadian tech company, CASE, added an AR layer to a school gym wall and turned it into a virtual ball game[44]. Kids can throw balls onto a wall to hit floating shapes. By making physical education more interactive and relevant, entices students to be more engaged during physical education lessons and as a result, receive the maximum benefits of education.

Workplace Training

Augmented reality is particularly helpful in workplace training when it involves technical skills that require lots of practice and hands-on experience. Unlike VR, augmented reality features are interactive and help perform a virtual practice. Examples of AR training include but are not limited to: augmented tutorials, digital modelling, and simulations.

An example of AR in the workplace can be simply onboarding a new hire. Instead of asking another employee to guide the new-hire through the on-boarding process, AR can help cut the cost by simply allowing the new-hire to go around the office and scan AR symbols with their phone to receive all the instructions or information they require [45]. Companies can leverage the technology by implementing AR programs to distribute information throughout the organization and speed up the training process while cutting cost.

Light Guide Systems
Light Guide Systems is an example of a successful AR training system that is "designed to make factory floors smarter, safer and better" by transforming manual assembly and manufacturing processes [46] According to Light Guide Systems, their product allowed their clients to improve quality by 80% and improve productivity by 40% in training [47].

Final Thoughts

High-quality education and training is essential to the development of our future leaders and thus the betterment of the human race. AR is undoubtedly an important innovation that will enhance this process. However, one of the biggest drawbacks of using AR in education is the decrease in human interaction with less involvement with a human instructor. While learners experience a more engaging learning experience and companies may cut cost, AR causes a higher level of reliance on digital information. When the human brain relies too much on digital information, there may be a decrease of working memory in the brain which in turn hinders the development of brain functions [48].

Another major concern is privacy. As AR systems collect personal information, user privacy can be negatively impacted. This can be especially problematic when biometric data of students and employees are collected, which calls for an extra need for the protection of personal information. We believe that companies and schools can minimize the risk and impact of unauthorized access by regularly purging the data and clarifying in the contract their own data retention and privacy responsibilities.

Augmented Reality in Navigation

Gatwick Airport Application

Similar to tourism, Augmented Reality is also making strides to increase the integration of navigational systems. By involving real-time visuals and data to the user, these systems can aid in increasing efficiency and effectiveness. This ranges across a variety of activities such as getting to a flight on time, navigating a new city, or transferring a shipment across the ocean. These systems can be implemented into personal electronic devices such as smartphones, screens, and tablets, or be hand-free fully integrated AR visors/goggles. In contrast to virtual reality navigational systems, augmented reality integrates the virtual images and data with the environment which makes them much more applicable when it comes to navigation. The section below will discuss some of the AR applications within navigation.

Gatwick Airport

Gatwick Airport’s Passenger Application, launched on December 2017 in response to November 2017 being one of the busiest months on record. This smartphone application allows users to navigate the airports two terminals with the help of its 2000 navigation beacons. Some of the primary features that it provides are listed below [49].

  • Live instructions on how to get to a destination which may be a shop or terminal gate (includes arrows)
  • real time updates on flights, gate changes, and queues (aids users in time management)
  • Information on facilities for the disabled
  • road and rail travel information

Furthermore, this application is more accurate compared to GPS as the airport has specifically deployed their own navigational beacons to ensure there is no disruption on the application. In terms of security the airport states that none of the consumers' personal information is saved, but information such as foot traffic will be noted so features of the app such as wait and travel times can be accurately measured. Overall, this technology opens the door for the adoption of other airports or transport providers to provide more convenient services to their customers [50].


Google Maps AR

In February, Google announced the launch of Google Maps AR which essentially is an elevated version of the original Google Maps. This came as a response to the criticism of Google Maps not accurately pin-pointing a person's GPS when using the map and instead the navigational indicator would be varied such as incorrectly assuming you were across the street when you were not. This application is different from Google Maps because as you hold your phone up, Google Maps AR will have an arrow that indicates where to turn and walk. It also has a safety function that prompts the user to put their phone down when crossing the road. Unfortunately, this application is only available to select guides in certain cities, and it has yet to be announced when Google will officially launch Google Maps AR [51].

Before Google does a full release of this application, safety features will need to be reassessed as it may not be enough just to send a prompt to the user to remember to look up from their phone. It will be interesting to see whether any major accidents could come as a result of this technology [52].

Mitsui OSK Lines (MOL)

Announced on Maritime Executive, the Mitsui O.S.K Line (MOL) is integrating Augmented Reality onto 21 of their Very Large Crude Carriers (VLCCs). Mitsui O.S.K Line is a Japanese transportation company that specializes in marine transportation, warehousing, and cargo handling services. They mainly transport items such as aluminum, cement, ore, wood, grain, coal, gasoline, and so forth across the ocean [53].

The news on MOL implementing an Augmented Reality system was launched in April 2019. The primary reason for developing this navigational system was to aid the VLCC’s in transporting a large amount of crude oil across the ocean in a safe environment. Its function is to display information on the vessels routes, ocean conditions, and surrounding sea levels (identify shallow water). The data will be visible on the ship's screens and tablets where it will analyze the direction the ship is travelling in and what surrounds it. Also, the application will notify the ship if any other vessels are nearby and if the ocean's conditions were to change through icons on the ships navigational screen. Overall, this system is beneficial because it can potentially prevent or predict any accidents the ship may face due to sea conditions or passing vessels [54].

Final Thoughts

Based on the recent launches we just discussed, augmented reality looks to benefit navigation in a significant way as it allows users to be more efficient in time management, especially in highly stressful situations such as navigating an airport or visiting a new country. While Google Maps AR and Gatwick’s Passenger application proves to be informative to users, it also makes users more socially isolated and reliant on their phone application as opposed to engaging in human interaction. Overall, we believe that the technology will be a net benefit for society; however, typically, people would ask for directions or assistance but with the creation of Augmented Reality navigation systems, one can argue that it makes us as a society more introverted and withdrawn.

Augmented Reality in Surgery

The State of Surgery Currently

AR-Assisted Surgery

In the past decade, we have seen large growth in health care technologies that help health care professionals provide the most optimal services to their patients. Despite this, image displays have stayed basically the same since 1950. Currently, data is shown on screens around the room which means that surgeons and their assistants must look away from their patients if they need additional information. Furthermore, surgeons must use skill, precision, and imagination to mentally project images onto their patients during procedures since the images are from the perspective of the imaging device rather than the surgeon himself. Also, there is no integration of data meaning results from a CT scan, blood test, X-Ray, and more are often displayed on different monitors. Thus, all of this information combined means that surgeons have a lot of work on their hands to make their procedures go as smoothly and safely as possible [55].

How Augmented Reality Can Improve Surgeries

With the issues of the current state of surgeries now established, this section will talk about the three main ways that AR-assisted surgery can help to solve these issues. These three main ways are listed below [56].

Better Data Visualization

Augmented-reality assisted surgery allows for digital images and other medically important data to be overlayed over a surgeon's field of view rather than being displayed on monitors around the room. This means that visualizing critical structures such as nerves, blood vessels, and other vital tissues would be much easier. For example, if a surgeon wants to view a patient's CT scan, all they need to do is put on the Hololens glasses and it can be displayed right in front of them. Also, AR could superimpose someone's veins onto their skin using heat signatures, which would make them much easier to locate [57].

Easier Interaction with the System

With all the data centrally integrated into one spot, surgeons are able to access all the information they need very easily and never have to look away from the patient to their multiple displays to gather and interpret information. Also, the AR Hololens glasses can be utilized using voice recognition to create voice commands which means the device is hands-free which is essential during surgeries [58].

Receive Instruction and Guidance

One of the great features of Hololens glasses for surgery is that data can be shared over an interactive network with other surgeons (as long as they have Hololens glasses too) which means that other experienced surgeons can remotely assist during exceptionally crucial surgeries [59].

Other Benefits

Augmented-reality assisted surgery also provides many other benefits such as making surgeries safer for the patients. For example, when someone has a pre-planned surgery, such as for a broken arm, they will often receive all of their pre-operative imaging and give all the relevant information to ensure the surgery will go smoothly. On the day of the surgery, they will have 4-8 medical professionals on-site to fill in all of the necessary roles. However, quite often emergencies can occur and if someone gets into a car accident and needs medical attention right away, there may only be 2 medical professionals on-site to perform the surgery. Furthermore, there may not be any rooms available that are designed for surgery meaning the surgery will have to be performed outside the operating room where all necessary monitors may not be present. To make matters worse, not all relevant data and imaging will have been performed on the patient or stored in the system. Therefore, in this case, augmented reality would be exceptionally helpful in integrating all data and imagery to ensure that the operation is successful [60].

In addition, augmented reality would reduce costs by reducing the use of redundant screens. Currently, many displays require that the hospital buy the whole system which can often be hundreds of thousands of dollars each. With so many different displays (ultrasound, bronchoscopy, endoscopy, etc.), this can increase costs exponentially [61].

Furthermore, augmented reality can be very helpful in training situations where junior surgeons can train using integrated visuals and data to better understand the inner workings of a surgery. Also, it allows for more experienced surgeons to train using the AR glasses so when they use them during a real surgery, they understand how to work most efficiently using them [62].

Final Thoughts

Based on our analysis of augmented reality-assisted surgery, we believe that it currently has the most potential for training introductory surgeons and for use in low-risk surgeries. While successful surgeries have been done in relation to scoliosis, we think that they are just the testing ground and will not necessarily become the norm for quite some time. Also, we believe that some key issues need to be addressed before AR can be used in common practice. For example, if someone dies due to a software malfunction, will the providers of the AR glasses, the hospital, or the surgeon be held responsible? Also, senior surgeons who may not be well-equip with technology may resist change and be very reluctant to accept AR in the workplace. Moreover, probably the most important issue is that it is not yet proven that AR would reduce the fatality rate and provide for more successful surgeries which means it is a risk in and of itself to utilize them.

Augmented Reality in Gaming

The Basics of Gaming AR

Gaming using augmented reality involves the integration of audio and visual content with the user's real-world environment. These games can often be played anywhere; whether it be outside at the park, inside a house, or at the shopping mall. They are usually played on personal electronic devices such as tablets, smartphones, and portable gaming systems; however, some AR gaming headsets do exist. This is in stark contrast to virtual reality gaming which often confines users to a room, only uses a cumbersome headset, and creates a new immersive environment [63].

Augmented reality gaming ranges from very simple to very complex games. For example, it could be playing a game of chess that is superimposed on a coffee table against your friend, or it could involve completely enhancing your current environment to a game's environment [64]. Several specific games will be discussed more in-depth below.

Pokemon Go

Pokemon Go

Pokemon Go, while not the first augmented reality game, is arguably the one that brought the technology to the forefront. Many people did not know what augmented reality meant before this game, and gaming companies have tried to emulate its success since it was released in 2016. The game uses the smartphone's gyroscope, GPS, and clock to create an experience for users that utilizes tracking and mapping technologies. As users walk around outside, Pokemon appear on the map in which users may then catch and train. Pokestops also exist at real-world locations where users may collect free pokeballs or pokemon eggs. Lastly, users may use their collection of pokemon to fight other users or control locations on the map [65].

The success of Pokemon Go can be largely attributed to several main reasons. First, the game is very nostalgic as many people in their late teens or early adults grew up playing Pokemon and have not done so since they were young, Next, the game utilizes the physical environment, providing a social atmosphere where people are incentivized to travel to new locations. Lastly, the game is free meaning any user can easily download and play it within 5 minutes [66].

However, despite Pokemon Go's initial success, the game started to decline almost as quickly as it increased in popularity. This is for several reasons with the first being Niantic Labs was not prepared for the huge initial adoption of the game meaning server outages were common in its first month of operating. Also, GPS spoofing was common where players manually changed their location in the game allowing them to appear in a location that they were not. Furthermore, while the game was nostalgic, it could not rely on this since overall there was not a lot to do and the game felt like a grind if people wanted to accomplish anything. Lastly, the game killed phone batteries very quickly which meant users could only go out and play it for a couple of hours at a time before needing a full phone charge [67].

It is hard to tell if the decline of Pokemon Go was because of the inner mechanics and functions of the game or whether it was because free mobile games naturally suffer from poor retention rates; however, the game went from 28.5 million players one week after release, to 15 million one month after release, to 5 million three months after release. The developers tried to revive the game by introducing many new features, but have been unsuccessful in gaining back their initial player base [68].

Harry Potter Wizards Unite

Another game that recently tried to take advantage of users' physical environment with AR is Harry Potter Wizards Unite which released on June 21, 2019. How it works is players are first given the ability to choose their wizarding house, wand, and profession. After, they may interact with their environment, by walking around the real-world (similar to Pokemon Go) and stopping at Inns to replenish spell power, greenhouses to get potion ingredients, and fortresses to fight creatures [69].

However, unlike Pokemon Go, Harry Potter Wizards Unite never got a huge spike of early adopters as it only saw 4% of the revenue on its launch weekend that Pokemon Go saw in its first four days after release. This has been attributed to several reasons with the first being that this style of gaming is not fresh anymore and is a bit repetitive when compared to similar games such as Pokemon Go, Jurassic World Alive, and Ghostbusters World. Also, Pokemon Go lent itself to travelling as that is a core part of the game; however, the harry potter series never really saw travel as a core part of its mechanics. Furthermore, the popularity of Harry Potter has decreased since the release of its final movie many years ago meaning fewer players have an interest in the series as a whole [70].

The Future of AR in Gaming and Final Thoughts

It is expected that augmented reality gaming will become mainstream by 2023, but it is still up in the air how the industry will expand moving forward. It is very likely that the industry will move beyond strictly mobile applications into an environment focused on more immersive experiences. The two likely ways that the industry could go are described below [71].

  1. Pairing mobile devices with a headset
  2. Standalone systems similar to a video console with AR-specific games

Overall, before this industry can expand, we believe that a few issues need to be addressed. First, most AR gaming on personal electronic devices drains the battery very quickly which can negatively affect the motivation and drive to play the game. Next, often times the hype does not match reality since games ride on the idea that if the game utilizes AR that it will automatically be good despite the fact that it lacks many other appealing features. Also, with this reliance on AR that I just discussed, it means that there is often stale innovation as companies tend to copy each other in their core premise making for a repetitive and unappealing experience. Lastly, the field of view issues need to be fixed since having to stare at a phone screen for hours on end can be exhausting. Thus, while we believe that there is a future in AR gaming, it will likely be a while before it will ever be as fun or as popular as the traditional console/handheld experience.

Augmented Reality in the Military

The basics of Military AR

AR in the Military

The Military has historically been known to be the driver of technological advancements as they often use emerging technologies for combat training and tactical purposes far before they are available to the public. This is mostly because the military is looking for opportunities to get ahead of their enemies when it comes to the evolving technological warfare environment. Augmented reality is no exception to this as the military has been experimenting with it for years. While we see advanced warfare, first-person shooting games as an embellishment of the truth, they are not too far off of what we could see soon [72].

The first area the military utilized AR was with fighter jet pilots who had data overlayed on their visor so they could keep their eyes straight ahead at all times and could have much better situational awareness. This information included spacial orientation data (e.g. altitude, angle, direction, distance) and weapons targetting. However, in seeing the benefits with AR in flight, the military has made plans to expand it to be utilized by ground troops through headsets and wearable glasses [73].

In November of 2018, Microsoft signed a $480 million contract with the U.S. military to provide them with a specialized, enhanced version of the HoloLens mixed reality headset called an HMD. The U.S military will be purchasing around 100,000 of these devices which is around twice as much as the last 2.5 years of total Microsoft Hololens sales combined. This is a huge step forward for Microsoft as they took the contract from Apple and Google who both haven't invested nearly as much into AR. A few other AR B2B vendors tried to secure this contract, such as Magic Leap, but were unable to do so. Overall, this contract will have huge benefits for the public since advanced technology in the military often ends up in the public's hands later on in the future, and it will provide Microsoft with the resources to continue investing heavily into augmented reality. [74].

Tactical Augmented Reality (TAR)

The tactical AR system is a pair of goggles that are mounted on the soldier's helmet and can operate during both day and night, similar to how night-vision goggles work. They have many great features that are listed below.

  • Shows positioning of enemy and ally forces
  • Shows own location similar to a handheld GPS
  • Shows distance and location of target through thermal sight on gun which is connected to the AR system
  • Display can be split into two where you can see frontal camera on helmet and where your gun is pointing at the same time
  • Allows for information and daata to be shared in real-time through wireless system

Overall, this technology increases situational awareness, decreases the number of devices needing to be managed and carried, and provides safety features to help keep more soldiers alive. However, it is still a long ways away and will likely be years before any significant usage will be done using this technology [75].

Potential Risks of AR in the military

By implementing new technology into the military for the use of ground troops, it completely changes the way that they will operate. This means that many additional risks will need to be addressed before the technology can be used for more than just combat training purposes. Below are the three primary risks that have been identified.

Overload of information on the AR devices may be distracting

Probably the biggest risk of using this technology is that the data points displayed on a soldier's field of view may be invasive rather than helpful which can be distracting when being put into a high-stress situation. However, the military assures that the opposite is true and this device will reduce cognitive load since it will help with the decision-making process and reduce how many devices have to be managed as it will operate through an integrated system [76].

Misleading display labelling ally or civilian as the enemy

Another huge risk is that if a sensor fails or there is a bug in the system, an innocent bystander, ally, or civilian will be labelled as an aggressor and as such will be shot. Developing an algorithm that could differentiate between enemy and civilian would be very difficult and is sure to have many errors that would have to be worked out. The military assures that this is not an issue because soldiers have control over their own decisions and actions, and thus will be able to override what the system is telling them. However, when put into a high-stress situation, soldiers may not have the time to think of whether someone is an enemy if the system is telling them so [77].

Security concerns using AR technology

One risk that needs to be addressed is how to properly control the devices to maintain high levels of security. These devices are going to store a lot of key information such as control points, mission intel, and communication. If an enemy soldier gets their hands on this device then they will be able to access this information and use it to their advantage.

Final Thoughts

Through our research, we believe that the military will be one of the driving forces to evolve augmented reality to the next level of sophisticated technology. It would be great to see some of this technology eventually be provided in the form of enhanced consumer products or experiences in our everyday lives. However, with the risks in mind, we believe it creates a lot of potential issues based on adoption and public perception. If a civilian is accidentally killed using this technology then will Microsoft be held liable for having a bug in the system, the military for not doing their due diligence, or the soldier who ultimately fired the shot. Public perception could increasingly disfavour this technology if bad things such as this were to occur. Furthermore, this technology raises the issue of why the military needs more advanced warfare techniques. As a society that can prosper without war, investing heavily into the military is done under the assumption that eventually, we will go to war again.

Augmented Reality Summary

In summary, augmented reality was a technology that was pretty much unheard of a decade ago; however, now you can find many common examples of it when you turn on your phone or go out in public. Through our analysis, we talked about the different kinds of augmented reality and their applications into many different areas including retail, tourism, hospitality, education, training, navigation, surgery, gaming, and the military. While these may seem like very different ways to apply AR, they are actually much more similar than they appear and can be tied together quite nicely. The section below will talk about some of these commonalities.

Increased User Satisfaction & Self-Fulfilment

All of these applications can be argued to increase the user's satisfaction or levels of self-fulfilment. For example, going to the museum and interacting with displays, playing Pokemon Go by utilizing the physical environment, learning in school through the use of AR, and being able to view a dish of food before ordering it all provide a higher level of satisfaction. Now, we are not saying that these applications of AR will necessarily make you happy, but rather that they will increase your satisfaction even if it is minor. A person may not be happy conducting surgery or fighting in the military; however, they will be more satisfied and less stressed using augmented reality rather than if they were not using it. I think this raises some key discussion points about the advancement of augmented reality. If AR is perceived to increase our satisfaction does that mean the more that we enhance it the happier we will be in our lives? This seems counterintuitive as more technology often leads to social isolation and increased stress.

Increased convenience

All of these applications can also be argued to increase convenience and save time in activities that may be small or large in nature. For example, being able to virtually try on objects, have all surgical information overlayed in front of you, see relevant navigation information in a foreign city, or have projected instruction s to guide you during training are all ways that activities can be performed using less effort and time. In the future, it is likely that AR will continue to be implemented that will make living life much simpler as even the most basic of tasks become enhanced with technology. As a society, we will either use this extra time to become more productive and focus on our own needs, or we will use it to become more reliant on technology.

Reduced Privacy

The idea that all of these applications of AR would result in reduced privacy seems ridiculous at first; however, upon deeper insights, it becomes more evident. Every time we open an application on our phone using AR, data is going to be gathered. This could be location data for navigation, preferences data for tourism, hospitality, or retail, or user demographics data for gaming. Even AR applications in the military would reduce our overall level of privacy as information becomes more centralized and accessible in AR systems and spy technology becomes more sophisticated. Also, with augmented reality-assisted surgery patients' integrated data can be easily accessible and shared with other medical professionals across the online network. As AR continues to develop over the next decade, privacy concerns are going to need to be addressed and an acceptable boundary established to determine when the technology goes too far.


Personal tools